{"schema":"coreai-d1-fixtures/1","what":"The fixture, the held-out set, the picture records and the red arms of the d1-3B port, with the provider's fp32 readout (the oracle) of every question and the two shipped decoders' probabilities (graph.mac_fp16: fp16 S=64, the Mac's; graph.iphone_int8mlp: int8mlp S=64, the iPhone's; each the AOT h16c asset of its bundle's .aimodel on an Apple M4 Max GPU; the held-out set on the int8mlp decoder only). 361 records / 394 questions (393 answerable), a held-out file of 120 and 12 picture records / 24 questions. The SemIf authored144 items, the model card's example requests, the two long states written for the port and the drawn pictures' requests carry their text; the transfer-v4 records (the Kev suite's file) are references with their hashes; the own_* records keep only their numbers (invented names that were never screened).","model":{"hf_id":"LiquidAI/d1-3B","revision":"da1fe36a861f24690f27f622dca1d8688503d113","license":"LFM Open License v1.0","oracle":{"code":"the provider's code at the revision (AutoModel + trust_remote_code, fp32, CPU, one thread): the row form per question (each question's own row) and the API path (several questions: the provider's Tree)","versions":{"python":"3.12.11","torch":"2.9.1","transformers":"5.19.0","tokenizers":"0.23.2","numpy":"2.5.3"},"platform":"macOS-27.0-arm64-arm-64bit"}},"graph":{"bundles":{"mac_fp16":"d1_3b_decode_fp16_pf64","iphone_int8mlp":"d1_3b_decode_int8mlp_pf64"},"shipped":"the released bundles are these with their debug locations stripped (d1_3b_decode_fp16_pf64_s, d1_3b_decode_int8mlp_pf64_s, d1_3b_vision_fp16w32_s); on the Mac they give every probability here bit for bit (gate-d1-3b-strip.json, bits)","tower":"d1_3b_vision_fp16w32","asset":"AOT h16c .aimodelc of each bundle's .aimodel (coreai-build compile --platform macOS --preferred-compute gpu --architecture h16c; the decoder with --expect-frequent-reshapes), Apple M4 Max GPU, macOS 27.0 26A428, SpecializationOptions.default(), Python runtime","p":"the hidden row at the answer slot through the host's readout (option rows of head/, group max, softmax over the options, float64)","transcripts":{"fixture":"gate-d1-3b-readout.json","heldout":"gate-d1-3b-heldout.json","pictures":"gate-d1-3b-images.json","red_arms":"gate-d1-3b-red.json"}},"contract":{"row":"one row per question, fresh zero states: BOS + '<|im_start|>user\\n' + [the pictures' token runs] + state block + '\\nQUESTION:\\n' + question block + '<|im_end|>\\n<|im_start|>assistant\\n'; the answer slot is the row's last token","readout":"z[id] = h_slot . E[id] (float64) over the question's option groups, group max, softmax over the options (the provider's vocabulary log-sum-exp cancels)","keys":"choice: the labels in request order; noul: ['yes', 'no']; score: '0'..'K-1'","near_tie":"oracle top-2 margin <= 0.02","row_ids_sha256":"sha256 of the row's token ids as little-endian int32","request_sha256":"sha256 of json.dumps(request, sort_keys=True, ensure_ascii=False) in UTF-8"},"sources":{"kev_fixture":{"what":"the LiteRT Kev lane's round-1 fixture (kev_work/fixtures/requests.json, sha256 dfe55fb145df7a3967ed7213ae24d67b5d4ec42da0315fdb409d5e4b702e3a48): its transfer-v4 rows (built from github.com/jaredpalmer/kev @ kev-1.0, evals/v4/transfer-v4/development.jsonl), its SemIf rows and its own_* rows","source_licenses":{"mmlu":{"dataset":"cais/mmlu @ c30699e8","licence":"mit"},"emotion":{"dataset":"dair-ai/emotion @ cab853a1","licence":"other"},"tweet_offensive":{"dataset":"cardiffnlp/tweet_eval @ b3a375ba","licence":"unknown","d1":"excluded"},"qnli":{"dataset":"nyu-mll/glue @ bcdcba79","licence":"other"},"paws":{"dataset":"google-research-datasets/paws @ 161ece95","licence":"other"},"sciq":{"dataset":"allenai/sciq @ 2c94ad3e","licence":"cc-by-nc-3.0"},"legacy_holdout":{"dataset":null,"licence":"generated by the Kev suite code (the kev repository's Apache-2.0)"},"composition_holdout":{"dataset":null,"licence":"generated by the Kev suite code (the kev repository's Apache-2.0)"}},"excluded":"the 20 tv4x_tweet_offensive_* records (licence unknown on the Hub card, real public figures' names and profanity) and red_arm_000 (a gate arm)","transfer_v4_what_is_redistributed":"nothing of the rows' text or token ids: each record names the file, the line and the row's _meta (ids and sha256), with request_sha256 and row_ids_sha256 to check a rebuild"},"semif_authored144":{"upstream":"github.com/TheoLeeCJ/SemIf @ ca3ba65f142967030ecb453346e94d6f476a69df","file":"benchmarks/data/authored144.jsonl (sha256 8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079)","license":"MIT","license_file_sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be","license_notice":"MIT License\n\nCopyright (c) 2026 TheoLeeCJ\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","what_is_redistributed":"the 144 rows' state, question and option texts in the request shape (records with source 'semif'), with the MIT copyright and permission notice above"},"card":{"what":"the three example requests of the model card (README.md of LiquidAI/d1-3B @ da1fe36a, section 'How to use'), as printed","license":"LFM Open License v1.0","readme_sha256":"cf063f71ce852ef4c924d192c34f61727f11d121c60b01ad9092ba93ac03bda3"},"own_long":{"what":"two JSON states written for this port (a cold-room log, an order ledger) at about 1,500 and 3,400 tokens of state block","license":"BSD-3-Clause, as the repository","names":"no person, organisation, product or place name: codes, dates, amounts, role words"},"own":{"what":"the LiteRT Kev lane's 12 own_* records (kev_work/scripts/own_records.py)","withheld":["own_email_03","own_fiveq_09","own_incident_02","own_invoice_07","own_long_contract_11","own_long_log_10","own_long_minutes_12","own_meeting_04","own_order_06","own_review_05","own_sensor_08","own_ticket_01"],"why":"they name invented people, companies and products, and no screen of those names is recorded (the house check: exact-phrase web search per name, DNS, App Store, a company register); each keeps its id, request_sha256 and numbers"},"pictures":{"what":"12 pictures drawn by conversion/d1/make_fixture_images.py (Pillow primitives from a fixed seed; no photo, font or text), CC0-1.0; not stored here: the script redraws them (raw RGB sha256 reproduced; the PNG bytes depend on zlib)"}},"records":[{"id":"tv4_000","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":0,"line_sha256":"65e5e955976c82abb47df3b3287409cf867c6af264235c0c32f606cb8681de83","_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292","group_id":"mmlu/test/13292","variant":"clean"}},"request_sha256":"a48ef8781d96d0b1c37fc8d2a3cb35120a6e9a5edcd9b27b0e49c90fc2ff0a70","questions":[{"type":"choice","n_options":4,"row_len":106,"slot":105,"row_ids_sha256":"1b9fd1cbfb82de38935175b12bf44f17442a3b9c88eed90894fe6edab31025f8","oracle":{"probs":[0.012302076516283622,0.030425813328589246,0.947932139979996,0.00933997017513106],"argmax":"c","top2_margin":0.9175063266514067,"near_tie":false,"group_logz":{"73":-4.4337005615234375,"267":-11.47968864440918,"74":-3.528177261352539,"293":-10.300138473510742,"75":-0.08918571472167969,"280":-7.087766647338867,"76":-4.709165573120117,"277":-11.419200897216797}},"graph":{"mac_fp16":{"p":[0.012278654065537022,0.030389674486733683,0.9479918105347979,0.009339860912931474],"max_abs_dp":5.967055480182726e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012317559602245474,0.030557784835101284,0.9478103831813423,0.009314272381311035],"max_abs_dp":0.00013197150651203873,"argmax_equal":true}}}]},{"id":"tv4_001","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":1,"line_sha256":"56f810d754b4cb990275166ed819a6fdd67b05710dacb06ed9731548a5fea1dc","_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588","group_id":"mmlu/test/588","variant":"clean"}},"request_sha256":"3993d58af1adad42c0a57c7463164412b440f8f8fdcde5d8f6d92a22ad3d8123","questions":[{"type":"choice","n_options":4,"row_len":77,"slot":76,"row_ids_sha256":"4aa0708261848fceff0c12d93546db2741cce7e255a8169dfd6e27d8daec706f","oracle":{"probs":[0.0011603837368078472,0.003069886708698145,0.9919266065473916,0.0038431230071023417],"argmax":"c","top2_margin":0.9880834835402893,"near_tie":false,"group_logz":{"73":-6.796102523803711,"267":-14.181131362915039,"74":-5.823212623596191,"293":-13.684246063232422,"75":-0.04520416259765625,"280":-8.015016555786133,"76":-5.598567962646484,"277":-13.088876724243164}},"graph":{"mac_fp16":{"p":[0.0011564291320507315,0.0030649005979227193,0.9919467820819102,0.0038318881881162898],"max_abs_dp":2.0175534518607208e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0011469111242165457,0.00304382684147775,0.9919810191023674,0.003828242931938439],"max_abs_dp":5.4412554975735716e-05,"argmax_equal":true}}}]},{"id":"tv4_002","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":2,"line_sha256":"668f5021cf322641a26d4cd4cbe417f80011784f61eeeb6ee18c3d5683877f9a","_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589","group_id":"mmlu/test/4589","variant":"clean"}},"request_sha256":"a395a1856eb321f102af71d4807ff0f0f682a264315d40972f4df31a35ed7525","questions":[{"type":"choice","n_options":4,"row_len":110,"slot":109,"row_ids_sha256":"ce0f3d6ca3790d5929bc2fdf6c2eb005d77d876ee28e783b775009086a17a724","oracle":{"probs":[0.9731359394576886,0.0052661614276986535,0.01420519113840435,0.007392707976208357],"argmax":"a","top2_margin":0.9589307483192843,"near_tie":false,"group_logz":{"73":-0.06461334228515625,"267":-8.253935813903809,"74":-5.283835411071777,"293":-12.700796127319336,"75":-4.291529655456543,"280":-11.565847396850586,"76":-4.944643020629883,"277":-11.656245231628418}},"graph":{"mac_fp16":{"p":[0.9731254585085373,0.0052785401589175085,0.014203343252693467,0.007392658079851795],"max_abs_dp":1.2378731218854952e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9727798744662309,0.00526216123810256,0.014503901808074881,0.007454062487591588],"max_abs_dp":0.000356064991457683,"argmax_equal":true}}}]},{"id":"tv4_003","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":3,"line_sha256":"07c8fc2c87043af4a643fa2664336d5dbe1bb67fb5263fde3b8f1e96defd5dcb","_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099","group_id":"mmlu/test/6099","variant":"clean"}},"request_sha256":"3909102a08f46ebb402bfaaeddc5a60471ca2bd166df3b46c482b82e2952d2e5","questions":[{"type":"choice","n_options":4,"row_len":94,"slot":93,"row_ids_sha256":"78b3da6a52fc207043393d7248eb5b402cff76afd71b225b02f6ed16c17a2d57","oracle":{"probs":[0.9199264706097146,0.04084019872146375,0.029818211182998485,0.0094151194858233],"argmax":"a","top2_margin":0.8790862718882508,"near_tie":false,"group_logz":{"73":-0.12885475158691406,"267":-7.731670379638672,"74":-3.2434816360473633,"293":-9.879180908203125,"75":-3.5580291748046875,"280":-10.20610523223877,"76":-4.710831642150879,"277":-10.668848037719727}},"graph":{"mac_fp16":{"p":[0.9198334848974701,0.04092974686133629,0.029832238371388625,0.00940452986980487],"max_abs_dp":9.29857122444755e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9195646832287427,0.040794011699545406,0.03007908187202458,0.009562223199687301],"max_abs_dp":0.00036178738097181196,"argmax_equal":true}}}]},{"id":"tv4_004","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":4,"line_sha256":"0b68b799521da2826e8661997053ccf67c4c896684e9406459f674fda5b2f287","_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139","group_id":"mmlu/test/6139","variant":"clean"}},"request_sha256":"febb34864cb3f1df3cb819981502d94057902c90499cbdb5c679acad738aab31","questions":[{"type":"choice","n_options":4,"row_len":82,"slot":81,"row_ids_sha256":"f1a60cba43b75d502b923b3c01f9dc312d1ed570854b2ca468061ebb0565bc13","oracle":{"probs":[0.0028989416255763355,0.004751913475314963,0.9856133825023121,0.006735762396796514],"argmax":"c","top2_margin":0.9788776201055156,"near_tie":false,"group_logz":{"73":-5.887844085693359,"267":-13.31503963470459,"74":-5.393642425537109,"293":-12.85434627532959,"75":-0.058925628662109375,"280":-7.690778732299805,"76":-5.0447587966918945,"277":-12.195676803588867}},"graph":{"mac_fp16":{"p":[0.0029011714115998863,0.00476326850651488,0.9855790032016438,0.006756556880241421],"max_abs_dp":3.4379300668296864e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.002898554705320874,0.004744195971485956,0.9856221948911377,0.006735054432055521],"max_abs_dp":8.81238882555202e-06,"argmax_equal":true}}}]},{"id":"tv4_005","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":5,"line_sha256":"72a72e39855ff80bac41e07ab26fdefde7d730c0ecd25723e41241d5d572ab53","_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810","group_id":"mmlu/test/13810","variant":"clean"}},"request_sha256":"9711cb3745a9a8b53ad5eab459464ac9ef15f001bec130779adbff86bfd94c4f","questions":[{"type":"choice","n_options":4,"row_len":93,"slot":92,"row_ids_sha256":"dc64925a7259fb14bce76b375e73199a52a26e17f7d9cc463f978009770623e0","oracle":{"probs":[0.9588720834109825,0.028685469101322133,0.008205336242621816,0.0042371112450735595],"argmax":"a","top2_margin":0.9301866143096604,"near_tie":false,"group_logz":{"73":-0.08384895324707031,"267":-8.389857292175293,"74":-3.5932159423828125,"293":-10.648548126220703,"75":-4.844821929931641,"280":-12.087629318237305,"76":-5.505724906921387,"277":-12.354567527770996}},"graph":{"mac_fp16":{"p":[0.9589055821928747,0.02864539577568856,0.008210920524387374,0.004238101507049422],"max_abs_dp":4.007332563357377e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9577248930112409,0.029707458642541533,0.008292579973071867,0.004275068373145741],"max_abs_dp":0.0011471903997416488,"argmax_equal":true}}}]},{"id":"tv4_006","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":6,"line_sha256":"a6b6b58f862a9d6b00840836ad6eb39240c7678f95758101815267d19d234362","_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434","group_id":"mmlu/test/12434","variant":"clean"}},"request_sha256":"7f0ae7b22ef87865e2e264c9cc34ee9b899961367f929552152cece17a7bbc63","questions":[{"type":"choice","n_options":4,"row_len":249,"slot":248,"row_ids_sha256":"1f59077e78c5696824c62957cd8a4477ae3be05ed82202a0419f7668ccb8657a","oracle":{"probs":[0.04309455413253722,0.7214230840897494,0.028872777563137598,0.2066095842145758],"argmax":"b","top2_margin":0.5148134998751737,"near_tie":false,"group_logz":{"73":-3.241861343383789,"267":-10.094707489013672,"74":-0.42403221130371094,"293":-6.937433242797852,"75":-3.6423587799072266,"280":-10.258532524108887,"76":-1.6744270324707031,"277":-7.86057186126709}},"graph":{"mac_fp16":{"p":[0.04289392491847002,0.7215465064078608,0.02878153428530951,0.20677803438835965],"max_abs_dp":0.0002006292140671953,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04278975837450032,0.726044298846848,0.028713219819969174,0.20245272295868266],"max_abs_dp":0.004621214757098535,"argmax_equal":true}}}]},{"id":"tv4_007","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":7,"line_sha256":"03798a32d17a3722b4a0812045f32cb3950273c26ae67b12f804cb91b29f8245","_meta":{"row":123,"text_sha256":"b91ae5229f181bae8ead6b7d5617ebf84c8a0a6e86eb7b45f9f04ff55413c613","row_sha256":"041edcea28c4c2a8b1b57e50a75d175e7efa26cce57a5e8dc554add9a2369644","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/123","group_id":"mmlu/test/123","variant":"clean"}},"request_sha256":"c11d4b4c2183f911b43f6192ab3efc729ef1e344ca68c9bf56b87cb7fe11473c","questions":[{"type":"choice","n_options":4,"row_len":95,"slot":94,"row_ids_sha256":"9baf36fe6023a27979b69ee2e973c680d7fdc85bb6fec7698ac7891048935013","oracle":{"probs":[0.023409989809698373,0.08497871575738493,0.8841990118545163,0.0074122825784002635],"argmax":"c","top2_margin":0.7992202960971314,"near_tie":false,"group_logz":{"73":-3.810110092163086,"267":-10.660807609558105,"74":-2.520872116088867,"293":-8.93621826171875,"75":-0.1785907745361328,"280":-6.829948425292969,"76":-4.960134506225586,"277":-11.294169425964355}},"graph":{"mac_fp16":{"p":[0.023356839709381592,0.08467378692227344,0.8845473061120761,0.007422067256268974],"max_abs_dp":0.00034829425755977805,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02355427695457015,0.08468132396828705,0.8843716786035336,0.007392720473609269],"max_abs_dp":0.00029739178909787833,"argmax_equal":true}}}]},{"id":"tv4_008","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":8,"line_sha256":"eec6954816087f3349953b57bb383aad1739198ea170c995d1c6888989bd1e78","_meta":{"row":2785,"text_sha256":"15d7296ec5506ce45f273de8dca692fd7ee3b091af4b89c5d8ff60e3aa972fd4","row_sha256":"0434623eedfb30e0c938e9f5342d1ebc1849aa07e10549d9bfccffc0d06e8ed6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2785","group_id":"mmlu/test/2785","variant":"clean"}},"request_sha256":"a741e426a47b521f45d5e099c91f3c8c6a2d63a7b943f9ca7b82bcde64e9aa89","questions":[{"type":"choice","n_options":4,"row_len":113,"slot":112,"row_ids_sha256":"030a5bbea7cd6b60060c6be7d79cdd95daa60984e5ee95fcd51aa6dcac3b6db1","oracle":{"probs":[0.04673025711777114,0.9407360092420657,0.004599263161901594,0.007934470478261567],"argmax":"b","top2_margin":0.8940057521242946,"near_tie":false,"group_logz":{"73":-3.1005659103393555,"267":-10.05649185180664,"74":-0.09829521179199219,"293":-7.09521484375,"75":-5.419061660766602,"280":-12.449775695800781,"76":-4.873741149902344,"277":-11.749137878417969}},"graph":{"mac_fp16":{"p":[0.04695963084270544,0.9404691120459661,0.004616300194915986,0.00795495691641258],"max_abs_dp":0.0002668971960996247,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04624466246856736,0.9413520315277383,0.0045887441205847744,0.007814561883109675],"max_abs_dp":0.0006160222856725905,"argmax_equal":true}}}]},{"id":"tv4_009","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":9,"line_sha256":"99228b97f13d5bc63c18ca3970111a36b887f0baf745f164de45feb7ce5481db","_meta":{"row":5717,"text_sha256":"ae688490966c0852a0980ac7935c42ea98308efbfaca49a1cd5589064dba6cac","row_sha256":"05d282bcb34f55ca7b59ecafcb9a63c6d36d97a015a1c48979f80a7d6c7a4ea6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5717","group_id":"mmlu/test/5717","variant":"clean"}},"request_sha256":"7e620603c335f038b1d6f4727e01fe460c081a3f30ac9a65132eb3731ed27566","questions":[{"type":"choice","n_options":4,"row_len":339,"slot":338,"row_ids_sha256":"78d3d94eb9da13e629ee1318d520d01f10a552e4cdb1d4998d11be7cfead1672","oracle":{"probs":[0.002141769173251945,0.003960618627868153,0.9894595263461583,0.004438085852721668],"argmax":"c","top2_margin":0.9850214404934367,"near_tie":false,"group_logz":{"73":-6.170557022094727,"267":-13.635284423828125,"74":-5.555788993835449,"293":-12.87110710144043,"75":-0.035030364990234375,"280":-7.253857612609863,"76":-5.4419660568237305,"277":-12.499109268188477}},"graph":{"mac_fp16":{"p":[0.0021453089408164923,0.003962005899630948,0.9894484240290693,0.004444261130483219],"max_abs_dp":1.1102317089006775e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.002104083886741297,0.0038868869284991296,0.9896229336451171,0.004386095539642405],"max_abs_dp":0.00016340729895880646,"argmax_equal":true}}}]},{"id":"tv4_010","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":10,"line_sha256":"c7255ac19ceaddddadcd644bd539d3093a959d96bc90a689b56c313da9f5be74","_meta":{"row":8037,"text_sha256":"6eb96bdf834743053b122379db03e7f056ad3dc8ff3f5513b290c40620d2555f","row_sha256":"064a938d631d75f6147d58bb15659a7e76b9afd06b060eced5705d607167a4dd","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8037","group_id":"mmlu/test/8037","variant":"clean"}},"request_sha256":"744930de87e9c2b85585db39c72358f091149b05ac72e10a05800277ba0198a2","questions":[{"type":"choice","n_options":4,"row_len":91,"slot":90,"row_ids_sha256":"ae2e5d5730983d8cf6e2efc4ee69125a2e566795ef451fc62784741c0fd567be","oracle":{"probs":[0.3950156360736631,0.21403040304820314,0.1717696006688479,0.2191843602092859],"argmax":"a","top2_margin":0.1758312758643772,"near_tie":false,"group_logz":{"73":-1.0335254669189453,"267":-7.78507137298584,"74":-1.6463327407836914,"293":-7.951137542724609,"75":-1.8662967681884766,"280":-8.072230339050293,"76":-1.622537612915039,"277":-7.403687477111816}},"graph":{"mac_fp16":{"p":[0.395350471814418,0.21392744390845214,0.1718735540131782,0.21884853026395173],"max_abs_dp":0.000335829945334154,"argmax_equal":true},"iphone_int8mlp":{"p":[0.3952036819167722,0.21370677075008546,0.17378669439411012,0.21730285293903218],"max_abs_dp":0.0020170937252622367,"argmax_equal":true}}}]},{"id":"tv4_011","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":11,"line_sha256":"ad470c6a57936d5e202ba37caad6a69604d5aa423f97612b5f4eba926186d1e4","_meta":{"row":1047,"text_sha256":"2a0d61f454a3b0592fa563c96cd9d4ec0981bf856112ed46603214e9573eac02","row_sha256":"065c2a3f0ad371d06e17d82fe4bc8bfc5855bca2a04de9c1c9adb08d551a8639","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1047","group_id":"mmlu/test/1047","variant":"clean"}},"request_sha256":"147377385f65ec171a611dcd89720bb0838aeebea4fbd3118e6576b9ffb4afc4","questions":[{"type":"choice","n_options":4,"row_len":129,"slot":128,"row_ids_sha256":"e8479ff429c996d0c16abe3787bf391cee8c559b8075ea0124dc5ef0a3f36e1a","oracle":{"probs":[0.017206759373902786,0.042671618568002494,0.4342674385672051,0.5058541834908896],"argmax":"d","top2_margin":0.0715867449236845,"near_tie":false,"group_logz":{"73":-4.157414436340332,"267":-11.330297470092773,"74":-3.24918270111084,"293":-10.051506042480469,"75":-0.9290561676025391,"280":-7.73140811920166,"76":-0.7764682769775391,"277":-7.153942108154297}},"graph":{"mac_fp16":{"p":[0.017223033154964318,0.04268407415063492,0.43409960837505207,0.5059932843193488],"max_abs_dp":0.00016783019215305117,"argmax_equal":true},"iphone_int8mlp":{"p":[0.017019257920871188,0.04210160499271242,0.44240475495910264,0.49847438212731365],"max_abs_dp":0.00813731639189752,"argmax_equal":true}}}]},{"id":"tv4_012","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":12,"line_sha256":"40bbc2da9f39e86fe5336e3bd8036472357e6ed9a8a082be37f75ea777d1de0f","_meta":{"row":4518,"text_sha256":"72be257ffdd59cdd7a74367581cd0aa5d75169cacfe687ab8b9bfbacafc817d0","row_sha256":"06f9edd28b068873d96ccfdb1df2848b00d64e1fd317e529fc71ac37a979eb39","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4518","group_id":"mmlu/test/4518","variant":"clean"}},"request_sha256":"c36c93ea66a43ed9cd575b0c7bd75cb2bfa224f721f57dc3466f8c210d564e57","questions":[{"type":"choice","n_options":4,"row_len":102,"slot":101,"row_ids_sha256":"af25ae70fafb31037c43d3c809d98e5f436d0ad3f3eda50dc2a83d795ebaa483","oracle":{"probs":[0.024707737418621718,0.004971013354436823,0.021435093620079688,0.9488861556068617],"argmax":"d","top2_margin":0.92417841818824,"near_tie":false,"group_logz":{"73":-3.7350597381591797,"267":-11.694536209106445,"74":-5.338552474975586,"293":-13.109330177307129,"75":-3.8771467208862305,"280":-11.732912063598633,"76":-0.08688735961914062,"277":-7.78993034362793}},"graph":{"mac_fp16":{"p":[0.02463104463696939,0.004975223311234988,0.021410353398546658,0.948983378653249],"max_abs_dp":9.722304638726254e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.025162364469647886,0.004991895377591859,0.02198798021221962,0.9478577599405407],"max_abs_dp":0.0010283956663210692,"argmax_equal":true}}}]},{"id":"tv4_013","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":13,"line_sha256":"dcdb88cdb38f650686756813eb037728e3dfc1d1c204bd95a5af0536ff5fa86f","_meta":{"row":1492,"text_sha256":"c76022d4ee47dfc12b1e1ba7ff196c857d494ffbcd679e8b1d655f668fd72500","row_sha256":"070ed9de720c112fbd8b78c4e6dbbd24c1734d23867fb249de507295794e6cdf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1492","group_id":"mmlu/test/1492","variant":"clean"}},"request_sha256":"bf6beed52ae480916823e344e155f4dec74d8707f8e64003504593082cfbe3be","questions":[{"type":"choice","n_options":4,"row_len":183,"slot":182,"row_ids_sha256":"57494bf25ed2020555237b22171b3bd4a85c6b6060b7deb2ccc911d9537ee914","oracle":{"probs":[0.00890125988287996,0.010404245507866328,0.4850875290776364,0.4956069655316172],"argmax":"d","top2_margin":0.01051943645398079,"near_tie":true,"group_logz":{"73":-4.869255065917969,"267":-11.861104965209961,"74":-4.713233947753906,"293":-11.671843528747559,"75":-0.8711185455322266,"280":-7.464031219482422,"76":-0.8496646881103516,"277":-7.135828018188477}},"graph":{"mac_fp16":{"p":[0.008882935492610093,0.010400798423212744,0.4854876435849487,0.4952286224992285],"max_abs_dp":0.00040011450731231024,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009058762885649663,0.010429055487935343,0.4786150836026119,0.5018970980238031],"max_abs_dp":0.006472445475024513,"argmax_equal":true}}}]},{"id":"tv4_014","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":14,"line_sha256":"b0d027a49011258e7f0ad072081b1ab6411ff341bea9f10f7f423df31f962312","_meta":{"row":4063,"text_sha256":"5cc13884a1a177fa42853b29a986afb3d31c76715d41b3ef3263fd69b6438330","row_sha256":"0733d390eb549c8c29ff274b7f6c2ef73f6dcc0629dd53b1b19b34e43f31f116","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4063","group_id":"mmlu/test/4063","variant":"clean"}},"request_sha256":"9850c294f934eb170c98c6a154936a55ab3b8711f872dfa10a23865ea719d68a","questions":[{"type":"choice","n_options":4,"row_len":101,"slot":100,"row_ids_sha256":"1e50dfb1d894a329e28d13460687ba97f34cbd89fe9842b78761532e82363bbc","oracle":{"probs":[0.03644405719062458,0.02064910039018882,0.9346719346402901,0.008234907778896454],"argmax":"c","top2_margin":0.8982278774496655,"near_tie":false,"group_logz":{"73":-3.346424102783203,"267":-10.378089904785156,"74":-3.9145307540893555,"293":-10.746712684631348,"75":-0.10200691223144531,"280":-7.0923309326171875,"76":-4.833820343017578,"277":-11.458911895751953}},"graph":{"mac_fp16":{"p":[0.03639545381755646,0.020643162509288507,0.934718795807757,0.00824258786539815],"max_abs_dp":4.860337306811929e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03668781975039574,0.0206073210571306,0.9345361609660363,0.008168698226437222],"max_abs_dp":0.0002437625597711615,"argmax_equal":true}}}]},{"id":"tv4_015","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":15,"line_sha256":"e6fa25992cab53f1dac576717376a7e08b467dde58cafe18efea22dd5f8de21d","_meta":{"row":11310,"text_sha256":"08836e0191901e64734fbe0f9a786efaf3709406e58f7fa495078bcb610c53a9","row_sha256":"07cea382ce1e459ddb0a46898382abcb1762fd9b713a087f06aca048f0923b2a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11310","group_id":"mmlu/test/11310","variant":"clean"}},"request_sha256":"0b714407a2bdbf388b46ee41e8598d589cc1a49b7e93bf31d6ea537412007824","questions":[{"type":"choice","n_options":4,"row_len":194,"slot":193,"row_ids_sha256":"6931b938f789b74614ed579b1d0f1fe0640028146e54346a19c2958c2bf8a0b3","oracle":{"probs":[0.8597244206852224,0.02829369652529482,0.10236129274479301,0.009620590044689773],"argmax":"a","top2_margin":0.7573631279404294,"near_tie":false,"group_logz":{"73":-0.18915748596191406,"267":-8.242509841918945,"74":-3.603130340576172,"293":-10.679710388183594,"75":-2.3172607421875,"280":-9.387063980102539,"76":-4.681863784790039,"277":-11.692798614501953}},"graph":{"mac_fp16":{"p":[0.8599115699761357,0.028185069434286815,0.10229005194530605,0.009613308644271377],"max_abs_dp":0.00018714929091334653,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8570055255202019,0.028306616318285367,0.1047158960285978,0.009971962132914873],"max_abs_dp":0.0027188951650204407,"argmax_equal":true}}}]},{"id":"tv4_016","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":16,"line_sha256":"5eb8cd7d0f9291ad40dfb323c2c3dbc75dc5712eb1117a38bac38fc8f81d73d1","_meta":{"row":6610,"text_sha256":"230af30dc37acdac5cace354b74414e19f977465f005f72fb6f13ce1c83fc01f","row_sha256":"080972877925a0e9071a647836ab7bc1b9f6fb860fd37fd74b9d115eb1a8a150","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6610","group_id":"mmlu/test/6610","variant":"clean"}},"request_sha256":"0b1892a2ef9c0703fe8c36cfc0949021363c58927ee5f7ca4d63c131b09e5ac6","questions":[{"type":"choice","n_options":4,"row_len":96,"slot":95,"row_ids_sha256":"d111bda40abf523d91496f941130d5d2708b9331923af720d68e824e9dba9d9d","oracle":{"probs":[0.010458884930136813,0.01102273770998335,0.04916102984218997,0.9293573475176898],"argmax":"d","top2_margin":0.8801963176754999,"near_tie":false,"group_logz":{"73":-4.609783172607422,"267":-12.57217788696289,"74":-4.55727481842041,"293":-12.209951400756836,"75":-3.0621337890625,"280":-10.602096557617188,"76":-0.12274169921875,"277":-7.459815979003906}},"graph":{"mac_fp16":{"p":[0.010452355244417972,0.011062981156636682,0.049084798064292806,0.9293998655346526],"max_abs_dp":7.623177789716634e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010236990537425522,0.010847904805709577,0.04911296757450091,0.9298021370823639],"max_abs_dp":0.00044478956467408626,"argmax_equal":true}}}]},{"id":"tv4_017","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":17,"line_sha256":"764e7d81d8e2293eb8261c9cfd55d4c0049e71b0c664d43957458d40e06f9479","_meta":{"row":9462,"text_sha256":"f658cc8e1232787da72bf6e03aa3f71f61d9b2e1cbd9cb34e78e29b12865dcfb","row_sha256":"0813758be42e718265867e9b104410830958994506ff946e678204da674ab60e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9462","group_id":"mmlu/test/9462","variant":"clean"}},"request_sha256":"cf3500f63bec8fe54efde1adcbb29c5fe02f22553224732c5b9ab325dc53e2d8","questions":[{"type":"choice","n_options":4,"row_len":94,"slot":93,"row_ids_sha256":"d28ec97afe622f3a1b08e41a7118e032f7356eb489f75b1d08df0c2fdb492fe2","oracle":{"probs":[0.004075307533010569,0.01463207416601156,0.9704851081274135,0.010807510173564306],"argmax":"c","top2_margin":0.955853033961402,"near_tie":false,"group_logz":{"73":-5.559298515319824,"267":-12.54492473602295,"74":-4.281028747558594,"293":-11.066184997558594,"75":-0.08644866943359375,"280":-7.151721954345703,"76":-4.584003448486328,"277":-11.203596115112305}},"graph":{"mac_fp16":{"p":[0.004066542322866343,0.014593233679943127,0.9705528225227483,0.010787401474442137],"max_abs_dp":6.771439533481693e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0040273202279488855,0.014460854398386723,0.9706701949150486,0.010841630458615706],"max_abs_dp":0.00018508678763506925,"argmax_equal":true}}}]},{"id":"tv4_018","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":18,"line_sha256":"9fdf49f65e1bbc1bfe397e1744d67d7dfad6cb9b53309bd1bd9cd70e4a36ce29","_meta":{"row":5208,"text_sha256":"31a70389d9e44cad3395616eac1504fa0ce169118cd1d71dce3763e10a62c45c","row_sha256":"08cdef9dcb2506c626af30d1df6d606b616e510de7a393b065c19acf5da24497","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5208","group_id":"mmlu/test/5208","variant":"clean"}},"request_sha256":"d7a16c62dd57cc768bcc23b3870550458ee8c5cdc33e374554aabd2d2f08503e","questions":[{"type":"choice","n_options":4,"row_len":84,"slot":83,"row_ids_sha256":"8550891e7ba6775fe10ed29159633397da410a843d0ed937283a13751198c9a9","oracle":{"probs":[0.0013989610588250167,0.004795384617389032,0.9897870401400202,0.004018614183765889],"argmax":"c","top2_margin":0.9849916555226311,"near_tie":false,"group_logz":{"73":-6.621480941772461,"267":-14.166420936584473,"74":-5.389556884765625,"293":-13.151130676269531,"75":-0.05972099304199219,"280":-7.945058822631836,"76":-5.5662736892700195,"277":-12.954267501831055}},"graph":{"mac_fp16":{"p":[0.0013990570237260788,0.0047933396012887295,0.9897930850337561,0.004014518341229169],"max_abs_dp":6.0448937359014465e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0013967661926885247,0.004770606129199828,0.9897949697859496,0.004037657892162058],"max_abs_dp":2.4778488189203945e-05,"argmax_equal":true}}}]},{"id":"tv4_019","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":19,"line_sha256":"a1965ab273faa7dfa644f1847e3bf6f1e2043182acd27c8ad8e6974bb53b1c64","_meta":{"row":9213,"text_sha256":"ad7d058864a902202cc16c98a71221e2eee376a5cb319653c0581de66893e99f","row_sha256":"0ad24336bd4a190abb0870d37ed56901090b8c288422d98b2b8dae935e0a5951","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9213","group_id":"mmlu/test/9213","variant":"clean"}},"request_sha256":"084d5cc46335adc0d0100b0ba21fe3ea2d2e657c8bbdf95278c8d6bf40fe157a","questions":[{"type":"choice","n_options":4,"row_len":150,"slot":149,"row_ids_sha256":"2b8b0a340d3457406e8920cd6887d8f37d078ffb0d0dfad6a7e3f0b4bb1b58e1","oracle":{"probs":[0.12787557548486514,0.6458938248577131,0.1389331744093864,0.0872974252480353],"argmax":"b","top2_margin":0.5069606504483266,"near_tie":false,"group_logz":{"73":-2.1059398651123047,"267":-9.383256912231445,"74":-0.4863624572753906,"293":-7.4547624588012695,"75":-2.0230045318603516,"280":-8.769452095031738,"76":-2.4876766204833984,"277":-9.201410293579102}},"graph":{"mac_fp16":{"p":[0.12663986123079923,0.6469416062205012,0.13883232240614019,0.08758621014255939],"max_abs_dp":0.001235714254065906,"argmax_equal":true},"iphone_int8mlp":{"p":[0.12037868787061767,0.6505263470317927,0.14136613565208217,0.08772882944550744],"max_abs_dp":0.007496887614247466,"argmax_equal":true}}}]},{"id":"tv4_020","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":20,"line_sha256":"34d5162147130e4d740b5f5ffdc6f097a139b4cb782323b77a9d2b4d79244487","_meta":{"row":12914,"text_sha256":"5afdeb67f2af871c752771dd9425be2bc6967bcb203b35de05f057220a27789a","row_sha256":"0ada8ef3402be468e4a35329742fa3232f1ff88aac6612afb0d092090c50e3f3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12914","group_id":"mmlu/test/12914","variant":"clean"}},"request_sha256":"1a389c9152805ca3c3db015478389c23a8d7a884b03a9088ae835d6a321a5bda","questions":[{"type":"choice","n_options":4,"row_len":98,"slot":97,"row_ids_sha256":"7b60626782777b8b89ecd773577846d7f3828dcfa6116b21b2dcef338e8e40fb","oracle":{"probs":[0.01594286306447771,0.07441062178315423,0.670101940908648,0.23954457424372],"argmax":"c","top2_margin":0.43055736666492805,"near_tie":false,"group_logz":{"73":-4.200872421264648,"267":-10.837442398071289,"74":-2.660284996032715,"293":-8.946913719177246,"75":-0.46245384216308594,"280":-6.881757736206055,"76":-1.4911441802978516,"277":-7.312372207641602}},"graph":{"mac_fp16":{"p":[0.015942007294819833,0.07457710927375218,0.6689432950831161,0.24053758834831188],"max_abs_dp":0.0011586458255319343,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015090248100480843,0.07282037874478363,0.6761365454043587,0.23595282775037674],"max_abs_dp":0.006034604495710694,"argmax_equal":true}}}]},{"id":"tv4_021","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":21,"line_sha256":"24e96525cc66ce161c1854342a4600204e85d284d29eb4b16b1dd916b98b1b06","_meta":{"row":724,"text_sha256":"e38af88bca0cd26b62120a5e330698e74c26820357a06f0fd8da19c39a867706","row_sha256":"0adf991fd97e7801b25429b4babec11df58b8d5dc9f6e81989c1ae1694338f59","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/724","group_id":"mmlu/test/724","variant":"clean"}},"request_sha256":"adefae50a3e1b776137fc9371bcb5991caad78ad26408222819b6fcaba44090b","questions":[{"type":"choice","n_options":4,"row_len":78,"slot":77,"row_ids_sha256":"ce57e71fba214bdae22bee481067736d9d1e989f362b0feb6aee1b18b5c95df0","oracle":{"probs":[0.21872710982628843,0.12115565114684329,0.48629067680587135,0.17382656222099688],"argmax":"c","top2_margin":0.26756356697958295,"near_tie":false,"group_logz":{"73":-1.6253957748413086,"267":-8.012173652648926,"74":-2.216144561767578,"293":-8.357555389404297,"75":-0.8264141082763672,"280":-7.008480072021484,"76":-1.8551626205444336,"277":-7.504066467285156}},"graph":{"mac_fp16":{"p":[0.21914105294359032,0.12107808861706139,0.48599692317396376,0.17378393526538458],"max_abs_dp":0.0004139431173018904,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2179683098990448,0.12111939339574523,0.4885219657376019,0.1723903309676081],"max_abs_dp":0.0022312889317305795,"argmax_equal":true}}}]},{"id":"tv4_022","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":22,"line_sha256":"e96062586430afadc9c1e839a584945437762330cc4958483cc498b1d793653d","_meta":{"row":4454,"text_sha256":"28649f1c96e825de7daa47754c989b906cd75b4644525f722544a5d298b8468c","row_sha256":"0b7df30f69dd9f462fbac0d3c7662a9b4b38a59a4d4f11310fc1969525ae8ae8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4454","group_id":"mmlu/test/4454","variant":"clean"}},"request_sha256":"03a62145d2d4a21636ad8927e164d6aeb92f798f6aecd996afcec7dc9eea0c04","questions":[{"type":"choice","n_options":4,"row_len":154,"slot":153,"row_ids_sha256":"1edf968e9fb33bd684e0513d62e62e03829ac47466ceb5ab0913966a88832c04","oracle":{"probs":[0.2688995519450486,0.3860917935447508,0.28351604818818343,0.06149260632201721],"argmax":"b","top2_margin":0.10257574535656738,"near_tie":false,"group_logz":{"73":-1.4286909103393555,"267":-7.648301124572754,"74":-1.0669536590576172,"293":-6.960615158081055,"75":-1.3757600784301758,"280":-7.22088623046875,"76":-2.904111862182617,"277":-8.496192932128906}},"graph":{"mac_fp16":{"p":[0.26874271385380377,0.38589026994546943,0.283853471879819,0.06151354432090779],"max_abs_dp":0.0003374236916355855,"argmax_equal":true},"iphone_int8mlp":{"p":[0.27186510921036905,0.3806789201076097,0.2859107431594915,0.06154522752252969],"max_abs_dp":0.005412873437141141,"argmax_equal":true}}}]},{"id":"tv4_023","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":23,"line_sha256":"a6b39e74e00a0b153bae5903309090d8c3fb22d15ad193eae6008ca370926641","_meta":{"row":8228,"text_sha256":"5eda8abb1505371de513af404a66c64321f24733a74fe9d7997751869896daca","row_sha256":"0bec9e299974e1bbd0299dc5b542170ed2e7f0c823d095e8c95bd34dd66c5ceb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8228","group_id":"mmlu/test/8228","variant":"clean"}},"request_sha256":"f1f39f9f7cb716f4154c480b16a8961abba4ea09ddb5de26b87e7fc0f4bcbb3e","questions":[{"type":"choice","n_options":4,"row_len":107,"slot":106,"row_ids_sha256":"103850a3acef3ed52964c270d49443007ea84c286400dc6da4772cec2c217764","oracle":{"probs":[0.5976530120662841,0.1681752977548392,0.10709913392004998,0.12707255625882685],"argmax":"a","top2_margin":0.4294777143114449,"near_tie":false,"group_logz":{"73":-0.5993461608886719,"267":-7.311859130859375,"74":-1.867349624633789,"293":-7.9560041427612305,"75":-2.318601608276367,"280":-8.43446159362793,"76":-2.1475982666015625,"277":-7.817018508911133}},"graph":{"mac_fp16":{"p":[0.5977788921309886,0.16798961571987397,0.10705678403609922,0.12717470811303824],"max_abs_dp":0.00018568203496521773,"argmax_equal":true},"iphone_int8mlp":{"p":[0.595148518791952,0.16829167294045347,0.10834171099185916,0.1282180972757354],"max_abs_dp":0.002504493274332109,"argmax_equal":true}}}]},{"id":"tv4_024","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":24,"line_sha256":"debbb575343af7e4b0b0cc5bd8822991f487607194abcb622e85cb917e75e4cb","_meta":{"row":13238,"text_sha256":"5c97128bd214131f1ddac5218553879a4bd91db54f482e32c57da5913e6d0f09","row_sha256":"0cace9a6eff08851fc6f07a62a24fcb1f1c06c96dceebc71c4dc67fec7a6bf10","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13238","group_id":"mmlu/test/13238","variant":"clean"}},"request_sha256":"9e2b3caaa5a10a559b5bd3411d0acf208c671d9f01fca6cbc7d2d5e99c15caf9","questions":[{"type":"choice","n_options":4,"row_len":122,"slot":121,"row_ids_sha256":"741a03931385c88cf98a0a8a9190e1bfcef33217fdee95c19855912720ba981d","oracle":{"probs":[0.09492758442286572,0.10249075307371831,0.03643979904317428,0.7661418634602417],"argmax":"d","top2_margin":0.6636511103865234,"near_tie":false,"group_logz":{"73":-2.674345016479492,"267":-9.667135238647461,"74":-2.597686767578125,"293":-9.071474075317383,"75":-3.6317977905273438,"280":-10.621732711791992,"76":-0.5860919952392578,"277":-7.132448196411133}},"graph":{"mac_fp16":{"p":[0.09529000507033476,0.10263123279884671,0.03646841888166405,0.7656103432491544],"max_abs_dp":0.0005315202110872663,"argmax_equal":true},"iphone_int8mlp":{"p":[0.09553981485992462,0.10280522176077696,0.03644130061463943,0.765213662764659],"max_abs_dp":0.0009282006955827482,"argmax_equal":true}}}]},{"id":"tv4_025","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":25,"line_sha256":"590d744bb4f3a367f1c5a9053a7b934ff16835b17893086a5d58597109a146f4","_meta":{"row":11404,"text_sha256":"c27531d6a1741c33d2f26d6e9fddc7936f384064c89cde64dcc2ccdb8481cf78","row_sha256":"0cd28fb4615e5cff84b7f6c042098abed66dd1a265c7c45364224dbff09e2c84","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11404","group_id":"mmlu/test/11404","variant":"clean"}},"request_sha256":"4b4969c4f73ce819be38898bedb4e4ecc1d54d91fb35fb1e9f730aeb51fc64bc","questions":[{"type":"choice","n_options":4,"row_len":193,"slot":192,"row_ids_sha256":"93cf17390e9314e21899eb8a76e7ea5b3e9123379371d6964a8459c9ec90fc50","oracle":{"probs":[0.28497221451141846,0.34106805890362085,0.29378808730374456,0.08017163928121611],"argmax":"b","top2_margin":0.04727997159987629,"near_tie":false,"group_logz":{"73":-1.3443269729614258,"267":-8.088127136230469,"74":-1.1646366119384766,"293":-7.549190521240234,"75":-1.3138599395751953,"280":-7.63624382019043,"76":-2.612548828125,"277":-8.706993103027344}},"graph":{"mac_fp16":{"p":[0.285029862967323,0.34132915060450775,0.29358044999633603,0.08006053643183306],"max_abs_dp":0.00026109170088689515,"argmax_equal":true},"iphone_int8mlp":{"p":[0.28373736082024537,0.3440337361444822,0.2924360687564247,0.0797928342788477],"max_abs_dp":0.0029656772408613308,"argmax_equal":true}}}]},{"id":"tv4_026","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":26,"line_sha256":"bbe8c56a361539948fc980ad5f65ed6410c6ce97da8f7d1d80085a7859f15f50","_meta":{"row":6593,"text_sha256":"fa18d8eb79c698b5e014bb3484bc1ac5132f0d94874455a0cead140a36871b4b","row_sha256":"0d02b58d02f3e95f74b821635dfdddc34a8b2ace30bdbc508f05f915cd44bfb3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6593","group_id":"mmlu/test/6593","variant":"clean"}},"request_sha256":"3b213416bb680d5b7f27a24f1d2da77a531a1040fdb5fe273ae96df688831da4","questions":[{"type":"choice","n_options":4,"row_len":109,"slot":108,"row_ids_sha256":"527c3dc32618d04d33733c6176c30867ad4daced250a0b651b5155ab30041d6d","oracle":{"probs":[0.001831442883832025,0.004111961010586428,0.9897392601898186,0.004317335915762872],"argmax":"c","top2_margin":0.9854219242740557,"near_tie":false,"group_logz":{"73":-6.340021133422852,"267":-13.615255355834961,"74":-5.531225204467773,"293":-12.922933578491211,"75":-0.0476837158203125,"280":-7.206822395324707,"76":-5.482486724853516,"277":-12.408442497253418}},"graph":{"mac_fp16":{"p":[0.0018302240048755668,0.004111137332440252,0.9897421056865721,0.004316532976111985],"max_abs_dp":2.8454967534452535e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0018476523725593926,0.004159554057132299,0.9896086602001829,0.004384133370125321],"max_abs_dp":0.00013059998963571573,"argmax_equal":true}}}]},{"id":"tv4_027","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":27,"line_sha256":"d3b4c0e91e7a2e3af4bf3c2ee7f019c026e2e47782753822e7e00ae9021900d8","_meta":{"row":3841,"text_sha256":"080abd0a3265a45f0136d1fa3f9b3eb724571503c178f9c082ca331d2550ee66","row_sha256":"0d43bd015715be6a656242ba73f2e7cb6e82355ad11e9c12914a2d9052eccd2b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3841","group_id":"mmlu/test/3841","variant":"clean"}},"request_sha256":"3febe7b43f82e302dcbf13595914cdf3efc3a473eb858fe206cc4c61b7930a80","questions":[{"type":"choice","n_options":4,"row_len":97,"slot":96,"row_ids_sha256":"14aa78383375382f7b362795823679616735b3c046339c4b6b287082e00cf604","oracle":{"probs":[0.005830614709838313,0.4115559677344701,0.19872073751312647,0.38389268004256516],"argmax":"b","top2_margin":0.02766328769190496,"near_tie":false,"group_logz":{"73":-5.316380500793457,"267":-11.718733787536621,"74":-1.0595579147338867,"293":-6.989295959472656,"75":-1.787602424621582,"280":-7.719382286071777,"76":-1.1291399002075195,"277":-6.899080276489258}},"graph":{"mac_fp16":{"p":[0.0058304706993581356,0.41181322893537,0.1993175295999239,0.3830387707653479],"max_abs_dp":0.0008539092772172352,"argmax_equal":true},"iphone_int8mlp":{"p":[0.005746275631064914,0.4174092851902978,0.201038726669371,0.37580571250926637],"max_abs_dp":0.008086967533298794,"argmax_equal":true}}}]},{"id":"tv4_028","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":28,"line_sha256":"245be49c410a548c03b05a78b3e8edcbaeace42ef8247d90f591e69f036f18e9","_meta":{"row":11705,"text_sha256":"73ee9198bb27004a774c7defd4636b24f6931b85320a3fb826f31dbd7d2bbe02","row_sha256":"0eb66519306fa8a8ecc1873b57d6bbf252d623dc5481e64207120b290c2359aa","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11705","group_id":"mmlu/test/11705","variant":"clean"}},"request_sha256":"459a1ad68768f86a8800beeeb43d958c43ab375f7642fbe6b1caa60c4f89fdf9","questions":[{"type":"choice","n_options":4,"row_len":316,"slot":315,"row_ids_sha256":"5efff8fb47c32b66ab1b7d74c5f53d0750430f99e08f6a2d0389528585ff4bcc","oracle":{"probs":[0.18193142570930382,0.3319019104340725,0.41329975281632253,0.07286691104030106],"argmax":"c","top2_margin":0.08139784238225001,"near_tie":false,"group_logz":{"73":-1.7831478118896484,"267":-8.597251892089844,"74":-1.1819381713867188,"293":-7.603697776794434,"75":-0.9626045227050781,"280":-7.368259429931641,"76":-2.6981430053710938,"277":-8.814598083496094}},"graph":{"mac_fp16":{"p":[0.18183010235157068,0.33212764755424,0.41285008210048957,0.07319216799369968],"max_abs_dp":0.0004496707158329616,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1797329831022597,0.3317404843501684,0.41575472537580166,0.07277180717177036],"max_abs_dp":0.0024549725594791294,"argmax_equal":true}}}]},{"id":"tv4_029","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":29,"line_sha256":"18fe39d4f3dac58636dadfce7d9023174a4a6df8c2e70e806a124f4112ff729e","_meta":{"row":2862,"text_sha256":"d429e7cd3c0676ad51fe99eb08872dabbb5f9f457cfed19084009276fcfb0ab5","row_sha256":"0f055897f863481cb6f33301380858951cd7a3fad82fec2a605fcfe4551d0596","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2862","group_id":"mmlu/test/2862","variant":"clean"}},"request_sha256":"1c2230f92314b8630797f507c969c1d4b6fc71bc78842f66a3f8f382ac890691","questions":[{"type":"choice","n_options":4,"row_len":81,"slot":80,"row_ids_sha256":"ae816bc3056919601ad004b7019a813eb9645f87a44d38d82dc6a0c9115c24fc","oracle":{"probs":[0.8448326193799207,0.040041848300715374,0.0468594916423815,0.06826604067698236],"argmax":"a","top2_margin":0.7765665787029383,"near_tie":false,"group_logz":{"73":-0.2299938201904297,"267":-7.642095565795898,"74":-3.279207229614258,"293":-9.882969856262207,"75":-3.121978759765625,"280":-9.791876792907715,"76":-2.7457199096679688,"277":-8.711938858032227}},"graph":{"mac_fp16":{"p":[0.844354946578135,0.04039012091764989,0.04690792407641241,0.0683470084278026],"max_abs_dp":0.00047767280178567795,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8424862325129933,0.0407213691419297,0.04748591007520981,0.06930648826986721],"max_abs_dp":0.0023463868669274524,"argmax_equal":true}}}]},{"id":"tv4_030","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":30,"line_sha256":"7af97b0ac0cde7df1c315b6e0f142d2b0cad82cfbf556916814d2d50e21ac5b5","_meta":{"row":9772,"text_sha256":"18cbaea8b014f81f1a3cae404be60d6cde12ff38a648f3ec08b8c51464b57137","row_sha256":"0f24142b228b4133d31115ae881abcd37fd4efd0afe4f91de60e11fb598211d1","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9772","group_id":"mmlu/test/9772","variant":"clean"}},"request_sha256":"8f96b871c1bd5a21833145e6f11fa25846e0c5bf82775f0f04648fc9950401de","questions":[{"type":"choice","n_options":4,"row_len":93,"slot":92,"row_ids_sha256":"f3df8f28694a737d9adcb72b473860a8253067599d668c890f691c86ade83927","oracle":{"probs":[0.008615290926598717,0.025294716095697056,0.9498871657187538,0.01620282725895056],"argmax":"c","top2_margin":0.9245924496230566,"near_tie":false,"group_logz":{"73":-4.798503875732422,"267":-11.753108978271484,"74":-3.721446990966797,"293":-10.428232192993164,"75":-0.09569931030273438,"280":-6.866693496704102,"76":-4.16685676574707,"277":-10.609167098999023}},"graph":{"mac_fp16":{"p":[0.008611997456101098,0.025260956161074544,0.9499746619529774,0.01615238442984693],"max_abs_dp":8.749623422366071e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.008537916648263878,0.02497135520592277,0.9503735749254077,0.01611715322040567],"max_abs_dp":0.0004864092066539927,"argmax_equal":true}}}]},{"id":"tv4_031","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":31,"line_sha256":"9151dd70b6903d8570a8a095c0a3a3122264a4d700ad345ff1a1f27f8be3c2bf","_meta":{"row":5094,"text_sha256":"2bafd22f31a9fff5917ac28f483d154f2a964ff090176b1ba0b4c9434403801c","row_sha256":"0f69624a2c67dc837033e8037a6c0e7f8748b6c94ed6ef265ec82be45c5cb8e7","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5094","group_id":"mmlu/test/5094","variant":"clean"}},"request_sha256":"78c82595491bca2caaf1359854adec31441e46b21d154439121696dc0cea8642","questions":[{"type":"choice","n_options":4,"row_len":86,"slot":85,"row_ids_sha256":"334a236c8ae4ad67af77aab421eaaafabf8b3d310939ea6c030688afc096a810","oracle":{"probs":[0.5571111047187172,0.43203950226959453,0.006887825990594722,0.003961567021093596],"argmax":"a","top2_margin":0.12507160244912263,"near_tie":false,"group_logz":{"73":-0.6497402191162109,"267":-7.889760971069336,"74":-0.9039878845214844,"293":-7.55183219909668,"75":-5.042749404907227,"280":-11.896690368652344,"76":-5.595865249633789,"277":-12.515955924987793}},"graph":{"mac_fp16":{"p":[0.554417196514493,0.43474160885944557,0.006884417451257974,0.003956777174803441],"max_abs_dp":0.002702106589851039,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5588751052765261,0.430271114047348,0.006914062789385197,0.003939717886740669],"max_abs_dp":0.0017683882222465086,"argmax_equal":true}}}]},{"id":"tv4_032","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":32,"line_sha256":"4b326f5221f5040a86c383896719fc43d6668931e4d4ef791bfc98184ba4fea2","_meta":{"row":10172,"text_sha256":"e063a35bc796e0c4eb85c16dc5c8b1fbb82701c11168fb320be646ba28476bfc","row_sha256":"0f9a5d1f0ef7225fd0f2bd67d68d0ebc47bb8ce1d98e3a082d23ca27d4cd5182","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10172","group_id":"mmlu/test/10172","variant":"clean"}},"request_sha256":"f57d1f19c21f024ab369fba99c8e35928d61c34d2665f1cd95f1cec528446b16","questions":[{"type":"choice","n_options":4,"row_len":148,"slot":147,"row_ids_sha256":"bdf700dbd9bce56a0c178b2f4a2fde5c5a222cbc585d6fde0898db1c63889fb0","oracle":{"probs":[0.0034189625598116126,0.008564758270588068,0.02770494375366125,0.960311335415939],"argmax":"d","top2_margin":0.9326063916622778,"near_tie":false,"group_logz":{"73":-6.047903060913086,"267":-13.166218757629395,"74":-5.129584312438965,"293":-11.977442741394043,"75":-3.955629348754883,"280":-10.723674774169922,"76":-0.40998268127441406,"277":-7.247200012207031}},"graph":{"mac_fp16":{"p":[0.003425941597154515,0.008590816794790557,0.027778149915869556,0.9602050916921853],"max_abs_dp":0.00010624372375378499,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0033573936350935535,0.008440589250473743,0.026510886863198926,0.9616911302512338],"max_abs_dp":0.001379794835294712,"argmax_equal":true}}}]},{"id":"tv4_033","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":33,"line_sha256":"983164def8eb976b604ae8e11dd535f502e204e8b7f21baaf08bef38422109ac","_meta":{"row":3349,"text_sha256":"2727ef590d6a7b74fc042faafcf9c9570b374400f88c35e1069784efabaa7ef7","row_sha256":"111ff188a463868073a49590f1be6da85ca9649c068f5dd639fcce931036c5c8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3349","group_id":"mmlu/test/3349","variant":"clean"}},"request_sha256":"6a57b2307f77b859577b2fee0613c82944ddb441e8ffdbdd122f98a8bc597ddf","questions":[{"type":"choice","n_options":4,"row_len":311,"slot":310,"row_ids_sha256":"53408e1317e9cabeed5588a569743b45f71a3ea2e21ab93e64361c832a1aea5d","oracle":{"probs":[0.003547173542149265,0.0056448038624503805,0.014646059885091529,0.9761619627103089],"argmax":"d","top2_margin":0.9615159028252174,"near_tie":false,"group_logz":{"73":-5.689090728759766,"267":-13.16041374206543,"74":-5.224506378173828,"293":-12.521767616271973,"75":-4.27107048034668,"280":-11.571535110473633,"76":-0.07161331176757812,"277":-7.435765266418457}},"graph":{"mac_fp16":{"p":[0.003551741529768449,0.0056471692000324295,0.01466309836665114,0.9761379909035479],"max_abs_dp":2.3971806760947345e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0034789669617435377,0.005585926043509287,0.014632745508129976,0.9763023614866173],"max_abs_dp":0.000140398776308448,"argmax_equal":true}}}]},{"id":"tv4_034","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":34,"line_sha256":"3f9a4d74fc7cd14b5a55c3f492d899e07d17e3f788c3d32cbfe6f67d3ad1b725","_meta":{"row":5191,"text_sha256":"526fc9864f083866076d59ec085a5840d2dc3a51de7d6140a37f0b06b082b8c2","row_sha256":"1176e707d52212efe4a01cc0c680f05ffac10f895743a6f755297e64ab15f122","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5191","group_id":"mmlu/test/5191","variant":"clean"}},"request_sha256":"73ea947c427db8fbda8d65c7e190b3df154863edb7266944d75b60de65fec59b","questions":[{"type":"choice","n_options":4,"row_len":94,"slot":93,"row_ids_sha256":"53ad4b3a6eda0ef8b246652d5701a832c620e63b62a3a36fa3215a33acc1f4df","oracle":{"probs":[0.08339656560285515,0.12523427038551943,0.2690254469902836,0.5223437170213419],"argmax":"d","top2_margin":0.2533182700310583,"near_tie":false,"group_logz":{"73":-2.61541748046875,"267":-9.029510498046875,"74":-2.20883846282959,"293":-8.378434181213379,"75":-1.444218635559082,"280":-7.543949127197266,"76":-0.7806987762451172,"277":-6.520784378051758}},"graph":{"mac_fp16":{"p":[0.08337395616484372,0.12549980338945926,0.2690410165756689,0.5220852238700281],"max_abs_dp":0.0002655330039398329,"argmax_equal":true},"iphone_int8mlp":{"p":[0.08244065135034599,0.12408618336112658,0.2750790374062813,0.5183941278822461],"max_abs_dp":0.006053590415997734,"argmax_equal":true}}}]},{"id":"tv4_035","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":35,"line_sha256":"be29f5314f080acdb814e8b7c9d28680b173f9bf365a72b0376940dd1ed6c928","_meta":{"row":1808,"text_sha256":"f770a51b608eda349ee8aa4125bde15c974a7c110b05eec9df0f7392b935225a","row_sha256":"123bf64fbafcd2f6fec39cf01f12d33a9bf995d806e619aa81bc351e8915a2af","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1808","group_id":"mmlu/test/1808","variant":"clean"}},"request_sha256":"3f30a6d69c29eb1d4de5166637e7f2f04bb9fbd863b7bd27ffd47ca4e506b633","questions":[{"type":"choice","n_options":4,"row_len":126,"slot":125,"row_ids_sha256":"d22b57571a1014f8f6cbd8e5549b403bedb384f86f31006c6bb6d237081b3913","oracle":{"probs":[0.36344413131872094,0.02934184016548674,0.5971996555679052,0.010014372947887065],"argmax":"c","top2_margin":0.23375552424918428,"near_tie":false,"group_logz":{"73":-1.102386474609375,"267":-7.092825889587402,"74":-3.618997573852539,"293":-9.442693710327148,"75":-0.6057605743408203,"280":-6.253911018371582,"76":-4.693990707397461,"277":-10.307473182678223}},"graph":{"mac_fp16":{"p":[0.36394044590173313,0.029321112423166974,0.596710501581347,0.010027940093752944],"max_abs_dp":0.0004963145830121918,"argmax_equal":true},"iphone_int8mlp":{"p":[0.3643729252972557,0.028999253717475245,0.5966229250435635,0.010004895941705486],"max_abs_dp":0.0009287939785347366,"argmax_equal":true}}}]},{"id":"tv4_036","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":36,"line_sha256":"19bd2669d53da414a2e36b4ec9c7366252ee730b90dd29211cb8b9da8a0eb33a","_meta":{"row":10509,"text_sha256":"f56bf336aa556755eac37356eee98ec47feea09017a3d173deca0090a0671730","row_sha256":"14383703de5b130b888fbfa3e419839116b75a84137a6acc94b9f9da1a536863","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10509","group_id":"mmlu/test/10509","variant":"clean"}},"request_sha256":"048257fb360722efb5c13d172fd7f2388b162106b4e566fce44d26792e44b552","questions":[{"type":"choice","n_options":4,"row_len":156,"slot":155,"row_ids_sha256":"67f9bbb87c7428b9ed95df7989ced207ec66151674c284d61b864eb140588677","oracle":{"probs":[0.15922201785940968,0.4434152465602214,0.3221198698900655,0.07524286569030339],"argmax":"b","top2_margin":0.12129537667015589,"near_tie":false,"group_logz":{"73":-1.9502334594726562,"267":-8.622849464416504,"74":-0.9260263442993164,"293":-7.419745445251465,"75":-1.2456092834472656,"280":-7.591779708862305,"76":-2.6998119354248047,"277":-8.69839096069336}},"graph":{"mac_fp16":{"p":[0.15911346217510006,0.4433204586324602,0.32233870135006665,0.07522737784237297],"max_abs_dp":0.0002188314600011232,"argmax_equal":true},"iphone_int8mlp":{"p":[0.15899162473376655,0.4411332942197072,0.324531739334676,0.07534334171185024],"max_abs_dp":0.0024118694446104705,"argmax_equal":true}}}]},{"id":"tv4_037","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":37,"line_sha256":"9479ee8318444cb983c72daa6d782387464eeb918fa38914d08b11f0578f76e7","_meta":{"row":13713,"text_sha256":"5f54dd5414d94f4e5d9b41b3446a051b265c3b6c54c95b2e5ad393b9186d495c","row_sha256":"14910a65dfdb4ef6cf2108d5ff4522050ab8408c532b968aae2b7de7aa64727b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13713","group_id":"mmlu/test/13713","variant":"clean"}},"request_sha256":"97b63de843ad65fb506182e80041aa94bb64c864426c11ccb64e807faccd029b","questions":[{"type":"choice","n_options":4,"row_len":83,"slot":82,"row_ids_sha256":"2cf4d70209f4db9747d99abe228ead664f80d058c5be45b39c7198276d22b8b5","oracle":{"probs":[0.015269160184701735,0.0730975646157383,0.7913774167585281,0.12025585844103184],"argmax":"c","top2_margin":0.6711215583174963,"near_tie":false,"group_logz":{"73":-4.268080711364746,"267":-10.995094299316406,"74":-2.702120780944824,"293":-9.194034576416016,"75":-0.3201408386230469,"280":-7.04267692565918,"76":-2.204294204711914,"277":-8.366110801696777}},"graph":{"mac_fp16":{"p":[0.015286384904964562,0.07314154485736298,0.7912667383754971,0.1203053318621754],"max_abs_dp":0.0001106783830310265,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015255767555297475,0.07294326128831027,0.7922760554495217,0.11952491570687052],"max_abs_dp":0.0008986386909936028,"argmax_equal":true}}}]},{"id":"tv4_038","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":38,"line_sha256":"3fdff0512549e464e5a1431c92838b0043e0a6a1c86b0c5f21a828c8cc679a8c","_meta":{"row":1807,"text_sha256":"63104fec8e11568f820894e0eb75b8c1e47b4bf437e5d78f65047e8c92ad6cfb","row_sha256":"149cc7e282a8181766092e95741e7d74584e56717f89aa15d7fb2de472b1776d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1807","group_id":"mmlu/test/1807","variant":"clean"}},"request_sha256":"202f50d3ca88863290de46bba53384c57997d461e230b5d521aa93db11657d5e","questions":[{"type":"choice","n_options":4,"row_len":149,"slot":148,"row_ids_sha256":"e7e18438a2903d12726e01aefe5fe679646ce1f22d2d2d46b83d313138fa429a","oracle":{"probs":[0.00260690232489956,0.02827153333466012,0.950227336221104,0.018894228119336203],"argmax":"c","top2_margin":0.9219558028864439,"near_tie":false,"group_logz":{"73":-6.00764274597168,"267":-12.938563346862793,"74":-3.6239500045776367,"293":-9.935802459716797,"75":-0.10910415649414062,"280":-6.770832061767578,"76":-4.026948928833008,"277":-10.355390548706055}},"graph":{"mac_fp16":{"p":[0.002608072562448326,0.028241604229897112,0.950295363593685,0.018854959613969442],"max_abs_dp":6.802737258104763e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0025898676716307506,0.0277437033005986,0.9507365013077395,0.018929927720031152],"max_abs_dp":0.0005278300340615209,"argmax_equal":true}}}]},{"id":"tv4_039","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":39,"line_sha256":"aea77b9e473e5b109a9433254ee27ae00b36e5a5b7d016bad19bc8f129377d56","_meta":{"row":4720,"text_sha256":"d9f86719ffa9dc8954a412b58b7482a7dc5bd0ea176d0f0ec31398637cf1648b","row_sha256":"14e515224ce2587fad9cf05bbfab4b5e27f211f949db0c413468ecb5c63e2ea0","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4720","group_id":"mmlu/test/4720","variant":"clean"}},"request_sha256":"4c97f0b73440e6568cccc0bf27a61c4fdf1ac4bac65bdc57b985dca73fbefa66","questions":[{"type":"choice","n_options":4,"row_len":87,"slot":86,"row_ids_sha256":"62cd785278773011f71435aff6fabf79166c65aab02ea667f77263661f3cb108","oracle":{"probs":[0.7898167907950899,0.20084750575102014,0.005699452921382006,0.0036362505325079637],"argmax":"a","top2_margin":0.5889692850440698,"near_tie":false,"group_logz":{"73":-0.2825126647949219,"267":-8.152045249938965,"74":-1.6517677307128906,"293":-8.51938247680664,"75":-5.2139434814453125,"280":-12.300525665283203,"76":-5.663360595703125,"277":-12.649381637573242}},"graph":{"mac_fp16":{"p":[0.7903157688407301,0.20037557336328626,0.00567825894734184,0.003630398848641799],"max_abs_dp":0.0004989780456401416,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7869099228236345,0.2037747928490143,0.005680794632439027,0.003634489694912135],"max_abs_dp":0.002927287097994169,"argmax_equal":true}}}]},{"id":"tv4_040","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":40,"line_sha256":"0968ec65c31f28779888feaaf4b7345c652623c8548b7c67cdc28393d688d39a","_meta":{"row":12568,"text_sha256":"a3a223bcafcf86355939d886cbe5cffcd1499240ff3007125d1629cfb6e6ce05","row_sha256":"15ee481e12ac1182a5e6741fc1dc006dd5acd23711420945c5cfddec3de8c7e5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12568","group_id":"mmlu/test/12568","variant":"clean"}},"request_sha256":"a80617a3d3642a4b6de61bf72f94849bf710eedbb79dd873b5aae1edd1015f57","questions":[{"type":"choice","n_options":4,"row_len":222,"slot":221,"row_ids_sha256":"56842a3e74db27b829b2cb2907fb8bfd64babecbac38de064c2464477bc81c12","oracle":{"probs":[0.05360518616815764,0.31460810425781016,0.050818093189081474,0.5809686163849507],"argmax":"d","top2_margin":0.2663605121271405,"near_tie":false,"group_logz":{"73":-2.992521286010742,"267":-9.956844329833984,"74":-1.22283935546875,"293":-7.7687578201293945,"75":-3.045914649963379,"280":-9.748311996459961,"76":-0.6094703674316406,"277":-6.948970794677734}},"graph":{"mac_fp16":{"p":[0.053333282085219584,0.31419193430581843,0.050766200580085366,0.5817085830288766],"max_abs_dp":0.0007399666439259489,"argmax_equal":true},"iphone_int8mlp":{"p":[0.05414145920424271,0.3226190387792545,0.05118100395816406,0.5720584980583387],"max_abs_dp":0.008910118326611927,"argmax_equal":true}}}]},{"id":"tv4_041","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":41,"line_sha256":"609a36b8975910a716c7cfbd056375cd7645e45b6f67dcd4d59d18bfb4213aaa","_meta":{"row":12163,"text_sha256":"357bec3dde33cccb1b6d97f805743e7804c397287ffbe3cfc449c9f3ce049e8c","row_sha256":"16592e464ce99da5a7f2f2e16f0251575aea7595e69d2fc1092656e96904297e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12163","group_id":"mmlu/test/12163","variant":"clean"}},"request_sha256":"907362bda5711a8e16736b3425cd58df379dc8688eb9a288ee17a184aef5980d","questions":[{"type":"choice","n_options":4,"row_len":229,"slot":228,"row_ids_sha256":"ac90d98573cebf8fcd31bd2b2bf62feae3a45d2cf336330c6d0a69c06ab1dc84","oracle":{"probs":[0.07698989704315282,0.17325123056352462,0.615969403441019,0.13378946895230348],"argmax":"c","top2_margin":0.4427181728774944,"near_tie":false,"group_logz":{"73":-2.643413543701172,"267":-9.093677520751953,"74":-1.8323450088500977,"293":-7.870871543884277,"75":-0.5638904571533203,"280":-6.670784950256348,"76":-2.0908203125,"277":-7.989580154418945}},"graph":{"mac_fp16":{"p":[0.0766884143157115,0.1732416878956411,0.6166707961625153,0.13339910162613206],"max_abs_dp":0.0007013927214962745,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07568814977010395,0.1701684898751513,0.621091126560255,0.13305223379448974],"max_abs_dp":0.005121723119235977,"argmax_equal":true}}}]},{"id":"tv4_042","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":42,"line_sha256":"bc7ced7f048db613944f55accd46cf315f91444881b02b79b5bfab2929dde9d0","_meta":{"row":6773,"text_sha256":"ad92e1941705bfa22d357ace87e80108d4f552db2d04d78c4c7dbcff69e62991","row_sha256":"16f57b256c903af68e1f29638cdea0c98d34cdfe847a6899b3974dfceb24522f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6773","group_id":"mmlu/test/6773","variant":"clean"}},"request_sha256":"97cf99d0c0e1a557813d914788ce36a256e1a02fc267d3e0fbf728565cf4f26f","questions":[{"type":"choice","n_options":4,"row_len":78,"slot":77,"row_ids_sha256":"cf7a6126b5af37c75fca2bdbe27aa470cf39d0389f3b735a4548542afe5cbb87","oracle":{"probs":[0.9147504316682973,0.05621139914159736,0.014039762230388782,0.014998406959716666],"argmax":"a","top2_margin":0.8585390325266999,"near_tie":false,"group_logz":{"73":-0.15307044982910156,"267":-7.752723693847656,"74":-2.9426021575927734,"293":-9.5266695022583,"75":-4.329828262329102,"280":-11.286428451538086,"76":-4.263777732849121,"277":-10.771951675415039}},"graph":{"mac_fp16":{"p":[0.9148341560454468,0.05620383178761728,0.01403608793095778,0.014925924235978008],"max_abs_dp":8.372437714954106e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9128311657590626,0.05781595685938738,0.014275822000795453,0.015077055380754448],"max_abs_dp":0.0019192659092346354,"argmax_equal":true}}}]},{"id":"tv4_043","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":43,"line_sha256":"bcc0b07bbdace8dc92fe41bb4000cf534b39b6eab6dc9a83658b4a4af5e46e6c","_meta":{"row":1982,"text_sha256":"d9ffe57f3bac0b6b5e36685e5fc5c980ca2c277cafee5911504bac119dc8a19b","row_sha256":"173b236e9e6235053ace7bfa297ab81a9dc1d8098f84e6605736ea0497db23d8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1982","group_id":"mmlu/test/1982","variant":"clean"}},"request_sha256":"57746bed9e17cb0dd751b66e59ebb44ac5804c7d2a523e9692bb2ad0409d7ffd","questions":[{"type":"choice","n_options":4,"row_len":79,"slot":78,"row_ids_sha256":"8a27358a4ddd1d655f58ac278a524fd4acb8dff635ceb3d511a0c50d3e714c85","oracle":{"probs":[0.03249727510357624,0.7009527892031219,0.07017282988112963,0.19637710581217224],"argmax":"b","top2_margin":0.5045756833909496,"near_tie":false,"group_logz":{"73":-3.4898977279663086,"267":-10.197124481201172,"74":-0.4186134338378906,"293":-6.880967140197754,"75":-2.7200927734375,"280":-9.009160995483398,"76":-1.6910171508789062,"277":-7.733455657958984}},"graph":{"mac_fp16":{"p":[0.03250869664407037,0.7003665455356092,0.07020895738068504,0.1969158004396355],"max_abs_dp":0.0005862436675126981,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03237563802323539,0.6991377241010455,0.07080058842124802,0.1976860494544711],"max_abs_dp":0.0018150651020764075,"argmax_equal":true}}}]},{"id":"tv4_044","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":44,"line_sha256":"b6fe8c36da22f3a68479e11f7873243ed6934433035ee60c8c3011b29c498314","_meta":{"row":3758,"text_sha256":"209b997de0aab123d791890481a71f42a3242b0de191ce264d3cc16192c12b6e","row_sha256":"173e90ff9c8e472cbc5633766bc1e4310718fa6cfc5738327921b238c04c4b16","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3758","group_id":"mmlu/test/3758","variant":"clean"}},"request_sha256":"c478fe94c15bcd45de86f6e3a59c1a9034bd37f2c7654e8df742518e4692c6fe","questions":[{"type":"choice","n_options":4,"row_len":96,"slot":95,"row_ids_sha256":"9daf9b07609c54d3546bf5f5bcea0857e7da99a5b2b50629d7d234487a6e6aea","oracle":{"probs":[0.001565191093411413,0.0045979971101823575,0.005304333488369794,0.9885324783080365],"argmax":"d","top2_margin":0.9832281448196667,"near_tie":false,"group_logz":{"73":-6.5143890380859375,"267":-14.6019926071167,"74":-5.436776161193848,"293":-13.601155281066895,"75":-5.293872833251953,"280":-13.656209945678711,"76":-0.06617546081542969,"277":-8.327640533447266}},"graph":{"mac_fp16":{"p":[0.001567362481961449,0.004597183110812893,0.0053045835733921305,0.9885308708338335],"max_abs_dp":2.1713885500359263e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0015374048486155442,0.004523938166161808,0.005232253708719911,0.9887064032765027],"max_abs_dp":0.0001739249684662436,"argmax_equal":true}}}]},{"id":"tv4_045","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":45,"line_sha256":"2d2cad279b11f0b10e6f488c94b48d030e523916ca07806dc14ec6eea4f7c9ca","_meta":{"row":1857,"text_sha256":"d1c63be268917217f4bf97cdd3bce1c57767d755fe12f1771a173baaf1eb0455","row_sha256":"1742bf69e66dfd9afd72eca8a80efdfea705670aa814f7c847d3d1a3e82d5bae","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1857","group_id":"mmlu/test/1857","variant":"clean"}},"request_sha256":"d20fb9410485b6bd30d143d12b30d6a24b0e8494f05b0e7f2f308684c8b24e4d","questions":[{"type":"choice","n_options":4,"row_len":112,"slot":111,"row_ids_sha256":"fe31100116bb00d688781a929dd4a79eb3d4f02980a98627d79d5b9a2280fb54","oracle":{"probs":[0.015724784846875676,0.20268899331363502,0.5849678116847831,0.1966184101547062],"argmax":"c","top2_margin":0.3822788183711481,"near_tie":false,"group_logz":{"73":-4.229769706726074,"267":-11.008307456970215,"74":-1.673335075378418,"293":-7.970257759094238,"75":-0.6134510040283203,"280":-6.986556053161621,"76":-1.7037429809570312,"277":-7.5795745849609375}},"graph":{"mac_fp16":{"p":[0.015693469872277693,0.20288933904174844,0.584711823300135,0.1967053677858388],"max_abs_dp":0.00025598838464802043,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01573957150471402,0.20105250906623068,0.5869172521719888,0.19629066725706662],"max_abs_dp":0.0019494404872056892,"argmax_equal":true}}}]},{"id":"tv4_046","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":46,"line_sha256":"09744b9091d552ca5da1f8a2594b21082be690afd31838073978bc63d968a308","_meta":{"row":2806,"text_sha256":"47d3aa72b23e7d16cbb68a5870a03acdf468c693fccc545e641995d1b9d9b59a","row_sha256":"1797ea252aabf8e7b5d92080fb1c53b73873154cfa2f9e0ddedaae07c3b25986","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2806","group_id":"mmlu/test/2806","variant":"clean"}},"request_sha256":"157afeca6d34c6c3bee55d18a1a0336f97b6b6471eaf76bc48584912de00b917","questions":[{"type":"choice","n_options":4,"row_len":83,"slot":82,"row_ids_sha256":"c9a1cedaafd0324a68773b9192d1cfc15a8e2585d40cad506de3cca39b73e3ac","oracle":{"probs":[0.9511277699509987,0.006907074477118398,0.039165589925169775,0.002799565646713108],"argmax":"a","top2_margin":0.9119621800258289,"near_tie":false,"group_logz":{"73":-0.09621238708496094,"267":-8.579292297363281,"74":-5.02131462097168,"293":-12.599237442016602,"75":-3.286062240600586,"280":-10.497303009033203,"76":-5.924396514892578,"277":-12.918901443481445}},"graph":{"mac_fp16":{"p":[0.9511563031161523,0.006895591136839934,0.03915282400816212,0.002795281738845562],"max_abs_dp":2.8533165153610263e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9504683441918248,0.0069338012249008886,0.03980182406628641,0.002796030516987832],"max_abs_dp":0.0006594257591738817,"argmax_equal":true}}}]},{"id":"tv4_047","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":47,"line_sha256":"e52bb207c5a1def7483a9260dadc9490e51d564a4cfb6a8f100adc005958cdc6","_meta":{"row":12630,"text_sha256":"c4fbf107fefb1392e03a9e502108a2e09e06a2848b1b1e732af643146f9ae716","row_sha256":"17b6e12c658cc0b30180f9170090381c8cf88caf34f84ef346ea61c0076d328f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12630","group_id":"mmlu/test/12630","variant":"clean"}},"request_sha256":"6d3d34b96dc176074b7a15b68370b6bb471200fac7442a7b71dbb88ed1400537","questions":[{"type":"choice","n_options":4,"row_len":86,"slot":85,"row_ids_sha256":"c14741f93e55caa11387f19b5304b73ea2b4373aeb24dcb6d711efe901a4755e","oracle":{"probs":[0.023075119455360546,0.12599435020152328,0.8218175404807175,0.02911298986239864],"argmax":"c","top2_margin":0.6958231902791943,"near_tie":false,"group_logz":{"73":-3.812650680541992,"267":-11.386568069458008,"74":-2.115168571472168,"293":-9.190027236938477,"75":-0.23988723754882812,"280":-7.670647621154785,"76":-3.580221176147461,"277":-10.588064193725586}},"graph":{"mac_fp16":{"p":[0.02307026517036586,0.12574391193350296,0.8220623155649257,0.029123507331205407],"max_abs_dp":0.0002504382680203199,"argmax_equal":true},"iphone_int8mlp":{"p":[0.022871447408469883,0.12930475862906735,0.819241319786117,0.028582474176345702],"max_abs_dp":0.0033104084275440704,"argmax_equal":true}}}]},{"id":"tv4_048","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":48,"line_sha256":"629f8fbf15fcb2da3b9bd0b5d7cfdafa09f9ffcf990d1e6f7f8270bccd9d1017","_meta":{"row":8523,"text_sha256":"c2d9284847102813614e2479684b859b9ad6a71aaf310c15851ed05a0c67e047","row_sha256":"17f07d503ffa0494aa586d7a4a18911b4b0d62da344c0dca33d1f6be0a925804","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8523","group_id":"mmlu/test/8523","variant":"clean"}},"request_sha256":"ee0f21e302f5c6211f5a81dc38c4eb70ce82213359ee35348d482eadef083497","questions":[{"type":"choice","n_options":4,"row_len":141,"slot":140,"row_ids_sha256":"f3d84e5776100396d0b9c5748227b23149ee5f9b33db9be8961f2973bd3e7bbe","oracle":{"probs":[0.009452260739083793,0.9604592838407027,0.023559422545627217,0.006529032874586291],"argmax":"b","top2_margin":0.9368998612950754,"near_tie":false,"group_logz":{"73":-4.697768211364746,"267":-12.111128807067871,"74":-0.07661056518554688,"293":-7.192522048950195,"75":-3.784496307373047,"280":-10.494762420654297,"76":-5.067763328552246,"277":-12.131481170654297}},"graph":{"mac_fp16":{"p":[0.009364615030567685,0.960627435660146,0.023473008496092962,0.006534940813193377],"max_abs_dp":0.00016815181944329627,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009133155001925589,0.9601513128570541,0.02423051889323586,0.006485013247784426],"max_abs_dp":0.0006710963476086428,"argmax_equal":true}}}]},{"id":"tv4_049","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":49,"line_sha256":"a4ce6e56a58b39a67368f28462029eadd354f148b62ce43f8717a3fc7780a10f","_meta":{"row":13067,"text_sha256":"0b689018d753bf1e0bab2566ca49fd325829eb7f0cce0eaf324421c61f7eb07b","row_sha256":"18d05dc22cd35d391f093924dbfcf17482a64e1d2d9780a90fd9b43ca842618f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13067","group_id":"mmlu/test/13067","variant":"clean"}},"request_sha256":"3f58210ecc091fae723389c1f8606cb157f91a2cde46035ea91fde6076dd95d3","questions":[{"type":"choice","n_options":4,"row_len":99,"slot":98,"row_ids_sha256":"f81fa77566c848ef1b0259d12f92069cd5a7c9e7331f80a4fb7a57e1fa75340f","oracle":{"probs":[0.0022291719149391937,0.05954339447287092,0.01377564357256196,0.9244517900396279],"argmax":"d","top2_margin":0.8649083955667569,"near_tie":false,"group_logz":{"73":-6.237693786621094,"267":-13.523137092590332,"74":-2.9526185989379883,"293":-9.655634880065918,"75":-4.416421890258789,"280":-11.454873085021973,"76":-0.21012306213378906,"277":-7.147531509399414}},"graph":{"mac_fp16":{"p":[0.0022242796929883833,0.0596079494866399,0.013787652896730236,0.9243801179236414],"max_abs_dp":7.167211598646528e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0021955986695278784,0.057900008806617706,0.013587133589566626,0.9263172589342877],"max_abs_dp":0.001865468894659883,"argmax_equal":true}}}]},{"id":"tv4_050","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":50,"line_sha256":"4ec5a97e18360e8ee46499c742858b2239e52576e7d635ba6e78632358f4c35c","_meta":{"row":2282,"text_sha256":"ca192f34bda6eb5a55ae413e76503d8d9a8481bf2c86e8bad24fcf2cadfd75d2","row_sha256":"18dffaae14139a6757aa0e55718041a4ea0795baf3588fae0d85de5093163620","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2282","group_id":"mmlu/test/2282","variant":"clean"}},"request_sha256":"850bacf0b76f6ed4dcc9441b1b216a1f35b4b02762d4c64b8060ae4fa8ef6570","questions":[{"type":"choice","n_options":4,"row_len":85,"slot":84,"row_ids_sha256":"c800ad75c36fca2ab00097d0c26b0e43e9dc50386283ac9e6fda85d3ac7d3883","oracle":{"probs":[0.2787586312722441,0.6865985118652801,0.013256347034765517,0.02138650982771035],"argmax":"b","top2_margin":0.407839880593036,"near_tie":false,"group_logz":{"73":-1.3348102569580078,"267":-8.751585006713867,"74":-0.4334068298339844,"293":-7.364543914794922,"75":-4.380680084228516,"280":-11.385028839111328,"76":-3.9023962020874023,"277":-10.632255554199219}},"graph":{"mac_fp16":{"p":[0.2786338040940451,0.6867610926533837,0.013232274502803872,0.02137282874976728],"max_abs_dp":0.00016258078810360566,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2842156390703758,0.6807675985244077,0.013229578502759968,0.02178718390245665],"max_abs_dp":0.005830913340872379,"argmax_equal":true}}}]},{"id":"tv4_051","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":51,"line_sha256":"cebedd3e0f613a4ac663adece2d01d91c41aca8ea6bb66f9cade23aad6035214","_meta":{"row":10179,"text_sha256":"ea4396ca54518fad07b8b5961da2009fae7b812110014f09cb227ca35c027a9c","row_sha256":"1953357a0b60b7cc6360565c6031204ac39ec21a61b1f3fb6c3cf9af4632a3cb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10179","group_id":"mmlu/test/10179","variant":"clean"}},"request_sha256":"99d073c8432b7e4e08c69f0eaf980caa986ca7c6edfdd8e15c1eef17a24210bf","questions":[{"type":"choice","n_options":4,"row_len":149,"slot":148,"row_ids_sha256":"3c267c9ea68935dc05e0242137b0443ab2b8b7ff0d262d73a03490e3515a5018","oracle":{"probs":[0.007419908285300375,0.02859624434558629,0.1698122201676141,0.7941716272014993],"argmax":"d","top2_margin":0.6243594070338851,"near_tie":false,"group_logz":{"73":-5.08510684967041,"267":-12.706703186035156,"74":-3.7359981536865234,"293":-10.891834259033203,"75":-1.954580307006836,"280":-9.07265853881836,"76":-0.4119739532470703,"277":-7.332006454467773}},"graph":{"mac_fp16":{"p":[0.007420923748178998,0.028622147894202005,0.17013504545647676,0.7938218829011423],"max_abs_dp":0.0003497443003569467,"argmax_equal":true},"iphone_int8mlp":{"p":[0.007425630963837511,0.02843379093956074,0.17286647618079698,0.7912741019158048],"max_abs_dp":0.0030542560131828767,"argmax_equal":true}}}]},{"id":"tv4_052","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":52,"line_sha256":"faa322f86bb627d8b77c5dc580be71498074247068883bbf18792998663a6823","_meta":{"row":2982,"text_sha256":"98874b205296b4f10427e4fd13b34e861d28b2cc1eb15e895246a288e4223ed3","row_sha256":"197fc93a439226b164e6b58a655da65fbb98aa5fa9efd63e0d8adcae40ae2b9a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2982","group_id":"mmlu/test/2982","variant":"clean"}},"request_sha256":"f212c5de41944a05f4b3e5bf5d49106303f1fd759abcc8ad0296cf97208f8934","questions":[{"type":"choice","n_options":4,"row_len":146,"slot":145,"row_ids_sha256":"36710752d2b57757112eea03c6aa52c2b4600a562dd7c3d50d56be1ab79819ed","oracle":{"probs":[0.9190405136867557,0.0640607164035125,0.007022972808943782,0.009875797100787979],"argmax":"a","top2_margin":0.8549797972832431,"near_tie":false,"group_logz":{"73":-0.11925125122070312,"267":-7.930459976196289,"74":-2.782750129699707,"293":-9.313433647155762,"75":-4.99339485168457,"280":-11.951580047607422,"76":-4.652494430541992,"277":-11.13908576965332}},"graph":{"mac_fp16":{"p":[0.9189342565396695,0.06415349563538314,0.0070279948311083576,0.009884252993839021],"max_abs_dp":0.00010625714708611955,"argmax_equal":true},"iphone_int8mlp":{"p":[0.919369045293595,0.06390018504195238,0.006870537466437501,0.009860232198015143],"max_abs_dp":0.0003285316068393218,"argmax_equal":true}}}]},{"id":"tv4_053","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":53,"line_sha256":"90353a7da2b83810b8456a350a29e316df9128ab7c03722b50ab34c4e0032acc","_meta":{"row":10687,"text_sha256":"8ad6ed1ee05fb239397328c42c3635969b0116c8cd706783db3b539d69442150","row_sha256":"1afab0887f5eed8238a62ad38710101c9a670e8fc7c0e297ee8870bc7da16ca5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10687","group_id":"mmlu/test/10687","variant":"clean"}},"request_sha256":"7a91643e508c32ad74795e676b04f259ee57cd32c4c0cc71809d577c91502b4c","questions":[{"type":"choice","n_options":4,"row_len":370,"slot":369,"row_ids_sha256":"4f2b47b37348dfad14f8b36a0a15b5dae1def13ae7aba68f25edccf57c7bdcdc","oracle":{"probs":[0.024923515235430762,0.8417990762969715,0.01857280818643943,0.11470460028115831],"argmax":"b","top2_margin":0.7270944760158131,"near_tie":false,"group_logz":{"73":-3.7553749084472656,"267":-10.625024795532227,"74":-0.23564529418945312,"293":-6.814642906188965,"75":-4.049488067626953,"280":-10.475631713867188,"76":-2.2288265228271484,"277":-8.24567985534668}},"graph":{"mac_fp16":{"p":[0.024946691686156732,0.8417591940525468,0.01854992064931837,0.11474419361197807],"max_abs_dp":3.988224442463828e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.024309586842851866,0.8440402486830265,0.01837140937786095,0.11327875509626066],"max_abs_dp":0.0022411723860550348,"argmax_equal":true}}}]},{"id":"tv4_054","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":54,"line_sha256":"959e55bf1d932e38ed0968c99a9ec1264ab3468ccf6031c843b6b4f1f214d969","_meta":{"row":931,"text_sha256":"bffdafb5dfcb1e38747797c28c9240ba377547c3a3b7918284db1bf3bcf291f0","row_sha256":"1b19c3c4efa3abc6ba91131741026207079d43fb02ec9d8da8d87d08eeae05ea","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/931","group_id":"mmlu/test/931","variant":"clean"}},"request_sha256":"68dc92957262ab788dd3adc66a2166cb68d5c09fe4bcf70317b78092a9f004d6","questions":[{"type":"choice","n_options":4,"row_len":114,"slot":113,"row_ids_sha256":"dc27b38c5da299842aed56ce83196d55a51cbfb7f1f9dc5677806a94d7b58359","oracle":{"probs":[0.12845700929608725,0.3321847780203702,0.25741003466655354,0.281948178016989],"argmax":"b","top2_margin":0.050236600003381204,"near_tie":false,"group_logz":{"73":-2.198719024658203,"267":-8.27485179901123,"74":-1.248621940612793,"293":-7.134809494018555,"75":-1.5036430358886719,"280":-7.344236373901367,"76":-1.4125900268554688,"277":-6.85469913482666}},"graph":{"mac_fp16":{"p":[0.1281805414815234,0.33255187013659426,0.2575511859253488,0.2817164024565335],"max_abs_dp":0.0003670921162240437,"argmax_equal":true},"iphone_int8mlp":{"p":[0.12820417735165549,0.3331642162866695,0.2583569695056455,0.2802746368560295],"max_abs_dp":0.001673541160959524,"argmax_equal":true}}}]},{"id":"tv4_055","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":55,"line_sha256":"7b687e66af486614648d8cd236bf7caf49938061890f1551b010a220fe26c3f1","_meta":{"row":10372,"text_sha256":"600f6c0edf1596b591557e7d464d168fee58a76d37bccd04899ea67277ca8440","row_sha256":"1d98a7be5205e8cdca077a6afd8d4b19c680d2b6dbeed56307facafc6211e788","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10372","group_id":"mmlu/test/10372","variant":"clean"}},"request_sha256":"006e4f8598db85a6fb9a466919e4d9fa384b33ec59fda322444eccd765abdafc","questions":[{"type":"choice","n_options":4,"row_len":174,"slot":173,"row_ids_sha256":"823bc01c7e1b132ea12371c072cc220b29a1a2b1a30b02a0abfd23f5ccff22de","oracle":{"probs":[0.2556864654052718,0.34532078614778344,0.3108383594169879,0.08815438902995686],"argmax":"b","top2_margin":0.034482426730795546,"near_tie":false,"group_logz":{"73":-1.4927740097045898,"267":-7.919608116149902,"74":-1.1922521591186523,"293":-7.285384178161621,"75":-1.2974529266357422,"280":-7.320071220397949,"76":-2.557636260986328,"277":-8.154911041259766}},"graph":{"mac_fp16":{"p":[0.2558063798558529,0.3447529491197904,0.31129773848375125,0.08814293254060539],"max_abs_dp":0.0005678370279930678,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2554423847157317,0.3447153403732318,0.31214725576576535,0.08769501914527113],"max_abs_dp":0.0013088963487774552,"argmax_equal":true}}}]},{"id":"tv4_056","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":56,"line_sha256":"ca4d4d000f48001968f8ec680408cf392bcac5adb9622b4396f0bddcf1261e07","_meta":{"row":3523,"text_sha256":"0a55d50ec128876b1b93a1817c50be49da2b8d9ba9da57fe6b8e68bc7fc1f804","row_sha256":"1f67a2db1cb8de1eb8994cb200cca72336ee543d0d712feeac0f7377fc9ccc0b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3523","group_id":"mmlu/test/3523","variant":"clean"}},"request_sha256":"5d71c49ba2678143aa814f1d8423b0d9a0662344db089ca05125e3c17fcf84cf","questions":[{"type":"choice","n_options":4,"row_len":88,"slot":87,"row_ids_sha256":"28e9fe4ed923f161eb4418fe617fec30db04ee633295d00a09492f62a8ef6f9d","oracle":{"probs":[0.0036623121125616462,0.017686467455127475,0.02475707079588844,0.9538941496364224],"argmax":"d","top2_margin":0.929137078840534,"near_tie":false,"group_logz":{"73":-5.723426818847656,"267":-13.590612411499023,"74":-4.148721694946289,"293":-11.836851119995117,"75":-3.812410354614258,"280":-11.748747825622559,"76":-0.16096878051757812,"277":-7.221408843994141}},"graph":{"mac_fp16":{"p":[0.0036645157378785652,0.017698057858917135,0.024800474283918027,0.9538369521192862],"max_abs_dp":5.7197517136176046e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0036621264570335078,0.0178131571333327,0.02452220274331483,0.954002513666319],"max_abs_dp":0.00023486805257361132,"argmax_equal":true}}}]},{"id":"tv4_057","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":57,"line_sha256":"c22f44bec87abf1b40d2db9595caa0a4088c772969cef65598625fe10127f39d","_meta":{"row":6245,"text_sha256":"232d792150da6346024c6becbeb1ed6e41d7b8225d3afc5408dc51c7d2883b3f","row_sha256":"1fa4a3b6a91ab2aeb7bb098782fa05e9bad721b7a1db556e4fbb62d55ab70c04","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6245","group_id":"mmlu/test/6245","variant":"clean"}},"request_sha256":"2c7383873af2a65697f4c6352ba78c5b71e1ad970637e734d6031850c1767762","questions":[{"type":"choice","n_options":4,"row_len":80,"slot":79,"row_ids_sha256":"267c3c98c2100b01ecc75eab020b0d7e7f31aaae090b7bbde60ebcc68d74c1c1","oracle":{"probs":[0.015799752017775264,0.0456415569924584,0.23907579353407227,0.699482897455694],"argmax":"d","top2_margin":0.46040710392162176,"near_tie":false,"group_logz":{"73":-4.502676963806152,"267":-11.500616073608398,"74":-3.441852569580078,"293":-10.097492218017578,"75":-1.7858905792236328,"280":-8.23232650756836,"76":-0.7123298645019531,"277":-6.900540351867676}},"graph":{"mac_fp16":{"p":[0.015838259721235766,0.04576170444175482,0.23973681539334582,0.6986632204436636],"max_abs_dp":0.0008196770120304508,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016050737012130684,0.04646936983906487,0.2390981487412663,0.6983817444075381],"max_abs_dp":0.001101153048155945,"argmax_equal":true}}}]},{"id":"tv4_058","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":58,"line_sha256":"3aa640a4bbf743d724bdb741ab8dc613c5cc4100d1c20dead76a6eccbbd12cbe","_meta":{"row":9182,"text_sha256":"c041a95d73eaf3622c37781715d7651e4e11b576acfd78959990153d966a28b4","row_sha256":"202af939f40c369cf54f711791efd619354fd75d0a8159f50bff9d69acd88401","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9182","group_id":"mmlu/test/9182","variant":"clean"}},"request_sha256":"3414ff4f10d90c7ede691c2ebe0da7cec1baf6c72176df8718a80bd3596dc07a","questions":[{"type":"choice","n_options":4,"row_len":152,"slot":151,"row_ids_sha256":"58cc3d395b77d42b3b5c3ac10a2f12410bcc421c29207a9f468627e780da01e6","oracle":{"probs":[0.07234532745327527,0.26625161476323256,0.6530949175885639,0.00830814019492824],"argmax":"c","top2_margin":0.3868433028253313,"near_tie":false,"group_logz":{"73":-2.67156982421875,"267":-9.52515983581543,"74":-1.3685789108276367,"293":-7.760261535644531,"75":-0.4712982177734375,"280":-6.96577262878418,"76":-4.835784912109375,"277":-11.358109474182129}},"graph":{"mac_fp16":{"p":[0.07252327597828262,0.26697712494422726,0.6522050703567212,0.00829452872076899],"max_abs_dp":0.0008898472318427109,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07529539639730087,0.26752285250461877,0.6489388257321742,0.008242925365906215],"max_abs_dp":0.004156091856389654,"argmax_equal":true}}}]},{"id":"tv4_059","source":"tv4","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":59,"line_sha256":"0f8cfb484311049160b9dcb55d9351563aa6530ec45c142c228ce42f8bdd02ca","_meta":{"row":5150,"text_sha256":"521d509885166a58de2627c79846a9aa42a22d22d83f8ffcfadf1e28d3e106e3","row_sha256":"2036db6f4f015742ccc14fe5697c9e51c566c5dd072f2b40b6f44fd892f4b76c","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5150","group_id":"mmlu/test/5150","variant":"clean"}},"request_sha256":"d55b1c11fab67118b079120f7965d45d31f562221ca37cf2c296ec1bf4b514ef","questions":[{"type":"choice","n_options":4,"row_len":116,"slot":115,"row_ids_sha256":"24fcc6bcc9b6fbcad0564f5e581b42cfd4f15fc368314fa9fccd38e01fd458c4","oracle":{"probs":[0.973006495161894,0.007083121292909364,0.004448923098081876,0.015461460447114668],"argmax":"a","top2_margin":0.9575450347147793,"near_tie":false,"group_logz":{"73":-0.07605552673339844,"267":-8.191511154174805,"74":-4.99873161315918,"293":-12.254314422607422,"75":-5.463784217834473,"280":-12.886198043823242,"76":-4.218095779418945,"277":-10.617271423339844}},"graph":{"mac_fp16":{"p":[0.9730305482453451,0.007071494827120832,0.0044428476083007385,0.015455109319233428],"max_abs_dp":2.405308345110324e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9725158955497172,0.007203854089716052,0.0044987224467916125,0.01578152791377516],"max_abs_dp":0.0004905996121767897,"argmax_equal":true}}}]},{"id":"tv4x_emotion_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":80,"line_sha256":"33fc5f1fd4759a5ebe299a9189036ce841399dc48f69d4bf5c0fa2a91a5c5542","_meta":{"row":1012,"text_sha256":"6aa999c7c36768c0b24075e2e822f1b740d65573d11fb4a480891f642000d34d","row_sha256":"00004983dc48fe12fb4541c466c19f3119120f8c61b237a660984044be80c62f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1012","group_id":"emotion/test/1012","variant":"clean"}},"request_sha256":"e7ccafee321fc0e43794c3a6e6d1295e928d2a6adcf4b767b5188a1dbb8e7e5b","questions":[{"type":"choice","n_options":6,"row_len":57,"slot":56,"row_ids_sha256":"4faa0fbd175b79d660094a72120598c8056ee86036ab18b31e18d282c1150ef8","oracle":{"probs":[0.13915788188359635,0.21979529726573543,0.5789310767818201,0.02574753681722862,0.018833707755084857,0.017534499496534645],"argmax":"love","top2_margin":0.35913577951608466,"near_tie":false,"group_logz":{"41":-2.0130834579467773,"334":-8.36631965637207,"42":-1.5559959411621094,"378":-7.655674934387207,"43":-0.5875091552734375,"340":-6.804902076721191,"44":-3.7003536224365234,"388":-10.176432609558105,"45":-4.013044357299805,"407":-10.288476943969727,"46":-4.084522247314453,"410":-10.635921478271484}},"graph":{"mac_fp16":{"p":[0.1392300294408167,0.22038481406896207,0.5783037923132598,0.025746649754957596,0.01881920895767744,0.017515505464326387],"max_abs_dp":0.0006272844685603074,"argmax_equal":true},"iphone_int8mlp":{"p":[0.13959517409195982,0.22028332840473588,0.5779941521462958,0.02564509167770539,0.01884830144071475,0.01763395223858844],"max_abs_dp":0.0009369246355243055,"argmax_equal":true}}}]},{"id":"tv4x_emotion_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":81,"line_sha256":"7c835cb702fc9570194cb4fdc6213ccc7efa554c634c74bc6e11c94896cab672","_meta":{"row":876,"text_sha256":"d1f1ee7d007de6f0a8f54aac3f50b417a1486ef7e7978ce1b94b83fe3e7281da","row_sha256":"00a2a1d85a3a3b058f34f6fe3cbe33246abe45ec564ad5a1c00a6e62a653436c","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/876","group_id":"emotion/test/876","variant":"clean"}},"request_sha256":"6abe93fc205731c195d88f7dbb0cbd4b26a7a71908ae20ff0059b091b9f882fe","questions":[{"type":"choice","n_options":6,"row_len":54,"slot":53,"row_ids_sha256":"4baba286f4f170400450336eb714029adfe42bac012b6b8268cbd531ebe122f7","oracle":{"probs":[0.26008477503236005,0.3286886133181376,0.03704704069325398,0.17281041262461772,0.07798414446793572,0.12338501386369494],"argmax":"joy","top2_margin":0.06860383828577754,"near_tie":false,"group_logz":{"41":-1.3996667861938477,"334":-8.407903671264648,"42":-1.1655635833740234,"378":-7.9079132080078125,"43":-3.3484859466552734,"340":-10.311298370361328,"44":-1.8084793090820312,"388":-8.519858360290527,"45":-2.6041688919067383,"407":-9.160745620727539,"46":-2.145364761352539,"410":-8.737131118774414}},"graph":{"mac_fp16":{"p":[0.2605709675501977,0.32840177735902315,0.037054340974504774,0.17286278086083212,0.07785573595728913,0.12325439729815318],"max_abs_dp":0.00048619251783765804,"argmax_equal":true},"iphone_int8mlp":{"p":[0.26364922602657404,0.32599254973949254,0.036939812217678065,0.17311288177035453,0.07786424275812193,0.1224412874877789],"max_abs_dp":0.0035644509942139857,"argmax_equal":true}}}]},{"id":"tv4x_emotion_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":82,"line_sha256":"936153aac3c55743e5de1ebc84fe7aeffd3f49b6aa947b3914fc1ca66509fc70","_meta":{"row":230,"text_sha256":"d4f0ae6c4554b83a2c946c3889646c017a9cb83069617197c6ceeb2e80e6da83","row_sha256":"017bb61de48393f84791a2a30f5106a6c15b572d2b180da2ab81f431b9185733","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/230","group_id":"emotion/test/230","variant":"clean"}},"request_sha256":"1fe5b117225dcb4b4079de323cc810758e4639c33721dff189f0211db56309e0","questions":[{"type":"choice","n_options":6,"row_len":51,"slot":50,"row_ids_sha256":"8fac472683e3e2c4d0be3dee6467a6b1a23c3382bfac1cfdda7b49dcab038eb5","oracle":{"probs":[0.9238131531752686,0.008894473050964758,0.006032920523754534,0.027282444809484375,0.03008961447984438,0.0038873939606833535],"argmax":"sadness","top2_margin":0.8937235386954242,"near_tie":false,"group_logz":{"41":-0.09383392333984375,"334":-7.869402885437012,"42":-4.736913681030273,"378":-12.473432540893555,"43":-5.125112533569336,"340":-12.927362442016602,"44":-3.616100311279297,"388":-10.953858375549316,"45":-3.5181636810302734,"407":-10.945449829101562,"46":-5.564604759216309,"410":-13.266057968139648}},"graph":{"mac_fp16":{"p":[0.9239259008132776,0.00887889927166791,0.006025920730873223,0.027236315007863356,0.03004704001962193,0.0038859241566959455],"max_abs_dp":0.00011274763800905863,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9243650687092994,0.008855022877918678,0.00599247457981993,0.02709498326331072,0.02982834484484939,0.0038641057248019806],"max_abs_dp":0.000551915534030778,"argmax_equal":true}}}]},{"id":"tv4x_emotion_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":83,"line_sha256":"b6590e1615dfb82ee6833351de5fa3880296d73d628f6a3d2a319d89bdd449ac","_meta":{"row":20,"text_sha256":"48561b46ae7c00bedfbe6e7560f1bc0c33837955c08902b3a62e5321eeb396ae","row_sha256":"01f0a333c9946da779ee21cf036b16ab5820bdeb220dba361488eec9aee20d57","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/20","group_id":"emotion/test/20","variant":"clean"}},"request_sha256":"c7ef26866f86800c8921049a12e666b7d67343267238ba96a5518082ce4aea19","questions":[{"type":"choice","n_options":6,"row_len":88,"slot":87,"row_ids_sha256":"a5a0f215f132121ca062fa2779812e5f9846e7875bba2f0a938468b5ef77539a","oracle":{"probs":[0.8162291367804331,0.09563744367705561,0.07853188983951137,0.003575764063774041,0.003721998872483546,0.0023037667667422817],"argmax":"sadness","top2_margin":0.7205916931033775,"near_tie":false,"group_logz":{"41":-0.21800804138183594,"334":-7.440655708312988,"42":-2.3621387481689453,"378":-9.168051719665527,"43":-2.5591983795166016,"340":-9.304247856140137,"44":-5.648524284362793,"388":-12.888381004333496,"45":-5.608442306518555,"407":-12.90800952911377,"46":-6.088157653808594,"410":-13.563095092773438}},"graph":{"mac_fp16":{"p":[0.8166458638313606,0.09529661500367366,0.07846458674058691,0.0035695509719020795,0.0037183345353429954,0.0023050489171336725],"max_abs_dp":0.0004167270509275456,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8158454486294244,0.09528532913564586,0.07932839437760934,0.0035365691586343034,0.0037201205662121067,0.0022841381324740216],"max_abs_dp":0.0007965045380979724,"argmax_equal":true}}}]},{"id":"tv4x_emotion_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":84,"line_sha256":"23f1a7fc08a7486d798ab07a4e59279139e1302ceb8b7f64e58d65b94dc83bb7","_meta":{"row":1635,"text_sha256":"25b4f38b3acba992e9f64c7a7ec53d190e099eb4a2618cb2428f157836f55f64","row_sha256":"024d94cd7aeffeb38fe329e8920ff3d127ddbf8d0d3b7b63d5dac2eebdedad67","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1635","group_id":"emotion/test/1635","variant":"clean"}},"request_sha256":"74c5b69039ee196e9777b0186c0ffb8467459520cc2cdf7a5bcdf5936bd90cf7","questions":[{"type":"choice","n_options":6,"row_len":54,"slot":53,"row_ids_sha256":"5b9522ddb2c47fa77b9408bb35f31ecb8cc9a881b9141c2ecbd8ec0c982bc488","oracle":{"probs":[0.009789011421377876,0.9629824180225043,0.019364912842990488,0.0032298316691736664,0.0018550334687598645,0.0027787925751937156],"argmax":"joy","top2_margin":0.9436175051795138,"near_tie":false,"group_logz":{"41":-4.638703346252441,"334":-12.36996841430664,"42":-0.04992866516113281,"378":-7.371223449707031,"43":-3.956501007080078,"340":-11.242691040039062,"44":-5.747533798217773,"388":-13.376091003417969,"45":-6.302061080932617,"407":-13.666913986206055,"46":-5.897947311401367,"410":-13.498947143554688}},"graph":{"mac_fp16":{"p":[0.009803308045087531,0.9629262873702715,0.019388471922958977,0.003236462660189437,0.0018615634581403413,0.0027839065433521715],"max_abs_dp":5.6130652232755374e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009723778174509267,0.9634817583615737,0.018980548806499534,0.003178847167339565,0.0018577437151298358,0.0027773237749480154],"max_abs_dp":0.0004993403390693985,"argmax_equal":true}}}]},{"id":"tv4x_emotion_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":85,"line_sha256":"dc0b748e87e2367c6d98f6eb3c74502fe06e68aeb9d2d1b7fe43cca9e2afdc7b","_meta":{"row":1435,"text_sha256":"665c7b7f8bc910f661810f975c8e50ae78eafc69f4331cbbfa4e0aa6ac02cc8a","row_sha256":"025d147634ffff48088a9ad95b2a2f87a3051a1fac9f6b88cacf7b9a02bc243e","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1435","group_id":"emotion/test/1435","variant":"clean"}},"request_sha256":"344f404bc1960a76e755a29e41701797c0afb7ab7195003e4d82101931102239","questions":[{"type":"choice","n_options":6,"row_len":67,"slot":66,"row_ids_sha256":"32cfaf6cdd545ae99763598b27bec808467d9485cf32aa8f6dc7a059bec0f59a","oracle":{"probs":[0.12881054447956852,0.15764507673194317,0.015030129376514486,0.684432313559413,0.008037226522659379,0.006044709329901309],"argmax":"anger","top2_margin":0.5267872368274699,"near_tie":false,"group_logz":{"41":-2.0733232498168945,"334":-9.267255783081055,"42":-1.8713197708129883,"378":-8.908242225646973,"43":-4.221609115600586,"340":-11.636022567749023,"44":-0.403076171875,"388":-7.518770217895508,"45":-4.84758186340332,"407":-12.04050350189209,"46":-5.132482528686523,"410":-12.711177825927734}},"graph":{"mac_fp16":{"p":[0.12882020315921144,0.15754180840290297,0.014985988713731009,0.684598810301487,0.008022376213036402,0.006030813209631286],"max_abs_dp":0.0001664967420739183,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1292977191369497,0.1577083809811105,0.014851586457688616,0.6841732876136168,0.007985093875785566,0.005983931934848771],"max_abs_dp":0.00048717465738118415,"argmax_equal":true}}}]},{"id":"tv4x_emotion_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":86,"line_sha256":"4a1664e67aa4eb1a1841e277e603cfa26b056ea759bd9f911366c13e8e4c75ab","_meta":{"row":639,"text_sha256":"cc97f8dda8e945ac9d7aabd8ef5e7ff18662a31aaf94beb49d72434087a4ebca","row_sha256":"02ab69252c9985b80f1d7818cd086cd6b4b0a8a3d338d127d5b3c832c7930390","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/639","group_id":"emotion/test/639","variant":"clean"}},"request_sha256":"56fe1ea7b2570a9c8663ae4034abac603b2920d9fd4cf1b6cbb6f9321be77527","questions":[{"type":"choice","n_options":6,"row_len":52,"slot":51,"row_ids_sha256":"a1d9c5f5bb779f7a78b85e4ba1108762d7d4d91da93345306ab3e1b555776e77","oracle":{"probs":[0.012584372143850279,0.2590718297402626,0.7122101484654091,0.00671530145332911,0.004774808327431111,0.004643539869717724],"argmax":"love","top2_margin":0.4531383187251465,"near_tie":false,"group_logz":{"41":-4.398931503295898,"334":-11.299914360046387,"42":-1.374281883239746,"378":-7.652937889099121,"43":-0.36301422119140625,"340":-6.785287857055664,"44":-5.026998519897461,"388":-11.824041366577148,"45":-5.368033409118652,"407":-12.007623672485352,"46":-5.395910263061523,"410":-12.377908706665039}},"graph":{"mac_fp16":{"p":[0.012585520959507216,0.2592017195654352,0.7120749662081598,0.006717253093637844,0.004777689470187829,0.004642850703072127],"max_abs_dp":0.0001351822572492889,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012472749142810682,0.2570353426281701,0.7145370225623919,0.006609634194851536,0.004745169105732177,0.004600082366043483],"max_abs_dp":0.002326874096982845,"argmax_equal":true}}}]},{"id":"tv4x_emotion_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":87,"line_sha256":"0977e7568010552a6b7178c2e5c60a5588f59dfa29d60847eb353e4b49bd048f","_meta":{"row":281,"text_sha256":"2d284bed307dbadc63087b1cf19db78694e59001d29833dbb6b7677743978954","row_sha256":"02eeedfe34d1d5c5c4089939a0fbbb334f7c3400c4230a5677cda20d3220dbcc","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/281","group_id":"emotion/test/281","variant":"clean"}},"request_sha256":"69bf203ae9cbd17070879dfa770b88ec4748037d2577a73c264664336991a198","questions":[{"type":"choice","n_options":6,"row_len":54,"slot":53,"row_ids_sha256":"5d5e37a4b855fdb1f81013e912e6f8c352819e72b1b97ee6910482e3e738d89d","oracle":{"probs":[0.8829131696727059,0.015149115253986593,0.007607928372744794,0.08087719001705605,0.009007993781914175,0.004444602901592322],"argmax":"sadness","top2_margin":0.8020359796556499,"near_tie":false,"group_logz":{"41":-0.14076995849609375,"334":-7.887605667114258,"42":-4.2060546875,"378":-11.808714866638184,"43":-4.894805908203125,"340":-12.66158676147461,"44":-2.531064987182617,"388":-9.709348678588867,"45":-4.725884437561035,"407":-12.366057395935059,"46":-5.432306289672852,"410":-13.074559211730957}},"graph":{"mac_fp16":{"p":[0.882493773478836,0.015189429067367154,0.007624044997456859,0.08118715356268373,0.009040754916206739,0.004464843977449571],"max_abs_dp":0.00041939619386999194,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8841579794692116,0.01516097569909447,0.007552455358070711,0.07979335059491584,0.008938364626453399,0.004396874252253932],"max_abs_dp":0.0012448097965056526,"argmax_equal":true}}}]},{"id":"tv4x_emotion_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":88,"line_sha256":"7f958610f2ec410e070de3aac8ab93a9a2ad21246ac4edd971dcaaa04bd67f7c","_meta":{"row":790,"text_sha256":"f03cc3f062e67ad773241fc1f28f1bd6aa5b3fe45ed5c39046feb017591253c6","row_sha256":"0448b9b74ada8136d2e437a1d7d92c25eede055a11baa59da5ab93cd91cc64e8","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/790","group_id":"emotion/test/790","variant":"clean"}},"request_sha256":"d6bb0af3d5bf781e9cd00b9b779b6370bf1697bf9a5ade16caeb1b6c9c18bd96","questions":[{"type":"choice","n_options":6,"row_len":69,"slot":68,"row_ids_sha256":"acade21a5430288dc53d94048f60cf562d25c0cf594c7a8d7726654ea1667530","oracle":{"probs":[0.22252433033063276,0.6695836932371565,0.09851741375608501,0.0039673599283681135,0.0025132218325998397,0.0028939809151577274],"argmax":"joy","top2_margin":0.4470593629065237,"near_tie":false,"group_logz":{"41":-1.5203399658203125,"334":-8.388275146484375,"42":-0.4187202453613281,"378":-7.17155647277832,"43":-2.3351430892944336,"340":-8.917754173278809,"44":-5.547275543212891,"388":-12.747337341308594,"45":-6.003810882568359,"407":-12.99553394317627,"46":-5.862743377685547,"410":-13.172870635986328}},"graph":{"mac_fp16":{"p":[0.22241785548773266,0.6695975894607339,0.09863387844719029,0.003958740321865926,0.0025063762179549174,0.0028855600645222895],"max_abs_dp":0.00011646469110528079,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2209196710722862,0.6717535252697592,0.09803020388451435,0.003915453885367191,0.002506935684200176,0.0028742102038729093],"max_abs_dp":0.002169832032602681,"argmax_equal":true}}}]},{"id":"tv4x_emotion_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":89,"line_sha256":"807617b6b07be087597761d0bdd624a3168f5465cdc5d8ec278d05be645e182f","_meta":{"row":1948,"text_sha256":"643e031f6d736b138b810ffcd5374c2edf907aad2227273fd3dac28ec809b77c","row_sha256":"067e3ced09e2e785889dbbf0883308f485c57906c95ba0be86a6eb2a67ecc2a9","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1948","group_id":"emotion/test/1948","variant":"clean"}},"request_sha256":"524b9267120163549a5778ea564aee62608b89c654087fcba3ef1fc94c735bb8","questions":[{"type":"choice","n_options":6,"row_len":76,"slot":75,"row_ids_sha256":"b374f4d4cc544b8324560bd8637851f75c6e8965083e629ca5b5c46f0963ab8b","oracle":{"probs":[0.7928570031122211,0.03846441228587972,0.08586979490032044,0.0621301493030717,0.007666304070596854,0.013012336327910275],"argmax":"sadness","top2_margin":0.7069872082119006,"near_tie":false,"group_logz":{"41":-0.25461387634277344,"334":-7.753948211669922,"42":-3.2805233001708984,"378":-10.466392517089844,"43":-2.4774246215820312,"340":-9.533855438232422,"44":-2.801025390625,"388":-9.829375267028809,"45":-4.8934221267700195,"407":-12.30698013305664,"46":-4.364358901977539,"410":-11.709341049194336}},"graph":{"mac_fp16":{"p":[0.7932538623084846,0.038360176179125266,0.08577736271852034,0.061995980716162115,0.007657572056665102,0.012955046021042534],"max_abs_dp":0.0003968591962635326,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7928110911394286,0.03816364482505036,0.08572680307227754,0.06253457248959247,0.007699980421237849,0.013063908052413253],"max_abs_dp":0.000404423186520772,"argmax_equal":true}}}]},{"id":"tv4x_emotion_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":90,"line_sha256":"7c75b791e662008acd41ebc10164fc9a079fbc3fd141a93398924ecf416eb9dd","_meta":{"row":616,"text_sha256":"4176ff72f271799d281cea4c289d899b8631b8843f7745fb1d57d3cee3ae5eb8","row_sha256":"06adb02feddb5c0bba16c41cced75756d41f423fbff9795916d1bdce28039f48","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/616","group_id":"emotion/test/616","variant":"clean"}},"request_sha256":"37539c9e310093524abe6b8e0273f73d949ef1aa10c58807522954f6ff95cd7a","questions":[{"type":"choice","n_options":6,"row_len":56,"slot":55,"row_ids_sha256":"19d4bab025ee5ca63b0c6906b10a1268d76c1d61c0dabd31016076434df2e933","oracle":{"probs":[0.6883296047501994,0.033213226384366405,0.017403888219884436,0.23635688704997176,0.013974578121843462,0.010721815473734575],"argmax":"sadness","top2_margin":0.4519727177002277,"near_tie":false,"group_logz":{"41":-0.40601348876953125,"334":-7.476472854614258,"42":-3.437333106994629,"378":-10.359964370727539,"43":-4.083587646484375,"340":-11.150634765625,"44":-1.4749383926391602,"388":-8.026168823242188,"45":-4.303041458129883,"407":-11.26901912689209,"46":-4.568000793457031,"410":-11.628996849060059}},"graph":{"mac_fp16":{"p":[0.6886250323019475,0.03313586999496217,0.01739947590543537,0.23619021394266754,0.013966179885972967,0.010683227969014374],"max_abs_dp":0.0002954275517480509,"argmax_equal":true},"iphone_int8mlp":{"p":[0.690980485064585,0.03336020830856984,0.017452349814814628,0.2336993153780936,0.01386893226562866,0.010638709168308362],"max_abs_dp":0.0026575716718781617,"argmax_equal":true}}}]},{"id":"tv4x_emotion_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":91,"line_sha256":"34979bafb95a05854ec7667e9829f0fbddcd8a11538585b435d8d426ce45cff6","_meta":{"row":1171,"text_sha256":"bfb1daa85dda1a6897035bac2a77661f7002d68a36cdd52b85c585af4d7958a8","row_sha256":"077454a516cd975420be15757456ce45684963fd916e47385f0f941d2f96518c","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1171","group_id":"emotion/test/1171","variant":"clean"}},"request_sha256":"92878930abfe0a12ef5f04bace87ab6ae275ae7fb47e0acb4f944332c8b55be6","questions":[{"type":"choice","n_options":6,"row_len":63,"slot":62,"row_ids_sha256":"560c643646a7c770759076634095319c7f9a2be653ed1f37f74bf427bbd765cf","oracle":{"probs":[0.015225442962337337,0.9578960110431468,0.01593678300282561,0.0035559226841831264,0.0027029185120518487,0.004682921795455213],"argmax":"joy","top2_margin":0.9419592280403212,"near_tie":false,"group_logz":{"41":-4.197052001953125,"334":-11.619893074035645,"42":-0.05528068542480469,"378":-7.265960693359375,"43":-4.151390075683594,"340":-11.250547409057617,"44":-5.651405334472656,"388":-13.089390754699707,"45":-5.925687789916992,"407":-13.046724319458008,"46":-5.376097679138184,"410":-12.655363082885742}},"graph":{"mac_fp16":{"p":[0.015226724382678174,0.9579047702349257,0.01592956131389053,0.0035518262677321573,0.002704351026348256,0.004682766774425214],"max_abs_dp":8.759191778895392e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015127206631764964,0.9581653971729689,0.01582162015015456,0.003513762855428854,0.0027024346567872163,0.004669578532895603],"max_abs_dp":0.0002693861298220801,"argmax_equal":true}}}]},{"id":"tv4x_emotion_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":92,"line_sha256":"cd9f14b4f6775370c3f3fe60352d5bc8a278f07a3544cdc2965a117f814b7f74","_meta":{"row":1126,"text_sha256":"6f7a8045cf2af8513305b849b51ea6b978643f9b7d2006fb29851c172089324b","row_sha256":"0776ff3e8ecc84647559298dc85c267389e4d6fcf9238915d5c0f81b7715ad45","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1126","group_id":"emotion/test/1126","variant":"clean"}},"request_sha256":"8b2e016bd84afec56d7e08217893cf574f51d26a1851e3f434a0e0eca6d619f7","questions":[{"type":"choice","n_options":6,"row_len":97,"slot":96,"row_ids_sha256":"72777da16bbd04c6a1a10c4d2d18a9b3225e4904398babc64267fbd44bd0b004","oracle":{"probs":[0.03348119017173233,0.18686886007034811,0.7515834285155204,0.015958981356810675,0.006522326358886184,0.005585213526702305],"argmax":"love","top2_margin":0.5647145684451723,"near_tie":false,"group_logz":{"41":-3.4286537170410156,"334":-10.181886672973633,"42":-1.7092304229736328,"378":-7.997814178466797,"43":-0.3174552917480469,"340":-6.745379447937012,"44":-4.169615745544434,"388":-10.87108325958252,"45":-5.064406394958496,"407":-11.682621002197266,"46":-5.219514846801758,"410":-12.14154052734375}},"graph":{"mac_fp16":{"p":[0.03355569929945177,0.18692607282695622,0.7514412096387174,0.0159603547334232,0.0065320118209212215,0.005584651680530163],"max_abs_dp":0.00014221887680299083,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03356012289268143,0.18700549643365694,0.7515554374683582,0.015828346423381417,0.0065024195871665885,0.005548177194755535],"max_abs_dp":0.000136636363308823,"argmax_equal":true}}}]},{"id":"tv4x_emotion_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":93,"line_sha256":"fcc67cb9232066f5c0efc81a5af7ffe6ab662908e7fd74682da0fb4438908de3","_meta":{"row":180,"text_sha256":"1f9892f77e7ed6d783b6d8da629f00bea0e96eed3f3ff7b7d7bbe32132e7aac9","row_sha256":"07ed2fa62f9c4bff059a7aa9e7d11ebb2b9cefb212f24b8950ae3e79f6adfe61","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/180","group_id":"emotion/test/180","variant":"clean"}},"request_sha256":"e98a01838064d1a7d7fddb5d914a7deadd7381ea0de3a5f15df02fb2959f918b","questions":[{"type":"choice","n_options":6,"row_len":75,"slot":74,"row_ids_sha256":"539e20b0f339d2468cd94366649ef4fd95f4493485ca4ae83b97d05bd7845e98","oracle":{"probs":[0.34997827691230454,0.013720142988036409,0.01068081607478991,0.003784221334822333,0.6021749467152021,0.019661595974844747],"argmax":"fear","top2_margin":0.25219666980289757,"near_tie":false,"group_logz":{"41":-1.0683021545410156,"334":-8.568883895874023,"42":-4.307308197021484,"378":-11.723503112792969,"43":-4.5577239990234375,"340":-12.01540756225586,"44":-5.595333099365234,"388":-13.262225151062012,"45":-0.5256252288818359,"407":-7.504579544067383,"46":-3.9475059509277344,"410":-11.190690040588379}},"graph":{"mac_fp16":{"p":[0.3491504255209412,0.013731110667876594,0.010681164805390882,0.0037807854984965215,0.6030649850747024,0.01959152843259245],"max_abs_dp":0.0008900383595003003,"argmax_equal":true},"iphone_int8mlp":{"p":[0.34827128617503406,0.013700761603033929,0.010577334715846754,0.003739924391786553,0.604561089578899,0.01914960353539984],"max_abs_dp":0.002386142863696872,"argmax_equal":true}}}]},{"id":"tv4x_emotion_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":94,"line_sha256":"6b49bef9f0105a819f693070629a18a876b0cbf37d4351c5d62d47e29b2fa5c0","_meta":{"row":1333,"text_sha256":"a5eb09875a8231481d14d26be9641d79af8f6515ea3c66d798dbd9188a5294d0","row_sha256":"08da79fda3a7a400c05c8070bd8424133272f026de32fbea20525eab986c845f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1333","group_id":"emotion/test/1333","variant":"clean"}},"request_sha256":"9b12098335342d424fad59f2344c4f7f933cca1b431520b177101b41f0b1efa7","questions":[{"type":"choice","n_options":6,"row_len":54,"slot":53,"row_ids_sha256":"625af67f49b6ed8e97cafb080548d20d6b602257d366bf36d37315c3e05c2d63","oracle":{"probs":[0.07814742183773565,0.1183307347861432,0.7889464363659957,0.006128362042521285,0.004590106184632311,0.0038569387829718747],"argmax":"love","top2_margin":0.6706157015798525,"near_tie":false,"group_logz":{"41":-2.5703659057617188,"334":-9.696737289428711,"42":-2.1554794311523438,"378":-8.862953186035156,"43":-0.25826454162597656,"340":-7.163623809814453,"44":-5.116035461425781,"388":-12.367786407470703,"45":-5.405059814453125,"407":-12.525016784667969,"46":-5.579089164733887,"410":-12.909978866577148}},"graph":{"mac_fp16":{"p":[0.07825856389813197,0.11838417825379695,0.7887837573061012,0.00612525607903029,0.004590728354430325,0.0038575161085093158],"max_abs_dp":0.00016267905989453268,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07791791207984093,0.11912579984563512,0.7884091132288671,0.0060903365472410285,0.004599922825769031,0.0038569154726467198],"max_abs_dp":0.0007950650594919229,"argmax_equal":true}}}]},{"id":"tv4x_emotion_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":95,"line_sha256":"dc65c90792da86deaf9bc0c625ea0bdb9bb2d9721950dd3baba5c9c3e271e796","_meta":{"row":254,"text_sha256":"3504fdaa8caeae7312ea8043b6b1aca152a3b60668c977b418fb2238a0b67456","row_sha256":"0cdffbc03f8cf7c3a8869ab0159b89aa4c7460f30b36bdeaf1a398e0475bff7f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/254","group_id":"emotion/test/254","variant":"clean"}},"request_sha256":"31ad2c424443e7f1a7f617bc7f4c0ebf6a1960de04ebc31d16191ebe33fcb994","questions":[{"type":"choice","n_options":6,"row_len":59,"slot":58,"row_ids_sha256":"b3ff7fe0e50999e6621dbfd49ae790d4d77e109288c97e2ab2a99ccf2513ab7e","oracle":{"probs":[0.008325036328658577,0.8102708364788606,0.1725034547585385,0.003391464771726754,0.0022809990051203457,0.003228208657095228],"argmax":"joy","top2_margin":0.637767381720322,"near_tie":false,"group_logz":{"41":-4.8054704666137695,"334":-12.177266120910645,"42":-0.2273693084716797,"378":-7.140392303466797,"43":-1.7743206024169922,"340":-8.433207511901855,"44":-5.7034759521484375,"388":-12.994872093200684,"45":-6.100124359130859,"407":-13.147401809692383,"46":-5.752810478210449,"410":-13.067371368408203}},"graph":{"mac_fp16":{"p":[0.00832480233087558,0.8105743935076404,0.17221974991339284,0.003384567497543242,0.0022759200964105404,0.0032205666541373387],"max_abs_dp":0.0003035570287798217,"argmax_equal":true},"iphone_int8mlp":{"p":[0.008220473998410974,0.8106236180035942,0.17237809332635254,0.0033244000988052846,0.0022651987777513594,0.003188215795085586],"max_abs_dp":0.0003527815247336141,"argmax_equal":true}}}]},{"id":"tv4x_emotion_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":96,"line_sha256":"fd6f4f97823bd2549ad52b736f778251cb93dc02f907a136a16361620e08cfe2","_meta":{"row":1401,"text_sha256":"166ab5bc6613551e8ad7e2fcbd87a91c45d1fde27d0c2e39aa6c74a7463d13cc","row_sha256":"0d130863dde313fe1ebd0f91e6dacf58ff43921c9bd180d33e24a96e7f7012ca","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1401","group_id":"emotion/test/1401","variant":"clean"}},"request_sha256":"a4fcde72a9d3488fe6bd4e59976af0461ad41b29a0a2407fce4a4e4aec92d07e","questions":[{"type":"choice","n_options":6,"row_len":65,"slot":64,"row_ids_sha256":"82617ec05412b57162dcb72dbb8f7a38c34cd9c23383215afb6167e3325fe7e3","oracle":{"probs":[0.9464718168823741,0.011036677991079349,0.016914797808708762,0.016213245684502105,0.005482707639330039,0.0038807539940056653],"argmax":"sadness","top2_margin":0.9295570190736654,"near_tie":false,"group_logz":{"41":-0.0660247802734375,"334":-8.300142288208008,"42":-4.517541885375977,"378":-12.529264450073242,"43":-4.090577125549316,"340":-11.996960639953613,"44":-4.132937431335449,"388":-11.940580368041992,"45":-5.217166900634766,"407":-13.3439302444458,"46":-5.562736511230469,"410":-13.650472640991211}},"graph":{"mac_fp16":{"p":[0.9462764067775663,0.011069132092112317,0.01695932467075175,0.016295087957284137,0.005502089703853302,0.003897958798432194],"max_abs_dp":0.00019541010480783605,"argmax_equal":true},"iphone_int8mlp":{"p":[0.946295765118087,0.011053134834254297,0.01703779458831242,0.01623829287180035,0.005506602685249723,0.00386840990229612],"max_abs_dp":0.00017605176428714397,"argmax_equal":true}}}]},{"id":"tv4x_emotion_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":97,"line_sha256":"141332b5ea5753e8dbf6e08229ff45a3a3c009852875886869a5faa3b777ac48","_meta":{"row":1025,"text_sha256":"6e30edf5e02d8cf4daf79f0498e8c75658975419d72d4de78abd43fce0c12c9f","row_sha256":"0d5648bb81f5516a5b3241e749701d3f6e6b0f74fde6d8546526a7e2ae0da18a","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1025","group_id":"emotion/test/1025","variant":"clean"}},"request_sha256":"08a69503d474f86a040805c921509bbaca9fab9cba8880d838f7fbaaf109cc9e","questions":[{"type":"choice","n_options":6,"row_len":73,"slot":72,"row_ids_sha256":"d9ef54af7fecddd8dadd6bd2a00edd4991f77e1ca7d15c96708f792fb5599773","oracle":{"probs":[0.6749184907457748,0.03133612311042861,0.10999736133903877,0.05060768768529289,0.11640550599487534,0.016734831124589702],"argmax":"sadness","top2_margin":0.5585129847508995,"near_tie":false,"group_logz":{"41":-0.427459716796875,"334":-7.513975143432617,"42":-3.4972801208496094,"378":-10.395986557006836,"43":-2.2415952682495117,"340":-8.869704246520996,"44":-3.0179481506347656,"388":-9.72728443145752,"45":-2.184971809387207,"407":-8.811652183532715,"46":-4.12455940246582,"410":-11.055152893066406}},"graph":{"mac_fp16":{"p":[0.6750586964514771,0.03129018815964046,0.10980225232664043,0.05072760675239486,0.11639826382669366,0.016722992483153523],"max_abs_dp":0.0001951090123983451,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6753322128693388,0.03127381185310924,0.1110876886288384,0.04988477564158012,0.11577653841197086,0.016644972595162587],"max_abs_dp":0.0010903272897996308,"argmax_equal":true}}}]},{"id":"tv4x_emotion_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":98,"line_sha256":"9aabd9db23490e4666c081b9c5040e52e1d6de50be315fe1778a277c872dfb26","_meta":{"row":1576,"text_sha256":"a3432a86c14e8fd2af65a2054d6f2b7b506dfd7b0fce0688701d813831942802","row_sha256":"0d969a10c0b2fe52d83ee11f899fbb5a0afec0690faca0c9dd67e77e0c208001","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1576","group_id":"emotion/test/1576","variant":"clean"}},"request_sha256":"55cfb5c2d951fda3eeb5cf9b61ee8733477c540c5417d8d00dba0f8825273ea5","questions":[{"type":"choice","n_options":6,"row_len":83,"slot":82,"row_ids_sha256":"870bec848abcccac38ce317c64d28d34db13a8ccf58ceb97e1dcef0367c415df","oracle":{"probs":[0.9296337318587908,0.010492410466485448,0.006037547682893971,0.01921561540646918,0.027664179559973627,0.006956515025386958],"argmax":"sadness","top2_margin":0.9019695522988173,"near_tie":false,"group_logz":{"41":-0.08697128295898438,"334":-8.011026382446289,"42":-4.571109771728516,"378":-12.39616584777832,"43":-5.1237640380859375,"340":-13.006752014160156,"44":-3.966038703918457,"388":-11.463810920715332,"45":-3.60162353515625,"407":-11.177523612976074,"46":-4.982083320617676,"410":-12.662673950195312}},"graph":{"mac_fp16":{"p":[0.929670402477949,0.010477521524204593,0.006027137746728612,0.019190530801342812,0.02767646785051393,0.006957939599261058],"max_abs_dp":3.667061915813008e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9298946139893757,0.010506269555048095,0.006015719232995259,0.019278724570249015,0.027461222536906388,0.0068434501154254835],"max_abs_dp":0.00026088213058483856,"argmax_equal":true}}}]},{"id":"tv4x_emotion_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":99,"line_sha256":"92ce35de5890de6c9f04f4595aa2fb6ea4ee61826887748f6dc13131ae0ec594","_meta":{"row":1022,"text_sha256":"1ddfafd83f06f7e23367fa275286cb4073ee3492459e8b893a15f320097224fe","row_sha256":"0dfffcf7f68dcd3b88fb7bf4eeb31f74e4f958e82c5eae9c104b8a36f10ba9d0","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1022","group_id":"emotion/test/1022","variant":"clean"}},"request_sha256":"d41a7adb5d94b0ed3f03d4f795b99086acb137de5a7053b2287ab524628943a2","questions":[{"type":"choice","n_options":6,"row_len":90,"slot":89,"row_ids_sha256":"7621578e371bedfdf4736624f10a6474d2b5f4366684e39cc414ede3a221d2a9","oracle":{"probs":[0.9304533969679818,0.020799498751689096,0.010175997207511911,0.018667287934466852,0.015780926485994794,0.004122892652355679],"argmax":"sadness","top2_margin":0.9096538982162927,"near_tie":false,"group_logz":{"41":-0.09098243713378906,"334":-7.705439567565918,"42":-3.891725540161133,"378":-11.364679336547852,"43":-4.606622695922852,"340":-12.1939697265625,"44":-3.9998817443847656,"388":-11.310835838317871,"45":-4.167852401733398,"407":-11.591985702514648,"46":-5.510099411010742,"410":-13.10831069946289}},"graph":{"mac_fp16":{"p":[0.9305998039836597,0.020768917989030435,0.010165246234605838,0.01862722640228562,0.015719026353586796,0.004119779036831665],"max_abs_dp":0.00014640701567791847,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9304142764219354,0.021123628164348294,0.010199739461957884,0.018504271692054385,0.015652953722944177,0.004105130536759753],"max_abs_dp":0.00032412941265919787,"argmax_equal":true}}}]},{"id":"tv4x_qnli_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":240,"line_sha256":"026b2c652020e71da46a5bc0d0affb87eca91429a3222eed26e15da098ccf788","_meta":{"row":1714,"text_sha256":"9845373b64960d3bca56c5c34aad2173004e49bb0ed8582d1aadc7ef9b6e19a8","row_sha256":"006ac40be6adb5b4046eb22f837b50966d8668fdcfdffd816563e7764bcb939d","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1714","group_id":"qnli/validation/1714","variant":"clean"}},"request_sha256":"c3d42eff2f6a3740b8b23098f8d3853dec781af7981770bad44aca64311b4030","questions":[{"type":"noul","n_options":2,"row_len":105,"slot":104,"row_ids_sha256":"9f650204b638a352fbc66403d164b28b99107286822700d60b7eb085d9b999c6","oracle":{"probs":[0.9838380389015631,0.016161961098437056],"argmax":"yes","top2_margin":0.967676077803126,"near_tie":false,"group_logz":{"11683":-0.03155708312988281,"12447":-4.470935821533203,"2243":-4.140357971191406,"4547":-8.078400611877441,"19598":-10.668298721313477}},"graph":{"mac_fp16":{"p":[0.9838642172222498,0.016135782777750053],"max_abs_dp":2.617832068700296e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9838779251926246,0.016122074807375302],"max_abs_dp":3.9886291061753953e-05,"argmax_equal":true}}}]},{"id":"tv4x_qnli_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":241,"line_sha256":"4e3cd7e0bf182d394d7142751f88fbbb3f98438c84437a9257cdb813b6b31e13","_meta":{"row":1900,"text_sha256":"27f0f252e66e0fac367b7361abfe1202b4b8f2f8905a8e843912e9ef3d4e8a84","row_sha256":"011c2da09bec40ee37ae0ab36f27fcedc599855eb59c4d52adb1a98352d98cd3","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1900","group_id":"qnli/validation/1900","variant":"clean"}},"request_sha256":"c90a4efa4eb9e6dc7cd6200c945ea1fcf41b02a4947123cb30f670222537719e","questions":[{"type":"noul","n_options":2,"row_len":105,"slot":104,"row_ids_sha256":"b62a740f7bc7e49eec19aeadbda848d5026ac269c829cb033b59c5712e8fd692","oracle":{"probs":[0.09344167820061339,0.9065583217993867],"argmax":"no","top2_margin":0.8131166435987733,"near_tie":false,"group_logz":{"11683":-2.414876937866211,"12447":-5.887236595153809,"2243":-0.14255905151367188,"4547":-3.545938491821289,"19598":-7.630428314208984}},"graph":{"mac_fp16":{"p":[0.09353695065629974,0.9064630493437001],"max_abs_dp":9.527245568652365e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0928313025900324,0.9071686974099676],"max_abs_dp":0.0006103756105809849,"argmax_equal":true}}}]},{"id":"tv4x_qnli_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":242,"line_sha256":"bd455f97bcd67715884ff4b8a9e03a10619606703668caf5e906eb7f34e8d1f7","_meta":{"row":2575,"text_sha256":"5e02a62572edafc14ce561623edf67e95b3b87fdd8029257eb585e419e0749a1","row_sha256":"020a27fa0285b3515c44f4ce37e48ebbff851a1859113e5ff3f3977fd6be5b7e","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2575","group_id":"qnli/validation/2575","variant":"clean"}},"request_sha256":"22738c3327dc9fe788aa38c80a455bc6966b5287829292b9bca560933790803c","questions":[{"type":"noul","n_options":2,"row_len":67,"slot":66,"row_ids_sha256":"b0abf2ab8f6351aa685cecdb35ee70e72c4530c3903a2e400a3b9e9ce87398ba","oracle":{"probs":[0.9056582178500582,0.09434178214994172],"argmax":"yes","top2_margin":0.8113164357001165,"near_tie":false,"group_logz":{"11683":-0.1186065673828125,"12447":-5.018804550170898,"2243":-2.3803443908691406,"4547":-7.002664566040039,"19598":-9.610243797302246}},"graph":{"mac_fp16":{"p":[0.9058053803287651,0.09419461967123485],"max_abs_dp":0.00014716247870694144,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9050256703731953,0.09497432962680487],"max_abs_dp":0.0006325474768631434,"argmax_equal":true}}}]},{"id":"tv4x_qnli_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":243,"line_sha256":"999801858a8041dc7107120264a0fdcad28e4c137dd14666e85bd6d578b6e1d9","_meta":{"row":1998,"text_sha256":"e1e3ab832833b63a93997f399656af5edae92f9b8fbb7700c39754e7df3c4e92","row_sha256":"022665c82b545868033098220fff3ba296025e2f926e2cb9ce7f77402579a052","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1998","group_id":"qnli/validation/1998","variant":"clean"}},"request_sha256":"4097f1de0c5826b4ffdcac4287d891adf4caa2e6140edcfce5743e2b786e6f41","questions":[{"type":"noul","n_options":2,"row_len":74,"slot":73,"row_ids_sha256":"936a912ae7324b7f7d58b525fc15456c02ffe33755636206a94773a438a0bd30","oracle":{"probs":[0.056640194433663624,0.9433598055663365],"argmax":"no","top2_margin":0.8867196111326728,"near_tie":false,"group_logz":{"11683":-2.911914825439453,"12447":-6.416194915771484,"2243":-0.09918594360351562,"4547":-3.538665771484375,"19598":-7.83254337310791}},"graph":{"mac_fp16":{"p":[0.056556175072249124,0.9434438249277509],"max_abs_dp":8.401936141450017e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.05747374520123872,0.9425262547987614],"max_abs_dp":0.0008335507675750961,"argmax_equal":true}}}]},{"id":"tv4x_qnli_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":244,"line_sha256":"31bcd9dae7e5eaa6675e35613bd6293986eebabc829b6873ce3229a12975f449","_meta":{"row":4117,"text_sha256":"7d7d122f20925140db28dd81b60bffb3e59ca6587ad47a6f8ddca0083ed3d0ea","row_sha256":"022dedc816cb845ec1ce5b0606af8270a0a54e5cceeae4217ac0efe1b93fc996","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4117","group_id":"qnli/validation/4117","variant":"clean"}},"request_sha256":"cba2eb3e50163ba7a1873e99074b2855df9bab706970445c584f7e766ead5963","questions":[{"type":"noul","n_options":2,"row_len":157,"slot":156,"row_ids_sha256":"a8e3ab6dfd3b47b26585103725d398c3d8e6447e6b28caa12bb7ab922edb032e","oracle":{"probs":[0.8799477778839534,0.12005222211604658],"argmax":"yes","top2_margin":0.7598955557679068,"near_tie":false,"group_logz":{"11683":-0.16594505310058594,"12447":-4.018255233764648,"2243":-2.1578807830810547,"4547":-5.5924577713012695,"19598":-8.984505653381348}},"graph":{"mac_fp16":{"p":[0.8803366016758992,0.11966339832410083],"max_abs_dp":0.00038882379194582484,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8797647092270884,0.12023529077291165],"max_abs_dp":0.0001830686568650708,"argmax_equal":true}}}]},{"id":"tv4x_qnli_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":245,"line_sha256":"4521f9acf0235d82529b59b7c1255faef76d4e2877c896f543157352d2f523dc","_meta":{"row":3930,"text_sha256":"1140e42273d8c9d12f7eea7c5227dee7cbf302068f77c87fa21f7a4e2a78fe2f","row_sha256":"025fc0affd878c088f092574c86ffff48cc7bc65c60a9252b783a5c641fcbf19","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3930","group_id":"qnli/validation/3930","variant":"clean"}},"request_sha256":"206ce87596e38c4b5937ec235e5dcbb4479d765a311568ba7d0dcd7128f29cb3","questions":[{"type":"noul","n_options":2,"row_len":90,"slot":89,"row_ids_sha256":"377a26c434c21aed172a4b0a58acc23ffc79c61857d1d29c2c20ce90b3b208ec","oracle":{"probs":[0.6302178783792297,0.3697821216207703],"argmax":"yes","top2_margin":0.2604357567584594,"near_tie":false,"group_logz":{"11683":-0.5042915344238281,"12447":-4.636475563049316,"2243":-1.0374431610107422,"4547":-4.972195625305176,"19598":-8.5222806930542}},"graph":{"mac_fp16":{"p":[0.6305909042160059,0.36940909578399395],"max_abs_dp":0.0003730258367763417,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6336776184261109,0.36632238157388897],"max_abs_dp":0.003459740046881321,"argmax_equal":true}}}]},{"id":"tv4x_qnli_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":246,"line_sha256":"9acda97976df3d8d0bc8d2092b2f6759c736e5304706b6f85902219e564c350c","_meta":{"row":5417,"text_sha256":"665140834a5729a9df029f9f5e07f1611018081116d85af5bf5bfbc638d639e8","row_sha256":"028ee36d7d2f94d0c48a2c94946cddbe624d57d570c501ecb0bf5ab83a3616fe","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/5417","group_id":"qnli/validation/5417","variant":"clean"}},"request_sha256":"09946f001ece271ce414ec17ee82d7106f8839d4e3707643adbfa92a491bfca3","questions":[{"type":"noul","n_options":2,"row_len":72,"slot":71,"row_ids_sha256":"f95fd46c8c99b0c14b1b8234966229557d19606376a3667dca7a5ca6e95bb461","oracle":{"probs":[0.867479086942279,0.13252091305772096],"argmax":"yes","top2_margin":0.7349581738845581,"near_tie":false,"group_logz":{"11683":-0.18145179748535156,"12447":-3.9442148208618164,"2243":-2.060302734375,"4547":-5.586132049560547,"19598":-8.905374526977539}},"graph":{"mac_fp16":{"p":[0.8677417875076019,0.13225821249239816],"max_abs_dp":0.0002627005653228487,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8679427461064257,0.13205725389357428],"max_abs_dp":0.0004636591641467014,"argmax_equal":true}}}]},{"id":"tv4x_qnli_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":247,"line_sha256":"26892773625aa4d3757d7a27497f653c9564a10ddd67c69f424d45a1a0db00b5","_meta":{"row":182,"text_sha256":"5aa8e116927f50cb4d36a21dfb4bd20b369f3bbda41b163546571a75d9c26aa6","row_sha256":"028ff272234176332316caf860273564fb760bdef0a6d4af1472efec657ddbf1","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/182","group_id":"qnli/validation/182","variant":"clean"}},"request_sha256":"24343ff0b2cf6c923b52ddb7f7a38b42ac86844206f8e1503e9b6d64934e0888","questions":[{"type":"noul","n_options":2,"row_len":64,"slot":63,"row_ids_sha256":"938c3ddc335180a7f64552e9806de610d75b64ca89bdc1fe49994a09cf30bf91","oracle":{"probs":[0.530005121336844,0.46999487866315603],"argmax":"yes","top2_margin":0.06001024267368793,"near_tie":false,"group_logz":{"11683":-0.6956119537353516,"12447":-4.421283721923828,"2243":-0.8157768249511719,"4547":-4.59486198425293,"19598":-8.082953453063965}},"graph":{"mac_fp16":{"p":[0.5292136660969512,0.47078633390304886],"max_abs_dp":0.0007914552398928243,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5288932506464922,0.4711067493535078],"max_abs_dp":0.0011118706903517905,"argmax_equal":true}}}]},{"id":"tv4x_qnli_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":248,"line_sha256":"7b88d8b2a861449a9515a72431bece435f0ee13a0b0629205777b927bbd90ce5","_meta":{"row":416,"text_sha256":"8b83c32333787aea3ae32141afadc2b8ba4d2b2e339362a161bd335ac007116d","row_sha256":"035a3573cc069f9e593d74cab4fe4385d061ea858ef2e3c92ee735923f3797aa","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/416","group_id":"qnli/validation/416","variant":"clean"}},"request_sha256":"f4817eaa03ced2d1826e5079758e546682a3d11e5a8ba0da68435a4b69a0e4f2","questions":[{"type":"noul","n_options":2,"row_len":88,"slot":87,"row_ids_sha256":"a4365f0607fce55d6af3567d6a1b0d766faa9aa24723fa69a632cece993945a7","oracle":{"probs":[0.10114587635061495,0.898854123649385],"argmax":"no","top2_margin":0.79770824729877,"near_tie":false,"group_logz":{"11683":-2.3150997161865234,"12447":-6.407008171081543,"2243":-0.13054275512695312,"4547":-4.291243553161621,"19598":-8.151061058044434}},"graph":{"mac_fp16":{"p":[0.10138773390673757,0.8986122660932624],"max_abs_dp":0.00024185755612261828,"argmax_equal":true},"iphone_int8mlp":{"p":[0.10178841679358465,0.8982115832064153],"max_abs_dp":0.0006425404429697412,"argmax_equal":true}}}]},{"id":"tv4x_qnli_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":249,"line_sha256":"7f5468eac1330f8f9c4484ebc62cec2ab62db4777ab94ffcdb96e3d7abb0cea1","_meta":{"row":2695,"text_sha256":"35ec3f161936ec2c98c6f8e0107e2d489fb892440f08271624b9a6cb4a15a003","row_sha256":"038c9f863ce04d7beee4ee19a6bcb0ac3adbe0fa4a39204e13eabf91d0897954","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2695","group_id":"qnli/validation/2695","variant":"clean"}},"request_sha256":"5f6900dea8724ae926ebf049c59011888f3d5ce73161a4e175f57ecbfc006c92","questions":[{"type":"noul","n_options":2,"row_len":66,"slot":65,"row_ids_sha256":"a33bbb7c9632da6e8a04fed407682243ab58b7f9b7d5ded655325baf3a141064","oracle":{"probs":[0.02558798355796915,0.9744120164420309],"argmax":"no","top2_margin":0.9488240328840618,"near_tie":false,"group_logz":{"11683":-3.685649871826172,"12447":-7.778419494628906,"2243":-0.04593849182128906,"4547":-4.184423446655273,"19598":-8.288270950317383}},"graph":{"mac_fp16":{"p":[0.02558446717699003,0.97441553282301],"max_abs_dp":3.516380979121797e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02560363881502024,0.9743963611849797],"max_abs_dp":1.5655257051161442e-05,"argmax_equal":true}}}]},{"id":"tv4x_qnli_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":250,"line_sha256":"193c3f84c070f0b0e3972f7520c539982431ecf24aafd725aa1993ef3b6f48aa","_meta":{"row":3379,"text_sha256":"7dba31276b7e6cf3363d040279a05471c77cba50efd16cb0c38f23589b7c748a","row_sha256":"03aea18d3d6cfd05c282a386c17b876674e7951a3e20a1b0a8fa3a530d6ab0cb","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3379","group_id":"qnli/validation/3379","variant":"clean"}},"request_sha256":"827e16945155b9664be3463cf041ef9f1a1205f198e2d2629a7823fb4c34f9f8","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"010ed916e9750ca8915dbf6f99e214447abf698c5995ff1d5f70b07d71f51477","oracle":{"probs":[0.016537061802824018,0.983462938197176],"argmax":"no","top2_margin":0.966925876394352,"near_tie":false,"group_logz":{"11683":-4.1120452880859375,"12447":-8.81496810913086,"2243":-0.026569366455078125,"4547":-4.978481292724609,"19598":-8.698589324951172}},"graph":{"mac_fp16":{"p":[0.01651395870097978,0.9834860412990202],"max_abs_dp":2.3103101844237917e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01668489679641478,0.9833151032035853],"max_abs_dp":0.00014783499359076122,"argmax_equal":true}}}]},{"id":"tv4x_qnli_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":251,"line_sha256":"f62cd3900a04ab320462180d05cf914ac0dc06541509b742f29f8f443ec135cb","_meta":{"row":4802,"text_sha256":"8afb77221b7ace20f32e4ecf513b5b903de006034311072ca64fa5c45c304d83","row_sha256":"042f68db744ac90ad8be2bb7b59bba1d5d1e1d3092cd11ee6a90fc1e05e243c6","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4802","group_id":"qnli/validation/4802","variant":"clean"}},"request_sha256":"0b36f15408523c33913040ccc5fb70575aad412e2593baa2d87e5c7a20d38e3f","questions":[{"type":"noul","n_options":2,"row_len":59,"slot":58,"row_ids_sha256":"e95b8b5e559d465a6de6d8474353b386b9796d11a092f22a962ae391a69b3d7e","oracle":{"probs":[0.9541278199846663,0.04587218001533376],"argmax":"yes","top2_margin":0.9082556399693326,"near_tie":false,"group_logz":{"11683":-0.06829643249511719,"12447":-4.364400863647461,"2243":-3.1032352447509766,"4547":-7.026505470275879,"19598":-9.628480911254883}},"graph":{"mac_fp16":{"p":[0.9541816393020169,0.045818360697983],"max_abs_dp":5.381931735076034e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9548802722933686,0.045119727706631266],"max_abs_dp":0.0007524523087024948,"argmax_equal":true}}}]},{"id":"tv4x_qnli_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":252,"line_sha256":"abe59d3e8161c2bbd8742e91b39e11bfee6889b22b43a46eccfdcea02393ba05","_meta":{"row":1028,"text_sha256":"ad9ce13c16c3a55ee54267faae9122c64ba4aea143fb6ef493355ae74a22f1d2","row_sha256":"0445460bb33270740ff1d3b49dd52a808d7d0eadb94231c0647e40b14caccc65","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1028","group_id":"qnli/validation/1028","variant":"clean"}},"request_sha256":"03d1952a17a21066afcdbb2224018fc8f454930d62a89e9ff189d1ada28ddc83","questions":[{"type":"noul","n_options":2,"row_len":96,"slot":95,"row_ids_sha256":"1515261588c41ae47ff50cee6575a6efaf400386a36106823da719bd87abb1a6","oracle":{"probs":[0.6897408932927008,0.3102591067072992],"argmax":"yes","top2_margin":0.3794817865854016,"near_tie":false,"group_logz":{"11683":-0.43524169921875,"12447":-3.813608169555664,"2243":-1.2341499328613281,"4547":-4.73554801940918,"19598":-8.199624061584473}},"graph":{"mac_fp16":{"p":[0.6903015423977142,0.30969845760228587],"max_abs_dp":0.000560649105013411,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6888797025550946,0.31112029744490544],"max_abs_dp":0.0008611907376062655,"argmax_equal":true}}}]},{"id":"tv4x_qnli_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":253,"line_sha256":"06d5f3c913543c9b47f80b37f15b7c8d4dd678314553766e9d5f94f3a089760f","_meta":{"row":1106,"text_sha256":"b22065dbd979956c062f2c27052ffbd3f408f3ab54ad3d943e955ab0dc87843f","row_sha256":"04820391cd042455592727f727b3e59250d725e0de10975a6105e863154454ec","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1106","group_id":"qnli/validation/1106","variant":"clean"}},"request_sha256":"be4dce088f51d0d153ff680e2bf40da510cafca761ce416b92c1f1e0fb22908b","questions":[{"type":"noul","n_options":2,"row_len":67,"slot":66,"row_ids_sha256":"e9a8eb0fb6c49329f5c42aba4c78f900b1681639afd26b3ef1601b2990bfa750","oracle":{"probs":[0.8879671628100777,0.1120328371899224],"argmax":"yes","top2_margin":0.7759343256201553,"near_tie":false,"group_logz":{"11683":-0.14796829223632812,"12447":-4.405337333679199,"2243":-2.218111038208008,"4547":-6.204156875610352,"19598":-8.914298057556152}},"graph":{"mac_fp16":{"p":[0.8879441330822403,0.11205586691775983],"max_abs_dp":2.3029727837436953e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8861075243070152,0.11389247569298477],"max_abs_dp":0.0018596385030624774,"argmax_equal":true}}}]},{"id":"tv4x_qnli_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":254,"line_sha256":"d388e078e32a136ec940c98aa90e8a419893168384b1468bd5cdae01a6d00345","_meta":{"row":1945,"text_sha256":"371cbdf5201d1559b62f91d2bdab6ad2d2104d7ac5f4d3db87bce3b9f4e964a6","row_sha256":"04dd278821eadceac2c61ff4f7488c30b9e4c18a2eb4b007a1f485da9ffaf9c2","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1945","group_id":"qnli/validation/1945","variant":"clean"}},"request_sha256":"404029ec5ea8d7ac3ff5fb4da3c9d6c9ef1845c3415a206435b86a6f742ce049","questions":[{"type":"noul","n_options":2,"row_len":84,"slot":83,"row_ids_sha256":"87f1c628623f543f07514ad0f6bad9dcdeff69a7e7334ce07f2547c0ad76cf9b","oracle":{"probs":[0.8652027289696156,0.13479727103038447],"argmax":"yes","top2_margin":0.7304054579392312,"near_tie":false,"group_logz":{"11683":-0.17016220092773438,"12447":-4.553023338317871,"2243":-2.0293540954589844,"4547":-6.326663970947266,"19598":-9.107070922851562}},"graph":{"mac_fp16":{"p":[0.8650884260444361,0.13491157395556383],"max_abs_dp":0.00011430292517944363,"argmax_equal":true},"iphone_int8mlp":{"p":[0.864906219118117,0.13509378088188304],"max_abs_dp":0.00029650985149856646,"argmax_equal":true}}}]},{"id":"tv4x_qnli_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":255,"line_sha256":"c3d9ab4bfed0b7e219fb577457ee2c688b439e2be9b34dcd2f6bded0481ed259","_meta":{"row":1749,"text_sha256":"ab21fd233b1d5a05e03dc9990df89cbceb3f1032a41c991231a917d6e602ba38","row_sha256":"056b7d64e157d80325e7e80c24a55671bf9d7b6022e3eb26087a49d991f39548","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1749","group_id":"qnli/validation/1749","variant":"clean"}},"request_sha256":"808e96873c92cdeb04466f63770757ff29d94d9dff03bb1b91a78194b486575e","questions":[{"type":"noul","n_options":2,"row_len":102,"slot":101,"row_ids_sha256":"96cf55723c435583ff73cb9a166a79c2011fcae12dc588919ef1eb9c91251872","oracle":{"probs":[0.9668905824713627,0.03310941752863717],"argmax":"yes","top2_margin":0.9337811649427256,"near_tie":false,"group_logz":{"11683":-0.05720329284667969,"12447":-4.151742935180664,"2243":-3.4314708709716797,"4547":-6.945878982543945,"19598":-10.058866500854492}},"graph":{"mac_fp16":{"p":[0.9668745280953612,0.03312547190463876],"max_abs_dp":1.605437600159243e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9671423613259578,0.032857638674042254],"max_abs_dp":0.0002517788545950417,"argmax_equal":true}}}]},{"id":"tv4x_qnli_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":256,"line_sha256":"23c7c59f3ffbf9477adde0e6048a831cedeec7f5d7f3ffd418ab1c28bcadedd2","_meta":{"row":5119,"text_sha256":"da65511d9914134b1b614c8881820ae01d37cccc093bc155540bddc0a5b257d5","row_sha256":"05abda4d70d457b761fea0ff9c7fc6a5ccfa8d1dac548841905ccedec7e1e283","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/5119","group_id":"qnli/validation/5119","variant":"clean"}},"request_sha256":"7bfcdefb6328bdbd8bde0c3339858cb065737828a16eada63052090a4a58ac09","questions":[{"type":"noul","n_options":2,"row_len":84,"slot":83,"row_ids_sha256":"bb079b0315fe345aabb4127187a7bb87fa82951be10a62c354ab9bfd48560965","oracle":{"probs":[0.7190192356929093,0.28098076430709074],"argmax":"yes","top2_margin":0.43803847138581853,"near_tie":false,"group_logz":{"11683":-0.3562488555908203,"12447":-5.206169128417969,"2243":-1.2958507537841797,"4547":-5.981653213500977,"19598":-8.905420303344727}},"graph":{"mac_fp16":{"p":[0.7193799540554495,0.28062004594455053],"max_abs_dp":0.0003607183625402577,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7179465190216732,0.28205348097832694],"max_abs_dp":0.0010727166712362068,"argmax_equal":true}}}]},{"id":"tv4x_qnli_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":257,"line_sha256":"2219a725d563c3a67d1f24fd3c036412106b7bce6a71f8939a353c092d94e967","_meta":{"row":4476,"text_sha256":"7228a5f445d5d35b480913bdd3c648476a6e5743ba2d8ce524c8c864316ca8c2","row_sha256":"06010ded8e680dae0e0b8eec96d4a90a90d3f7ea17897dd4468d683b49f94dc1","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4476","group_id":"qnli/validation/4476","variant":"clean"}},"request_sha256":"537e93af78801a603571f6eae24e2f838843fd742c935accece02517b2015c8d","questions":[{"type":"noul","n_options":2,"row_len":90,"slot":89,"row_ids_sha256":"673afe6b2107eb57257617e53be1662e176b307308822b61c95d2c7242883cab","oracle":{"probs":[0.9605541372901587,0.039445862709841295],"argmax":"yes","top2_margin":0.9211082745803174,"near_tie":false,"group_logz":{"11683":-0.06174278259277344,"12447":-4.272796630859375,"2243":-3.254323959350586,"4547":-6.877861022949219,"19598":-9.891407012939453}},"graph":{"mac_fp16":{"p":[0.9605409336040405,0.0394590663959596],"max_abs_dp":1.3203686118304359e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9606989289025797,0.039301071097420266],"max_abs_dp":0.00014479161242102884,"argmax_equal":true}}}]},{"id":"tv4x_qnli_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":258,"line_sha256":"b90b940516c641be814682defc7e84695b4252ce2b4a8c78f7808c87ed4bde1e","_meta":{"row":2679,"text_sha256":"91255f5369751974f74ec78c071ea2fb8bfda176031828a6e6af872db6f95ac1","row_sha256":"060ce662495649b49fc2ff4109d4713187bba4cd8685a357c77a0234f4d1e26a","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2679","group_id":"qnli/validation/2679","variant":"clean"}},"request_sha256":"f97e55f2118d08cdbbbcd72e5a1028efc03e6392882edcfd5d53830f76b71d34","questions":[{"type":"noul","n_options":2,"row_len":54,"slot":53,"row_ids_sha256":"df205a4d2a9c0ede381b8790b358b9ae61d800b26f7f820621d96117dd7e221b","oracle":{"probs":[0.07380180556818587,0.9261981944318142],"argmax":"no","top2_margin":0.8523963888636283,"near_tie":false,"group_logz":{"11683":-2.6350364685058594,"12447":-6.753408432006836,"2243":-0.1053314208984375,"4547":-4.211617469787598,"19598":-8.191120147705078}},"graph":{"mac_fp16":{"p":[0.07385719741225531,0.9261428025877447],"max_abs_dp":5.53918440694412e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07603642389813839,0.9239635761018616],"max_abs_dp":0.0022346183299525313,"argmax_equal":true}}}]},{"id":"tv4x_qnli_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":259,"line_sha256":"59631df8f1f52d9611d3fe1d357caa04ba5d3df1ac928c7bdc534083ab7ca779","_meta":{"row":4070,"text_sha256":"5e1c285ed6a50a364098eeca9e5780115daa7cfb5bc9a9d2f51f006bd6a80033","row_sha256":"0639e94bbf5f42ba589154f51cb10fdf3e39f6abea72fff9db4d59d28eb92533","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4070","group_id":"qnli/validation/4070","variant":"clean"}},"request_sha256":"78705e8bfb6a47240bd2de2924b473c4f23c2cf3c2281f629944223c32c253c2","questions":[{"type":"noul","n_options":2,"row_len":102,"slot":101,"row_ids_sha256":"ada087decfa2daa66164aed04889130c4c84120da0696a1dade4697b5a1afdb7","oracle":{"probs":[0.8517180903187406,0.14828190968125943],"argmax":"yes","top2_margin":0.7034361806374811,"near_tie":false,"group_logz":{"11683":-0.179840087890625,"12447":-5.320247650146484,"2243":-1.9279804229736328,"4547":-6.802043914794922,"19598":-9.573732376098633}},"graph":{"mac_fp16":{"p":[0.8518913500303487,0.1481086499696514],"max_abs_dp":0.00017325971160808695,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8492035387496425,0.15079646125035753],"max_abs_dp":0.0025145515690981057,"argmax_equal":true}}}]},{"id":"tv4x_paws_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":320,"line_sha256":"d7978ad13641a8bbaf1871f2d7e5ad396e7e34ee02eaabbe4a0c680459e4aa41","_meta":{"row":856,"text_sha256":"b39024f1673d530e60fd0d97fa23dd8dfd7450e969715e17a362906e4e4ae287","row_sha256":"00807b6c80dc3f42f08d9dea69d9f943b048b6c5da3d8136157d9872c54d18e2","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/856","group_id":"paws/test/856","variant":"clean"}},"request_sha256":"0d8502c45005aa9900186d55ac88bab19338845fa7f522297e39bc53c9a05c28","questions":[{"type":"noul","n_options":2,"row_len":99,"slot":98,"row_ids_sha256":"1ade4e377e892a97bf04c602833f30afd0caad090cad8714ae33f3c2c318185c","oracle":{"probs":[0.08962906736312783,0.9103709326368722],"argmax":"no","top2_margin":0.8207418652737444,"near_tie":false,"group_logz":{"11683":-2.4284019470214844,"12447":-7.497211456298828,"2243":-0.1102294921875,"4547":-5.41181755065918,"19598":-8.0919189453125}},"graph":{"mac_fp16":{"p":[0.08982889024633645,0.9101711097536634],"max_abs_dp":0.00019982288320874453,"argmax_equal":true},"iphone_int8mlp":{"p":[0.08875337971759,0.9112466202824101],"max_abs_dp":0.0008756876455379148,"argmax_equal":true}}}]},{"id":"tv4x_paws_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":321,"line_sha256":"5bea2f352f447831591a52edb5e7610eb090b97029b052052405e7e147605dbc","_meta":{"row":2686,"text_sha256":"cdbfeff1f36776ebeccc77fca491ca4a854315378393b5883e67c710d8e1aae6","row_sha256":"01c37641efdb2bc558fcf5c870c7a77c26dd5ece7458d4a70da998b9a6ce600a","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2686","group_id":"paws/test/2686","variant":"clean"}},"request_sha256":"3df6c7fc51a21cb478c190309fc728dc14949e13b6d317c7e0e6a32865aa1dec","questions":[{"type":"noul","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"5903d09127eca883febdef3a7bc521efc519c2da030991f67f2e9ef5e8c0eaef","oracle":{"probs":[0.5369816344852568,0.46301836551474324],"argmax":"yes","top2_margin":0.07396326897051358,"near_tie":false,"group_logz":{"11683":-0.6684341430664062,"12447":-5.6151885986328125,"2243":-0.8166313171386719,"4547":-5.576801300048828,"19598":-8.102577209472656}},"graph":{"mac_fp16":{"p":[0.5375721306882253,0.4624278693117747],"max_abs_dp":0.0005904962029685268,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5382085812953378,0.4617914187046622],"max_abs_dp":0.0012269468100810088,"argmax_equal":true}}}]},{"id":"tv4x_paws_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":322,"line_sha256":"4890c0cfa4c04aeb6e4c026e61cfa2161876248afd898761f5adff69a942a5e8","_meta":{"row":4712,"text_sha256":"fc460bd5c072efdbf42379535d10451ba8e5babae251aab671dfe6d4d963f6e9","row_sha256":"0253c7c4a41cf8d64770b7e74a3fda4c265f663de0acb195275af4a9dca07196","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4712","group_id":"paws/test/4712","variant":"clean"}},"request_sha256":"46d03ae1c4b72df2dedf802f20247b1c6d574aa5fd53458127c7171a7b4d2ce5","questions":[{"type":"noul","n_options":2,"row_len":116,"slot":115,"row_ids_sha256":"a065dc0811710f5e614c86a67560a051155c03659c3c53d77c95881d356c8bfb","oracle":{"probs":[0.029176914271635178,0.9708230857283648],"argmax":"no","top2_margin":0.9416461714567297,"near_tie":false,"group_logz":{"11683":-3.5487117767333984,"12447":-7.938507080078125,"2243":-0.0439453125,"4547":-4.706940650939941,"19598":-8.028658866882324}},"graph":{"mac_fp16":{"p":[0.02918163631680719,0.9708183636831927],"max_abs_dp":4.722045172100842e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02929627774886143,0.9707037222511384],"max_abs_dp":0.0001193634772264085,"argmax_equal":true}}}]},{"id":"tv4x_paws_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":323,"line_sha256":"803072f969be273c45840ed2812626359538bb1e2b65d2d015365d1e0e377517","_meta":{"row":3056,"text_sha256":"25a8f4bce60585921fc7bb50e0660c0849b8a20c10fcd9f7197460986be184ef","row_sha256":"03554f9384a8812787f16fc356bff6adf4002a872cdc69f10ce4a97928ce4bd9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3056","group_id":"paws/test/3056","variant":"clean"}},"request_sha256":"07f43028b31f04fc976520e6c152981f11c61453d6f1281dcb6438530e7e054f","questions":[{"type":"noul","n_options":2,"row_len":123,"slot":122,"row_ids_sha256":"c3e8b55e5866bb173a4cdff8d60865a3ecd75479f211d798f1986cd956abdec4","oracle":{"probs":[0.9765076182920791,0.023492381707920902],"argmax":"yes","top2_margin":0.9530152365841582,"near_tie":false,"group_logz":{"11683":-0.0314483642578125,"12447":-6.505998611450195,"2243":-3.7587547302246094,"4547":-9.806617736816406,"19598":-11.397977828979492}},"graph":{"mac_fp16":{"p":[0.976475955453422,0.023524044546577916],"max_abs_dp":3.166283865707609e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9766297132654226,0.02337028673457745],"max_abs_dp":0.0001220949733434784,"argmax_equal":true}}}]},{"id":"tv4x_paws_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":324,"line_sha256":"0aabce5f38b32ab620b13060f7679fb7e3041fe05d45e1192574cc229b253592","_meta":{"row":3990,"text_sha256":"efb066a292e49b3856fdb6ad9386a4002ab5685940c32dd10c116cad3fa166a8","row_sha256":"0396ad8f63e4a76587672c5b296a89c9cd22a9665a3e7dccefc3016d33b4f1ce","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3990","group_id":"paws/test/3990","variant":"clean"}},"request_sha256":"938bcf3402db964acc9722f98587e18f9c962ebc5273da01115d8757ce9f6034","questions":[{"type":"noul","n_options":2,"row_len":105,"slot":104,"row_ids_sha256":"c72d0b899835b594de946f6ee79f13c8307596d0492ab07146568f4cf4c8e776","oracle":{"probs":[0.9597293397718862,0.04027066022811385],"argmax":"yes","top2_margin":0.9194586795437724,"near_tie":false,"group_logz":{"11683":-0.05186271667480469,"12447":-6.173532485961914,"2243":-3.222890853881836,"4547":-8.936793327331543,"19598":-10.741933822631836}},"graph":{"mac_fp16":{"p":[0.9598273291540507,0.040172670845949415],"max_abs_dp":9.798938216443176e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9598285517017413,0.04017144829825868],"max_abs_dp":9.921192985516691e-05,"argmax_equal":true}}}]},{"id":"tv4x_paws_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":325,"line_sha256":"4decf072eb0f08000a429d636267cca1633ce0510d823f92052b7b2fda01ae21","_meta":{"row":529,"text_sha256":"c0a8c191543c31a01d1d7206f6177150e46e5078671c906d2c43a69e350be12f","row_sha256":"03a964f3ae534adfacfb83f756e93404c932ebc053ecbfd3e66e8a203775ecee","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/529","group_id":"paws/test/529","variant":"clean"}},"request_sha256":"3760aabf2be99abd5d04990759f95cdad0692df9ad4ec4fa525e2eaed4b5e230","questions":[{"type":"noul","n_options":2,"row_len":107,"slot":106,"row_ids_sha256":"bd80a77f5c763c657ddea014ee50b4588353fa0f60d483c88d2e8496d4ce8a30","oracle":{"probs":[0.6726144260218171,0.3273855739781829],"argmax":"yes","top2_margin":0.3452288520436342,"near_tie":false,"group_logz":{"11683":-0.4237995147705078,"12447":-5.853687286376953,"2243":-1.1438331604003906,"4547":-6.59089469909668,"19598":-8.888201713562012}},"graph":{"mac_fp16":{"p":[0.6722829892216802,0.32771701077831983],"max_abs_dp":0.00033143680013691945,"argmax_equal":true},"iphone_int8mlp":{"p":[0.678248578043272,0.32175142195672796],"max_abs_dp":0.005634152021454952,"argmax_equal":true}}}]},{"id":"tv4x_paws_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":326,"line_sha256":"92aa29ec229c1229638b4b65c7726fbf3e1dc8447bb2e407c46b308740d18085","_meta":{"row":116,"text_sha256":"e663c5b92e7a5cc9a1b2c6bcb939063a0295d062ed9c6d2933b5b513fdc5f3aa","row_sha256":"0417933bf84a525da6959a3548e9c533ce9d8655b267c55db8f6e423239fafa1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/116","group_id":"paws/test/116","variant":"clean"}},"request_sha256":"eaba9e8da848e11d8f936ad52f33568ff3383353b7bc55a2e29d004ba059d051","questions":[{"type":"noul","n_options":2,"row_len":112,"slot":111,"row_ids_sha256":"a50d4c7ea06c95d63cc5cdab80da8f0d327724c47069eed4b24a2cb6d51d634b","oracle":{"probs":[0.4207362520282527,0.5792637479717473],"argmax":"no","top2_margin":0.15852749594349458,"near_tie":false,"group_logz":{"11683":-0.9106082916259766,"12447":-5.795294761657715,"2243":-0.5908565521240234,"4547":-5.460338592529297,"19598":-8.045952796936035}},"graph":{"mac_fp16":{"p":[0.42008897447409055,0.5799110255259096],"max_abs_dp":0.0006472775541622733,"argmax_equal":true},"iphone_int8mlp":{"p":[0.42559892663247567,0.5744010733675243],"max_abs_dp":0.004862674604222961,"argmax_equal":true}}}]},{"id":"tv4x_paws_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":327,"line_sha256":"cd8b9d9fcc7ef707945c90d1246521918ca842244435a2f728fb8112c589f0a1","_meta":{"row":2414,"text_sha256":"5a17b5264afce263aeb32953a5d710f1cd45bf535d620989515c265f560b2181","row_sha256":"0455513e097769adc3aaefd350d551118caaecac19e6c47183293e3809388e4c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2414","group_id":"paws/test/2414","variant":"clean"}},"request_sha256":"c24d9de6dc810bab94b442c55891d935e8919b026cd5c6279c62e599a2ecc364","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"2eda0bc2cb732bfdd80719efdb5e496d96b7e69654ca08ecfc8328da180847bd","oracle":{"probs":[0.9788977510949393,0.021102248905060808],"argmax":"yes","top2_margin":0.9577955021898784,"near_tie":false,"group_logz":{"11683":-0.028818130493164062,"12447":-6.263824462890625,"2243":-3.865865707397461,"4547":-9.489706039428711,"19598":-11.201850891113281}},"graph":{"mac_fp16":{"p":[0.9788725002924785,0.021127499707521518],"max_abs_dp":2.5250802460741184e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9791625583193536,0.02083744168064642],"max_abs_dp":0.0002648072244143887,"argmax_equal":true}}}]},{"id":"tv4x_paws_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":328,"line_sha256":"bb8f320b79ef4b0200e5328443ed000dac54d585af300e5e59d3df6fae5f134e","_meta":{"row":2795,"text_sha256":"82dd4cf1090e4e6896653803a3291e25fe70e8edf60f86bcec183dc4eba36bbd","row_sha256":"04dc1933b0c811617487064c704c6d3487e7b9822fbe80fc85c4b9fed7f06ed7","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2795","group_id":"paws/test/2795","variant":"clean"}},"request_sha256":"d37d6a0e9cf71f54f8762214d60143f70b866371967cc6ef830a3bcbd8572bb4","questions":[{"type":"noul","n_options":2,"row_len":90,"slot":89,"row_ids_sha256":"3f0952091f726c022eb1505edeec646aa2f60b330db864426ff79f97800f3da1","oracle":{"probs":[0.9740491523269831,0.025950847673016825],"argmax":"yes","top2_margin":0.9480983046539663,"near_tie":false,"group_logz":{"11683":-0.036823272705078125,"12447":-6.208402633666992,"2243":-3.662080764770508,"4547":-9.376594543457031,"19598":-11.022549629211426}},"graph":{"mac_fp16":{"p":[0.9740416332831427,0.025958366716857334],"max_abs_dp":7.5190438405085824e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9741285476557465,0.025871452344253625],"max_abs_dp":7.939532876344302e-05,"argmax_equal":true}}}]},{"id":"tv4x_paws_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":329,"line_sha256":"752cbeee5c6a1326c45f82764097089378ec23e54532362e7e04bacf196f4a28","_meta":{"row":4273,"text_sha256":"0c10330e40010bb5442145d4e3b0042b75b8fcd7aa951e492ea08ddf187fd541","row_sha256":"0538754a83f520428a85681e700f292635f2560767b6f6f802ea2e86d87427e3","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4273","group_id":"paws/test/4273","variant":"clean"}},"request_sha256":"7c0b2c69c2f26e729cc2ca2fe6c048d29c538f9eee5614dabe22b2c22431133e","questions":[{"type":"noul","n_options":2,"row_len":140,"slot":139,"row_ids_sha256":"8c477716cce7161776e871ed7ffdd3cdbf03739758d3dfad21757fa43cb15082","oracle":{"probs":[0.9347014162785328,0.06529858372146731],"argmax":"yes","top2_margin":0.8694028325570654,"near_tie":false,"group_logz":{"11683":-0.08417892456054688,"12447":-6.254488945007324,"2243":-2.7454357147216797,"4547":-8.625347137451172,"19598":-10.425210952758789}},"graph":{"mac_fp16":{"p":[0.9346106193392443,0.06538938066075578],"max_abs_dp":9.079693928848442e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9353892174793321,0.06461078252066782],"max_abs_dp":0.0006878012007994899,"argmax_equal":true}}}]},{"id":"tv4x_paws_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":330,"line_sha256":"a1b8f87b5d633ca6f4629a549b178c7b67f617b157ef31ab567f6d7408d9d9d8","_meta":{"row":6236,"text_sha256":"4ae695c0138e1d65ee6f33aedf344c6527643060c363334abb66f32a95a60815","row_sha256":"0555f5b1b45405dbbd5811279db88ba1850d474ed716a30816a100ffe8934ac1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6236","group_id":"paws/test/6236","variant":"clean"}},"request_sha256":"e57419ffe8f229a63e7700e4cd1b7d586c2c10932d40c2b45c0bf88ec5fa2087","questions":[{"type":"noul","n_options":2,"row_len":95,"slot":94,"row_ids_sha256":"1c4fc2f9903c3bf7b43df8398c79fa0912ea4e9ece5739db2534ab2186d3ae6f","oracle":{"probs":[0.07138293691344835,0.9286170630865518],"argmax":"no","top2_margin":0.8572341261731035,"near_tie":false,"group_logz":{"11683":-2.655611038208008,"12447":-7.612464904785156,"2243":-0.08997344970703125,"4547":-5.302050590515137,"19598":-8.23068618774414}},"graph":{"mac_fp16":{"p":[0.07132441115742151,0.9286755888425785],"max_abs_dp":5.852575602684207e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0728793235133221,0.9271206764866778],"max_abs_dp":0.0014963865998739445,"argmax_equal":true}}}]},{"id":"tv4x_paws_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":331,"line_sha256":"28e6c82df5777d2e1540f9ce1b0bfa8aefb7dcbe8d1c6339f2789a3fb7d9ab3a","_meta":{"row":1125,"text_sha256":"e075a882bf75d143e262395005d4c9f23dd624839239c751861f485f5e914734","row_sha256":"05692822d545561130a006ccb671d0266b60879047c2459fa8308d90320ae5e3","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1125","group_id":"paws/test/1125","variant":"clean"}},"request_sha256":"5ea06ee7fb8e33e180a215e2e9b5b4b648c2365a89a21db3405df8e478231d91","questions":[{"type":"noul","n_options":2,"row_len":110,"slot":109,"row_ids_sha256":"82b8f0088dd473dfb07824dbcf732de4ca362d33fcf6e08ee9d9c8869c6e3e5e","oracle":{"probs":[0.9597715583914812,0.04022844160851882],"argmax":"yes","top2_margin":0.9195431167829624,"near_tie":false,"group_logz":{"11683":-0.05546379089355469,"12447":-5.986591339111328,"2243":-3.2275848388671875,"4547":-8.863140106201172,"19598":-10.603376388549805}},"graph":{"mac_fp16":{"p":[0.9597485923204178,0.04025140767958232],"max_abs_dp":2.296607106349463e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9595944714835011,0.04040552851649897],"max_abs_dp":0.0001770869079801493,"argmax_equal":true}}}]},{"id":"tv4x_paws_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":332,"line_sha256":"cfff039c22d7d4c03cb0223ea3b2d686cfdba00bafeb96738bf955e128a3c766","_meta":{"row":3102,"text_sha256":"ba89f073ecf1fdcdbb239b309dc4b29f23f1df50db37a615d9947e9d4b749c59","row_sha256":"0601c0d70fe89e8132cc5430dbcf48c4ed76173df68d9e67c2e89318fbfcbaa9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3102","group_id":"paws/test/3102","variant":"clean"}},"request_sha256":"3e10c805de77a98c4bcf548619e229a775a41d66aca2f8e3626097479af70ee5","questions":[{"type":"noul","n_options":2,"row_len":100,"slot":99,"row_ids_sha256":"a4981a206346ca171471931e77def6713500869d8849052d93437cf828c109ce","oracle":{"probs":[0.9728778148030311,0.027122185196968733],"argmax":"yes","top2_margin":0.9457556296060624,"near_tie":false,"group_logz":{"11683":-0.03491973876953125,"12447":-6.664306640625,"2243":-3.614826202392578,"4547":-9.734756469726562,"19598":-11.36105728149414}},"graph":{"mac_fp16":{"p":[0.9728721800962837,0.02712781990371627],"max_abs_dp":5.634706747536383e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.972753861400124,0.027246138599876027],"max_abs_dp":0.0001239534029072939,"argmax_equal":true}}}]},{"id":"tv4x_paws_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":333,"line_sha256":"4a36abbccadb0abd52de91a695de9dbc920948234c4b65dd664e5bb63a2a045d","_meta":{"row":4325,"text_sha256":"b173fc9a6d018360204b3b606b878cedac8f25f2f97ec1465ccece0d58c7d695","row_sha256":"060adb7606bfe00f2da3af0b2abc1d9a594120a92cafadf8af5e9d433172999f","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4325","group_id":"paws/test/4325","variant":"clean"}},"request_sha256":"e23bce732a82d2f02bc9af308ac63d24cf0a28dab40e33582154f0c33081fc56","questions":[{"type":"noul","n_options":2,"row_len":132,"slot":131,"row_ids_sha256":"a8c58f683c9d617c6e6568181f9672db1b80bcff07358857f9afb14fc0285df5","oracle":{"probs":[0.7806321204615337,0.21936787953846631],"argmax":"yes","top2_margin":0.5612642409230675,"near_tie":false,"group_logz":{"11683":-0.28240394592285156,"12447":-5.459718704223633,"2243":-1.5517578125,"4547":-6.5101776123046875,"19598":-9.027320861816406}},"graph":{"mac_fp16":{"p":[0.7806166984346752,0.21938330156532482],"max_abs_dp":1.542202685855809e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7823246877034993,0.21767531229650075],"max_abs_dp":0.0016925672419655646,"argmax_equal":true}}}]},{"id":"tv4x_paws_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":334,"line_sha256":"1a43d63f2bae20e58f972d5b8dcafb9d2cd3262c15e4ed3c35d7b8a9a8185e81","_meta":{"row":2474,"text_sha256":"287b64cf4368977a32078e00b9bb44428641276bf8831998e7e86c28197669d4","row_sha256":"068cbf67767ac9c3764014e96b7ea646e8582833120700f3166b352a64ecdb33","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2474","group_id":"paws/test/2474","variant":"clean"}},"request_sha256":"e46eb200684e71fac266c21dd7d64221e4b037f86ecd345063de9de922b09ea8","questions":[{"type":"noul","n_options":2,"row_len":102,"slot":101,"row_ids_sha256":"3b3c8ee2a7b06070c627d7300d890a14df41a07aace6a77e74f3c3d25c687310","oracle":{"probs":[0.04198350181629207,0.9580164981837079],"argmax":"no","top2_margin":0.9160329963674159,"near_tie":false,"group_logz":{"11683":-3.179433822631836,"12447":-8.180953979492188,"2243":-0.051845550537109375,"4547":-5.550153732299805,"19598":-8.534322738647461}},"graph":{"mac_fp16":{"p":[0.04200431667191556,0.9579956833280844],"max_abs_dp":2.081485562355212e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.042584811649091815,0.9574151883509082],"max_abs_dp":0.0006013098327997471,"argmax_equal":true}}}]},{"id":"tv4x_paws_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":335,"line_sha256":"204234e14333a20843b5d85b278fe6e710ac3741f02f77a271159fa1953e53a8","_meta":{"row":1182,"text_sha256":"7d25b7f25bb15a92566f269a9c0f8bea5b36a4573027eabbd305eecc4868609c","row_sha256":"06ffc8dedb991a70b4010f2dec9af450598bf560ff1b3543204df8e9808ab72c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1182","group_id":"paws/test/1182","variant":"clean"}},"request_sha256":"c6e3054c9be24ee33c55e9971d320d52d1381f8fb03cb954e58c17b522ee6e22","questions":[{"type":"noul","n_options":2,"row_len":99,"slot":98,"row_ids_sha256":"bba5a79199440a39ed7c16bcd38fc919a71e0beca380a33d1758bf9848e5345a","oracle":{"probs":[0.9604625426352102,0.039537457364789856],"argmax":"yes","top2_margin":0.9209250852704204,"near_tie":false,"group_logz":{"11683":-0.052410125732421875,"12447":-6.256664276123047,"2243":-3.2425765991210938,"4547":-8.938382148742676,"19598":-10.908878326416016}},"graph":{"mac_fp16":{"p":[0.9604188297345336,0.03958117026546639],"max_abs_dp":4.371290067661793e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9610876780833568,0.03891232191664325],"max_abs_dp":0.000625135448146609,"argmax_equal":true}}}]},{"id":"tv4x_paws_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":336,"line_sha256":"fd110b85e3fb06f2b2b0bf729dc2f4099af7abc5f8dd6fc32a9a54306bfeea20","_meta":{"row":504,"text_sha256":"2e39c8cfc1bdc208756dc06f91477606f02eb689ed79e78bd27567fa00bcddfe","row_sha256":"0740e6c8ba70a47cd9f0000926f5ce2fcb2609119181c3f2ba3feb6f8326634f","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/504","group_id":"paws/test/504","variant":"clean"}},"request_sha256":"eae3660e80f193bfd4244e91c25fb60fb8233a76e24646434091a3d6c276efb6","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"4ade7db3d1bcee3ce9d492768fe5e3de81be6372fa08ad359eb822725be30c60","oracle":{"probs":[0.9743829434242138,0.025617056575786088],"argmax":"yes","top2_margin":0.9487658868484278,"near_tie":false,"group_logz":{"11683":-0.031261444091796875,"12447":-7.56682014465332,"2243":-3.6698074340820312,"4547":-10.669713020324707,"19598":-11.822896957397461}},"graph":{"mac_fp16":{"p":[0.9744385191706774,0.025561480829322702],"max_abs_dp":5.557574646353203e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9743630906381696,0.02563690936183037],"max_abs_dp":1.9852786044282505e-05,"argmax_equal":true}}}]},{"id":"tv4x_paws_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":337,"line_sha256":"8a47e7ccd120522fd03ed48e169592f00f630314b045718ed0f5e7af966a06ad","_meta":{"row":5828,"text_sha256":"655f415c50be03e8af2444637e0f8301b59dd81bbae377ca3cc73f28cbe07113","row_sha256":"0797b909b6f352589b88f6698648caafc779462616bbc174a27232600691c57d","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5828","group_id":"paws/test/5828","variant":"clean"}},"request_sha256":"e675cbda76f9243d895cea44a9b8e9a593ead887ed933d8d97c1ef2dbec60777","questions":[{"type":"noul","n_options":2,"row_len":95,"slot":94,"row_ids_sha256":"f1b01ff9825447839ca0cc7a65eab6684224f506e46655dc66b946cd861d98ee","oracle":{"probs":[0.5983264052591056,0.4016735947408945],"argmax":"yes","top2_margin":0.19665281051821115,"near_tie":false,"group_logz":{"11683":-0.547637939453125,"12447":-6.134782791137695,"2243":-0.9461345672607422,"4547":-6.457633018493652,"19598":-8.862045288085938}},"graph":{"mac_fp16":{"p":[0.5977864604602884,0.40221353953971156],"max_abs_dp":0.000539944798817249,"argmax_equal":true},"iphone_int8mlp":{"p":[0.599872848387551,0.40012715161244916],"max_abs_dp":0.0015464431284453184,"argmax_equal":true}}}]},{"id":"tv4x_paws_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":338,"line_sha256":"ec67d308b0f54902fa4e140675d51c7aec9f4b0f8cf7a4e22872af336a7e6580","_meta":{"row":6633,"text_sha256":"2bfe00279009ab77bf1263ca9e7a96fe542673b040ec466fd96bcfb6a15a46f9","row_sha256":"07edfe0d91a3190629f417abff85b410c7785b83cfe54a509fe650bdead2d7a4","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6633","group_id":"paws/test/6633","variant":"clean"}},"request_sha256":"a365d0dd7d8762af84e9efc4aead8c049bc648e6439151b82748d3eafb2669ce","questions":[{"type":"noul","n_options":2,"row_len":101,"slot":100,"row_ids_sha256":"80c01c8f44c743660a54abd48b8f262cc564642bbd467c11009409af6dd5e9f3","oracle":{"probs":[0.016670479213401253,0.9833295207865987],"argmax":"no","top2_margin":0.9666590415731975,"near_tie":false,"group_logz":{"11683":-4.107017517089844,"12447":-8.480697631835938,"2243":-0.029712677001953125,"4547":-4.731683731079102,"19598":-8.035316467285156}},"graph":{"mac_fp16":{"p":[0.01667185316140222,0.9833281468385978],"max_abs_dp":1.3739480009648808e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016758059731974174,0.9832419402680259],"max_abs_dp":8.758051857292101e-05,"argmax_equal":true}}}]},{"id":"tv4x_paws_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":339,"line_sha256":"d511731122bd9f502e10787dc1ae2295daded404b09c681cd64f4d5a845661ce","_meta":{"row":6102,"text_sha256":"30cf30c9dcfb4e1863f825ceda91572b0b98548220a3bd0be3ecaa52c096961f","row_sha256":"088b142ff5d38a084ee446facb1d90b967dedd418715c321fe367ebc78938ae1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6102","group_id":"paws/test/6102","variant":"clean"}},"request_sha256":"ca42b4d2b010167d9b284085a4df5135b57b1180de10fab3a770a0d82a489f78","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"ae5691c62e66eb47cfde5ae706a5279f41a3d605af965bc8cc3aa649aa848b04","oracle":{"probs":[0.9788307065169074,0.021169293483092624],"argmax":"yes","top2_margin":0.9576614130338148,"near_tie":false,"group_logz":{"11683":-0.029314041137695312,"12447":-6.355495452880859,"2243":-3.8631210327148438,"4547":-9.75361442565918,"19598":-11.39783763885498}},"graph":{"mac_fp16":{"p":[0.9787964626277187,0.021203537372281336],"max_abs_dp":3.424388918871202e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9789497852628141,0.021050214737185984],"max_abs_dp":0.00011907874590666445,"argmax_equal":true}}}]},{"id":"tv4x_sciq_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":400,"line_sha256":"4cdb2a949918db7cce1e281b6c27036971127eb8760d6ffb02568b3a255a3415","_meta":{"row":936,"text_sha256":"09202918cfc420e946051466763f6037d2b455dbf90d9b8fb9d79040e1047556","row_sha256":"00dd55f0b43115d6d413815d432e3b5f7dd9e15a3d341973d40cfc7c3b5defe4","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/936","group_id":"sciq/test/936","variant":"clean"}},"request_sha256":"229c3feae7b9f05ab04d7d5a7c54cd8ab51c20e477c49d60eef265498f65a961","questions":[{"type":"choice","n_options":4,"row_len":163,"slot":162,"row_ids_sha256":"5f6a7563d87fff25f545692a7dbacf4abd0a43a43923023a4572a789c4eae293","oracle":{"probs":[0.00143097816732928,0.9877404243217085,0.004034614155444402,0.006793983355517879],"argmax":"b","top2_margin":0.9809464409661907,"near_tie":false,"group_logz":{"73":-6.580230712890625,"267":-14.15575885772705,"74":-0.04316902160644531,"293":-7.901012420654297,"75":-5.543678283691406,"280":-13.055364608764648,"76":-5.022551536560059,"277":-12.335060119628906}},"graph":{"mac_fp16":{"p":[0.0014305394178400366,0.9877479077971338,0.004032769412257148,0.006788783372769041],"max_abs_dp":7.483475425340558e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0014267986359288625,0.9878156987944242,0.00401226999875992,0.006745232570886992],"max_abs_dp":7.527447271571397e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":401,"line_sha256":"ad7d08cf815560aee2cf9fc8b69961a2ad62f7ae481cf588c5ef0a2ff7cf32a5","_meta":{"row":925,"text_sha256":"3ca591694cd9b7c4e5d45687af0e57894ee1793a91164d5985918a7a91ef5c02","row_sha256":"013ea20efdab1c499fa2c6dfa4fdf58df6cdc0e11ffbbbd41f9cdfefbed6f00c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/925","group_id":"sciq/test/925","variant":"clean"}},"request_sha256":"9b79c99b1a38c41da76075154ef5886c0b9bbef0e8e6b19dbed124269352dba3","questions":[{"type":"choice","n_options":4,"row_len":71,"slot":70,"row_ids_sha256":"ac44ed967954d2556e630202e51e4d9e9349f0c966a4f7f86f199872dcd15443","oracle":{"probs":[0.011318849279913694,0.12255851539923843,0.042588880616681805,0.823533754704166],"argmax":"d","top2_margin":0.7009752393049276,"near_tie":false,"group_logz":{"73":-4.533307075500488,"267":-11.947271347045898,"74":-2.1511878967285156,"293":-9.233696937561035,"75":-3.2081832885742188,"280":-10.550670623779297,"76":-0.2461719512939453,"277":-7.311090469360352}},"graph":{"mac_fp16":{"p":[0.01131421566576461,0.12266758741083598,0.04260315694587208,0.8234150399775273],"max_abs_dp":0.00011871472663871341,"argmax_equal":true},"iphone_int8mlp":{"p":[0.011030936028754717,0.12215972437074786,0.04274735506267809,0.8240619845378192],"max_abs_dp":0.0005282298336531932,"argmax_equal":true}}}]},{"id":"tv4x_sciq_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":402,"line_sha256":"a3773044d1d5fd28f60fa0d98ef24e2d7316fe7ba20ca70d949ebeca47dcd6cd","_meta":{"row":269,"text_sha256":"7f83e7a09b5eb8a44209e77843fa327982ef4d485e7cb8ae2a3dde88eb18439e","row_sha256":"01db09ca82e091471fdca3a997ad6c4a6abc4979cffd033573cc6eaa9522e1cb","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/269","group_id":"sciq/test/269","variant":"clean"}},"request_sha256":"1c162b7ffdfd78e97fc4024b006b9283c73975345cbeb123121e65693a848c3f","questions":[{"type":"choice","n_options":4,"row_len":118,"slot":117,"row_ids_sha256":"0d0c75f9c2e53c15197944303ea4d4aba644f0bfffcbfa0326aa8fee2edc5dcf","oracle":{"probs":[0.9862972195585545,0.008587791870202435,0.0022607968831569202,0.0028541916880862276],"argmax":"a","top2_margin":0.9777094276883521,"near_tie":false,"group_logz":{"73":-0.03783416748046875,"267":-8.910972595214844,"74":-4.781450271606445,"293":-12.165817260742188,"75":-6.116074562072754,"280":-13.816699028015137,"76":-5.883003234863281,"277":-12.970968246459961}},"graph":{"mac_fp16":{"p":[0.9862829292042915,0.008599709158504585,0.0022630253163835354,0.0028543363208204777],"max_abs_dp":1.429035426303571e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9861149518693166,0.008730464693228215,0.002272245610900152,0.002882337826554856],"max_abs_dp":0.0001822676892379027,"argmax_equal":true}}}]},{"id":"tv4x_sciq_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":403,"line_sha256":"a6cefa8df233127f140cfa8fa8018d2a48f259364c5cdefc6cea612df5015903","_meta":{"row":258,"text_sha256":"9a0acb41813d28adadd7e5f2bc3f5f7d0019a90c6bdbed335de9f56425fdd42f","row_sha256":"0271ef05064f4cf27418547ced77b40684b58177241e24f80d14c36a269019a2","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/258","group_id":"sciq/test/258","variant":"clean"}},"request_sha256":"6a7ce184ed3d8ecacbb54cf9965e40397b49e165732879bcf21a6c1483cf850a","questions":[{"type":"choice","n_options":4,"row_len":109,"slot":108,"row_ids_sha256":"33bfc7d7e9d683213a0fdd780145dd0d3d0452813d098a122ef6f5095e00adc2","oracle":{"probs":[0.0014946823792268736,0.0038563680054306193,0.005023482561060665,0.9896254670542819],"argmax":"d","top2_margin":0.9846019844932212,"near_tie":false,"group_logz":{"73":-6.545413970947266,"267":-14.93433666229248,"74":-5.597601890563965,"293":-14.117450714111328,"75":-5.33320426940918,"280":-14.08782958984375,"76":-0.05000114440917969,"277":-9.025169372558594}},"graph":{"mac_fp16":{"p":[0.0014972049962084768,0.0038635274184795863,0.0050251470559652565,0.9896141205293466],"max_abs_dp":1.1346524935285096e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0014803166903850917,0.0038287137876856104,0.004987247167309059,0.9897037223546201],"max_abs_dp":7.825530033822581e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":404,"line_sha256":"c90e1f990474131f9fd7a5d468851c255921e60b0a489da651bf06e56bd27eea","_meta":{"row":46,"text_sha256":"406cd4f25d6709cacecb1f41876271eaf073801ecce8249da6e328a8abaffebb","row_sha256":"02874e7f33f2dadb8cf77157ad30509158c7a3e5bbfb249dcd369fb8b4a947c0","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/46","group_id":"sciq/test/46","variant":"clean"}},"request_sha256":"f1c6ba91d351af2b5c4c39fcec740d6835fded49716f577d1f58a2b705d17a23","questions":[{"type":"choice","n_options":4,"row_len":134,"slot":133,"row_ids_sha256":"020a4f0023246100c6b3b79afabce06d23ff5999a7740313605867ccbd897e51","oracle":{"probs":[0.0016795182225494583,0.9883860077819582,0.006382631359150079,0.0035518426363422228],"argmax":"b","top2_margin":0.9820033764228081,"near_tie":false,"group_logz":{"73":-6.418820381164551,"267":-14.17305850982666,"74":-0.04125404357910156,"293":-8.281150817871094,"75":-5.083746910095215,"280":-12.771036148071289,"76":-5.66986083984375,"277":-13.657729148864746}},"graph":{"mac_fp16":{"p":[0.0016774122084924244,0.9883928411398616,0.006380290776047058,0.0035494558755989067],"max_abs_dp":6.8333579034129954e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0016695319620468002,0.9883841803686901,0.006401317514535552,0.0035449701547275716],"max_abs_dp":1.8686155385473115e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":405,"line_sha256":"a6f649edcc1465c627551a7a1dd62151694d49865392406616ff9610e51317e6","_meta":{"row":670,"text_sha256":"d37ce9e469bb0c63715ef88cbbb7be115861e039ac45b1f9c9d3119c2e93ab5e","row_sha256":"0318d75b01d26524998217711113c87ac19d228d04d99c31afe8234882e94a0f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/670","group_id":"sciq/test/670","variant":"clean"}},"request_sha256":"624cf0955f4d7c82a9416caae8ba3e8025c9f71e8ba883f6ec79f55308cc103c","questions":[{"type":"choice","n_options":4,"row_len":86,"slot":85,"row_ids_sha256":"62502b9abd017b0f8713e24165f73eae704e8f23e4810a546238569af36b2743","oracle":{"probs":[0.9685975112949671,0.011460737771919777,0.01104689718593744,0.00889485374717574],"argmax":"a","top2_margin":0.9571367735230473,"near_tie":false,"group_logz":{"73":-0.06498527526855469,"267":-8.650235176086426,"74":-4.5019073486328125,"293":-12.057065963745117,"75":-4.538684844970703,"280":-12.26252555847168,"76":-4.755361557006836,"277":-11.564847946166992}},"graph":{"mac_fp16":{"p":[0.9686391980231353,0.011459617140709439,0.011019634657738376,0.008881550178416919],"max_abs_dp":4.1686728168177645e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9680848590750535,0.011619606041646822,0.011199914913439083,0.009095619969860665],"max_abs_dp":0.0005126522199135897,"argmax_equal":true}}}]},{"id":"tv4x_sciq_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":406,"line_sha256":"83af7dd3cec95f5fff65bcdb7c03de2fd2f4f846bba1113b13b7fd6a75ed8992","_meta":{"row":731,"text_sha256":"b705e7cbdf2e88ef234f0b163cfbcde607edf28974e4cc5bbb98544c6458e46f","row_sha256":"034547b8fa6238bee490397732c2ca0f662ccd5594e22b38eda01360c9a0a26a","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/731","group_id":"sciq/test/731","variant":"clean"}},"request_sha256":"79260eef44a98591df664f4a2ad590ae375d6402a4a75e7bb0a665f280982c78","questions":[{"type":"choice","n_options":4,"row_len":204,"slot":203,"row_ids_sha256":"15136197167fb7e07f09990da01d96b09899d0c6c11cb7970ad95385ee92e234","oracle":{"probs":[0.001213460570578227,0.9935010877392585,0.0029840012756856627,0.002301450414477661],"argmax":"b","top2_margin":0.9905170864635728,"near_tie":false,"group_logz":{"73":-6.734509468078613,"267":-14.91215705871582,"74":-0.026750564575195312,"293":-8.835100173950195,"75":-5.834720611572266,"280":-14.319990158081055,"76":-6.094446182250977,"277":-14.540639877319336}},"graph":{"mac_fp16":{"p":[0.001212276734080581,0.9935100936651083,0.002981065497869905,0.002296564102941132],"max_abs_dp":9.005925849847607e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0012099994417116948,0.9935370841996637,0.0029754491476947447,0.00227746721092989],"max_abs_dp":3.599646040519655e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":407,"line_sha256":"abe30fa93bb6ec2856d7f4c00a0d7e31ab9f52e3fd88fe02aa079f422716cce9","_meta":{"row":955,"text_sha256":"b3994ef4d3e0fd3fbe40ada60b67c11b6d9689a06c5679c6ed227899a93fe2e9","row_sha256":"035d44aeb1e246ed7b45f8091a77dc4cbd122786a6365d556fe8dd25d1d56633","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/955","group_id":"sciq/test/955","variant":"clean"}},"request_sha256":"c382b6acaa5e03fee8d46c29ec844f32a11d909ad7ae547d52073e0a3adb0382","questions":[{"type":"choice","n_options":4,"row_len":104,"slot":103,"row_ids_sha256":"13860772a63013e0f0373743c37f18b6251556f8bebe04c71b823eab34b3ccbf","oracle":{"probs":[0.0011290406942184325,0.9931774456667474,0.0034033555918099668,0.002290158047224322],"argmax":"b","top2_margin":0.9897740900749374,"near_tie":false,"group_logz":{"73":-6.8131866455078125,"267":-14.889242172241211,"74":-0.0336456298828125,"293":-8.67920207977295,"75":-5.7097930908203125,"280":-13.92779541015625,"76":-6.105934143066406,"277":-14.386617660522461}},"graph":{"mac_fp16":{"p":[0.0011261573521293166,0.9932006193149681,0.0033948012139491295,0.0022784221189535245],"max_abs_dp":2.3173648220686793e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0011286977898510451,0.9931885388715463,0.0033956617471991987,0.0022871015914034485],"max_abs_dp":1.1093204798973133e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":408,"line_sha256":"b8d698a0939d582c3fc95d43acf552d785ae3d0587504d72a2f67d11893bb43d","_meta":{"row":490,"text_sha256":"03ef14027d83f2bbb741b29c375fbaddfdb53ba7221da59eda4d7907699b2e1c","row_sha256":"039ab8cd042a271800e6c44a7f9f3f6e1c3be9bea2568c5024bed37cecc4ad22","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/490","group_id":"sciq/test/490","variant":"clean"}},"request_sha256":"689d5b13bdbc1daa02c3ac582b4aef7b45ded653275853e9c5866569c8b63443","questions":[{"type":"choice","n_options":4,"row_len":179,"slot":178,"row_ids_sha256":"494fb83c7b665c41a049b086bad39d4c6e6767a93e4c4254a28152a0446959ce","oracle":{"probs":[0.9897267768696575,0.005169810461194183,0.002660733575918384,0.0024426790932298],"argmax":"a","top2_margin":0.9845569664084634,"near_tie":false,"group_logz":{"73":-0.03255271911621094,"267":-9.401494979858398,"74":-5.287145614624023,"293":-13.170026779174805,"75":-5.951379776000977,"280":-14.26479721069336,"76":-6.036886215209961,"277":-13.537343978881836}},"graph":{"mac_fp16":{"p":[0.9897090519471201,0.005179560592221231,0.002666306408616232,0.002445081052042562],"max_abs_dp":1.7724922537376564e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9897669951905903,0.005125353522041281,0.0026593423428168416,0.0024483089445515548],"max_abs_dp":4.445693915290216e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":409,"line_sha256":"997b0ed41221127f1a948bddd14e0f85aa999d3c5d9215168f633804d8513f49","_meta":{"row":633,"text_sha256":"cf1e953f57b50c302376674da8534dc4ba1e6bf640905c7ddb206b9f1a988a35","row_sha256":"03a0482f6be75a66d516a3de837ca4ffc6f9346cac36bc8de3951eb67f6dcbd9","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/633","group_id":"sciq/test/633","variant":"clean"}},"request_sha256":"530f398b78f93ce23d5c581a5ca17e3e1f4eacdadd7729a99c0824201eed4f28","questions":[{"type":"choice","n_options":4,"row_len":148,"slot":147,"row_ids_sha256":"df9242e3e5f1a1c485e3d064831b92a87c786a723797d36344d3caa1c88781a3","oracle":{"probs":[0.0036274727383954347,0.005303998471252835,0.006223960575803716,0.984844568214548],"argmax":"d","top2_margin":0.9786206076387443,"near_tie":false,"group_logz":{"73":-5.659970283508301,"267":-13.694276809692383,"74":-5.280045509338379,"293":-13.154682159423828,"75":-5.120100021362305,"280":-13.362140655517578,"76":-0.05602264404296875,"277":-8.265037536621094}},"graph":{"mac_fp16":{"p":[0.00362465743433867,0.005293516649111899,0.006217855551881874,0.9848639703646677],"max_abs_dp":1.9402150119640638e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.003576851869364066,0.005250932572359501,0.0062027920250115436,0.9849694235332648],"max_abs_dp":0.00012485531871675448,"argmax_equal":true}}}]},{"id":"tv4x_sciq_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":410,"line_sha256":"1ac849f8360df25cc6433c580afca11423ae4f760ad41a1e05ab756f29b1a485","_meta":{"row":889,"text_sha256":"5f0f535f37629cada78055161524ded18f6d0fdba7163ccf28665d36bcb3c7fe","row_sha256":"03cc76d7b491ab3112d651f3a926efd30577affd480652d2a9e17b243dec867c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/889","group_id":"sciq/test/889","variant":"clean"}},"request_sha256":"b3a4d91ac4b5288d4daee63f1f9daec28aa5c19b3d7c07747b33b978fcceedf8","questions":[{"type":"choice","n_options":4,"row_len":98,"slot":97,"row_ids_sha256":"23ee3a2631a1627ef13e3f0904a4aa6f3d5f612669d8e244fc35788f7303394b","oracle":{"probs":[0.0032679763841249655,0.004977113193708355,0.004184607551224789,0.9875703028709418],"argmax":"d","top2_margin":0.9825931896772335,"near_tie":false,"group_logz":{"73":-5.766166687011719,"267":-14.150683403015137,"74":-5.345487594604492,"293":-13.846275329589844,"75":-5.518924713134766,"280":-14.141111373901367,"76":-0.05508995056152344,"277":-8.645147323608398}},"graph":{"mac_fp16":{"p":[0.0032641306239696633,0.004979768004639256,0.0041844040160491204,0.9875716973553419],"max_abs_dp":3.845760155302136e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.003235525496118858,0.0049303414988333735,0.004162129587611896,0.987672003417436],"max_abs_dp":0.0001017005464941656,"argmax_equal":true}}}]},{"id":"tv4x_sciq_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":411,"line_sha256":"6bb61c8043de5018ac28688718a6d2cd222121bb5518db481483dc69f57b90a7","_meta":{"row":291,"text_sha256":"5395c80c7508db20b33d4d1a9f78cc0acbba008ed35cbc07343ca98a1b5e9fba","row_sha256":"03e1c68453f4cc08d4794024d3a5cf1e411f357cb693b539383f020c45149fc1","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/291","group_id":"sciq/test/291","variant":"clean"}},"request_sha256":"a09d572273ee33c003b3a11715fb0c39982131f5a9b5ece5a4298d39f29a83f3","questions":[{"type":"choice","n_options":4,"row_len":378,"slot":377,"row_ids_sha256":"a861700d845bdf10ffbb3c38d96bc6f2178946424dfa6605bf9d4024456422cc","oracle":{"probs":[0.016783290838875618,0.007191180473263134,0.9728860041984629,0.0031395244893983655],"argmax":"c","top2_margin":0.9561027133595873,"near_tie":false,"group_logz":{"73":-4.113712310791016,"267":-11.890843391418457,"74":-4.961240768432617,"293":-12.810162544250488,"75":-0.053829193115234375,"280":-7.985836982727051,"76":-5.790024757385254,"277":-13.438405990600586}},"graph":{"mac_fp16":{"p":[0.01680891173802268,0.007212778331923517,0.9728307100963879,0.003147599833665973],"max_abs_dp":5.529410207505414e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016635511970714993,0.007067709457264438,0.9731920610563687,0.0031047175156519386],"max_abs_dp":0.00030605685790574544,"argmax_equal":true}}}]},{"id":"tv4x_sciq_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":412,"line_sha256":"f530797ad48a6fad9c87bb1cc8a1faf3cefefeb59ea9d073e01e9925aa3ac106","_meta":{"row":558,"text_sha256":"11a31672ec8b03161c35a52dd5d804506fa719826358f64986d4429ffc53c0d0","row_sha256":"04439e11e08fb71e5618a97d90d879814f5cdc5e8dc249a630a21a3c8bec4e74","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/558","group_id":"sciq/test/558","variant":"clean"}},"request_sha256":"49e65daf46da071a0edcdcc5b033211a51a36800987627c13c8d0c0926676f7f","questions":[{"type":"choice","n_options":4,"row_len":202,"slot":201,"row_ids_sha256":"d1f658e4c7be3a30959668465a841e46262596cd901041216a3ca76ce8acd215","oracle":{"probs":[0.0006940856784755691,0.00408653652514841,0.9901456586883549,0.005073719108021185],"argmax":"c","top2_margin":0.9850719395803337,"near_tie":false,"group_logz":{"73":-7.288808822631836,"267":-15.484533309936523,"74":-5.515951156616211,"293":-13.829168319702148,"75":-0.025796890258789062,"280":-8.556407928466797,"76":-5.299574851989746,"277":-13.033012390136719}},"graph":{"mac_fp16":{"p":[0.0006952405140343348,0.0040975515104979205,0.9901274696711425,0.0050797383043251895],"max_abs_dp":1.818901721239996e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0006896313343647874,0.004103588745822633,0.9901208784128576,0.0050859015069548786],"max_abs_dp":2.478027549734918e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":413,"line_sha256":"9c1c5c627fc9c633b2dc65af23922c421967cb07abdf273775d6a604bdce3d9a","_meta":{"row":712,"text_sha256":"15b7801931da9a1ca23589aee1a4ad7ad7d2cc2ae19f929a946248eb30517c84","row_sha256":"049cd85994b3f74ed4a086eefd408676a46826bdf9136728dcf1c06db493b008","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/712","group_id":"sciq/test/712","variant":"clean"}},"request_sha256":"4a6223dba56791f95ef37bff97934c8f4609c722bf9c7096be1fc37bba0f5804","questions":[{"type":"choice","n_options":4,"row_len":163,"slot":162,"row_ids_sha256":"d2c092f1f1510f4bf2637405ff39f93a96b075e7cbf49e3227f590de45c39666","oracle":{"probs":[0.9851315204652779,0.0070291595034418074,0.002969276869120191,0.004870043162160188],"argmax":"a","top2_margin":0.978102360961836,"near_tie":false,"group_logz":{"73":-0.0436248779296875,"267":-9.298261642456055,"74":-4.986332893371582,"293":-12.785228729248047,"75":-5.848081588745117,"280":-14.038614273071289,"76":-5.353297233581543,"277":-12.736499786376953}},"graph":{"mac_fp16":{"p":[0.9851683426322448,0.007010023736923892,0.002960590481675065,0.00486104314915626],"max_abs_dp":3.68221669669877e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9850066704092886,0.007106932957314175,0.0029857199663220085,0.004900676667075367],"max_abs_dp":0.00012485005598927845,"argmax_equal":true}}}]},{"id":"tv4x_sciq_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":414,"line_sha256":"53f1137167f7dd430a04dbb8af1b15104e3997063a225cbb39ddb350cb5cbd58","_meta":{"row":651,"text_sha256":"e44331cc4691a83e686fddd0238b1d13f60e6ffd856c7d256cd9a19667f9772a","row_sha256":"0604cb4d9a31f55964a2068967ac26bf90af1f8c9dad1047550013cf2c2ec7c8","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/651","group_id":"sciq/test/651","variant":"clean"}},"request_sha256":"d084555c59ca4437b48afe5c3e27f8690ab50de17361e53c971551fef8128966","questions":[{"type":"choice","n_options":4,"row_len":141,"slot":140,"row_ids_sha256":"b83d04cfe8d27ce1bb90796b73d450d2a5f0d12e41319e0fd423b2872350006a","oracle":{"probs":[0.0032827107985508733,0.005213807037884508,0.00646237988872331,0.9850411022748412],"argmax":"d","top2_margin":0.9785787223861179,"near_tie":false,"group_logz":{"73":-5.749311447143555,"267":-13.935068130493164,"74":-5.286670684814453,"293":-13.367679595947266,"75":-5.071983337402344,"280":-13.581670761108398,"76":-0.04529762268066406,"277":-8.36523151397705}},"graph":{"mac_fp16":{"p":[0.0032886264392608822,0.005217280058903498,0.006455206861740591,0.9850388866400951],"max_abs_dp":7.173026982719086e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0032204774264077652,0.005218223173245354,0.006456961970439747,0.9851043374299071],"max_abs_dp":6.323515506589938e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":415,"line_sha256":"f13977e4c4737c9bde54b9bf37ae5937aaa88586ecd4a63edb0643b2b1bdc2a8","_meta":{"row":524,"text_sha256":"5e99a00019fbcd39a4382e01e593dbba1dbd5c883a8d986bbe1c0cfbb08d6646","row_sha256":"0650d359b3b1912cd9d46e148ec5b9a5fc892b52edeaffcd00390d6910b026f8","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/524","group_id":"sciq/test/524","variant":"clean"}},"request_sha256":"12ed04706729805aabd36d929666f9295985a2e4d5d6b53a85d0d92cfc00ae6d","questions":[{"type":"choice","n_options":4,"row_len":277,"slot":276,"row_ids_sha256":"73a5cd3d9b25857b7997031abe67123f6c000c5ef1e5c7037c5f611fd3bc6c36","oracle":{"probs":[0.001162422806324112,0.992461207237681,0.00291102358575361,0.0034653463702411205],"argmax":"b","top2_margin":0.98899586086744,"near_tie":false,"group_logz":{"73":-6.780792236328125,"267":-14.892120361328125,"74":-0.031110763549804688,"293":-8.700943946838379,"75":-5.862793922424316,"280":-14.247782707214355,"76":-5.688486099243164,"277":-13.922109603881836}},"graph":{"mac_fp16":{"p":[0.0011582305553441668,0.9924906523295377,0.002901479683536364,0.0034496374315817037],"max_abs_dp":2.944509185665023e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0011589821376340112,0.9924712250197233,0.002915184543642354,0.0034546082990002643],"max_abs_dp":1.073807124085625e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":416,"line_sha256":"215b7b818df0e6f99fe7ab117cfdcdf0c2d42a92c8a5426cdc1714b270ae07c8","_meta":{"row":512,"text_sha256":"e4a0b75c4e5ecd5e9f29c31646003078448ef767e1d5c44d62b98f5cc1fbc6f6","row_sha256":"073a82faba43b6089d6d78bb990ae8b523fc90830aa80d3121fcde4c0486d880","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/512","group_id":"sciq/test/512","variant":"clean"}},"request_sha256":"70b6815acda68da4731614db310c9e3001cad05abeab58ee39533f5ecc4ab369","questions":[{"type":"choice","n_options":4,"row_len":352,"slot":351,"row_ids_sha256":"8cb323e0a2df9b5f833e4e305023506faf1d59ee116dc9bdb12de13a552ac595","oracle":{"probs":[0.04060002101460204,0.11981850880743954,0.08232178724425575,0.7572596829337026],"argmax":"d","top2_margin":0.6374411741262631,"near_tie":false,"group_logz":{"73":-3.270111083984375,"267":-11.079057693481445,"74":-2.187901496887207,"293":-9.445730209350586,"75":-2.563243865966797,"280":-10.03091049194336,"76":-0.3441734313964844,"277":-7.493202209472656}},"graph":{"mac_fp16":{"p":[0.04048897154085078,0.11949089433008747,0.08208741121887848,0.7579327229101832],"max_abs_dp":0.000673039976480605,"argmax_equal":true},"iphone_int8mlp":{"p":[0.040155561371839746,0.12676251740876116,0.0840403252330687,0.7490415959863304],"max_abs_dp":0.00821808694737225,"argmax_equal":true}}}]},{"id":"tv4x_sciq_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":417,"line_sha256":"91a6e6de806b639f0c5000aa408937ce00b26e8a958abc36c746cc4dc1f2b987","_meta":{"row":246,"text_sha256":"87779c312fe24a99cf39b66c82215a33948d2f96384705dffd8ff7062b58b9c9","row_sha256":"076a1b72202038496a7147eb57a5d3e7c492ada07fd06e79bf6f6074b25ca10f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/246","group_id":"sciq/test/246","variant":"clean"}},"request_sha256":"43cc941b445d01b3e9a477ee3d28811a2e35e644e64cd6f886f2b8c8889c0b58","questions":[{"type":"choice","n_options":4,"row_len":185,"slot":184,"row_ids_sha256":"e0f02fd3635b3b693737c6c55d27b36917d4064ad938f3ef1b436cf49d6a94a8","oracle":{"probs":[0.9859949131090663,0.008474868822392252,0.003814258409345053,0.0017159596591963925],"argmax":"a","top2_margin":0.977520044286674,"near_tie":false,"group_logz":{"73":-0.03561973571777344,"267":-9.288657188415527,"74":-4.792165756225586,"293":-12.373340606689453,"75":-5.590524673461914,"280":-13.388540267944336,"76":-6.389298439025879,"277":-13.647522926330566}},"graph":{"mac_fp16":{"p":[0.9860319880372161,0.008450382814689546,0.0038053775760888437,0.0017122515720055308],"max_abs_dp":3.7074928149860575e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9860462272305416,0.008440835036615043,0.003789387247985245,0.0017235504848581871],"max_abs_dp":5.131412147529879e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":418,"line_sha256":"4c649d8f9ec50e6a54ee1bbb7354352348a493b3e7362c8eb36bf720ee2a2971","_meta":{"row":844,"text_sha256":"a890e73029e550d303219f02577f9b6d5c521a3c4fdae7ee9843e487a9957dc9","row_sha256":"080069712d10fb8edd1f56f02ab5421d4d3f6e4530c42440f335904163cf9fdc","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/844","group_id":"sciq/test/844","variant":"clean"}},"request_sha256":"c2e3cbc0b76f526a66fe764f5ea119ef28dc4e6ca00ce1d9f18735f55570a00b","questions":[{"type":"choice","n_options":4,"row_len":78,"slot":77,"row_ids_sha256":"87cc3d3796f475b5d7b74717db44474122138a43f3f12147bf59794ad3f16e0d","oracle":{"probs":[0.9757712958577519,0.01008256919577707,0.007319299920960593,0.006826835025510437],"argmax":"a","top2_margin":0.9656887266619748,"near_tie":false,"group_logz":{"73":-0.04901885986328125,"267":-9.335384368896484,"74":-4.621438980102539,"293":-12.500779151916504,"75":-4.941732406616211,"280":-12.980079650878906,"76":-5.011385917663574,"277":-12.240154266357422}},"graph":{"mac_fp16":{"p":[0.9757854503709543,0.01006467998320064,0.007325679845160749,0.006824189800684276],"max_abs_dp":1.788921257642878e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9758579287551856,0.010057148252075133,0.007285117234347158,0.00679980575839203],"max_abs_dp":8.663289743371383e-05,"argmax_equal":true}}}]},{"id":"tv4x_sciq_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":419,"line_sha256":"2c35219562a8eaeb19a24eea306214cd036d361e9de0e44748706a85ce01b8ac","_meta":{"row":810,"text_sha256":"1a0b83d783e4e41ca3fbc13d015560c96917c1a361cdf7a3b8465840181b9cce","row_sha256":"08154610d80750177a3d6188d374164cf122f86b567cd3721c8d6d025747f98b","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/810","group_id":"sciq/test/810","variant":"clean"}},"request_sha256":"1f5b704d09d975f6f53becb642b9837d154e23309dacff91d7b735c8d2cce00b","questions":[{"type":"choice","n_options":4,"row_len":90,"slot":89,"row_ids_sha256":"6f3dfc42905bef3a4ea7f56238fb3f4a9b065f1d33d026cf9250455dcdff15a1","oracle":{"probs":[0.9942958074688091,0.002603214560938694,0.0017952735220418859,0.0013057044482102063],"argmax":"a","top2_margin":0.9916925929078704,"near_tie":false,"group_logz":{"73":-0.03274345397949219,"267":-9.563350677490234,"74":-5.978031158447266,"293":-14.247640609741211,"75":-6.349620819091797,"280":-14.927791595458984,"76":-6.668035507202148,"277":-14.399955749511719}},"graph":{"mac_fp16":{"p":[0.9942898602916022,0.0026070872759878247,0.001797890970792394,0.0013051614616175421],"max_abs_dp":5.94717720692195e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9943205148162627,0.0025896953130356954,0.001786148660183118,0.0013036412105184093],"max_abs_dp":2.470734745363412e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":588,"line_sha256":"b1b9890a119683f7704bb0d150e9a07d8939fd7c5402e36053c8b3c4bf394c8e","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0000","pair_id":"v3-legacy-transfer-authorization-0000","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0000","id":"contrastive/authorization/v3-legacy-transfer-authorization-0000/a","text_sha256":"c98a9666364a0d6e1490f365bb02551f8c08384c385be2cc2415203c56933e7f","row_sha256":"dbc61f5880a17d29f23b679f391879417818ec7732c8a1550582e5b340b416cb","group_id":"v3-legacy-transfer-authorization-0000","variant":"clean"}},"request_sha256":"21a2033c168461bef140bc6b71a1efa0b3e065a95b41b02f4cc93a922cc5ccc0","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"a6b0a10648d034c02ea9b434ad2aec650b8d43479fd4bec4b78e351d2071166f","oracle":{"probs":[0.9817914525719219,0.01820854742807806],"argmax":"yes","top2_margin":0.9635829051438438,"near_tie":false,"group_logz":{"11683":-0.028356552124023438,"12447":-5.193033218383789,"2243":-4.015844345092773,"4547":-8.592519760131836,"19598":-10.271533012390137}},"graph":{"mac_fp16":{"p":[0.9818137622848386,0.018186237715161403],"max_abs_dp":2.230971291672379e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9819611184484716,0.018038881551528472],"max_abs_dp":0.0001696658765496517,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":589,"line_sha256":"6a081290f35a0954199ccc450f7311d0df6d92866e28200810e71a0c00310a32","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0000","pair_id":"v3-legacy-transfer-authorization-0000","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0000","id":"contrastive/authorization/v3-legacy-transfer-authorization-0000/b","text_sha256":"e0b3e4ca4b46805650635de7381e7384fb45e77fc493303af3c59a1d5a2a351e","row_sha256":"476a73b46b9a452d10e7b216a9ea835497cf33d02a904b49f458d15435084670","group_id":"v3-legacy-transfer-authorization-0000","variant":"clean"}},"request_sha256":"4f9e54d132694e1791c02113db791d19c671a1e3eddefa87f3c690c63cb54003","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"47e36ab35e10f9fa4df69fdf63fc3d3f0a1c57a9607a548900db92be4843eb15","oracle":{"probs":[0.010194147463275025,0.9898058525367248],"argmax":"no","top2_margin":0.9796117050734497,"near_tie":false,"group_logz":{"11683":-4.590173721313477,"12447":-10.098020553588867,"2243":-0.014478683471679688,"4547":-5.947196006774902,"19598":-9.025171279907227}},"graph":{"mac_fp16":{"p":[0.010187632338390177,0.9898123676616098],"max_abs_dp":6.515124885031476e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010278400315132621,0.9897215996848674],"max_abs_dp":8.425285185759655e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":590,"line_sha256":"192bd78345eeb00c2402d9ce647aee89b60c4ffff7f43e16b46b2b3dde3000bf","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0001","pair_id":"v3-legacy-transfer-authorization-0001","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0001","id":"contrastive/authorization/v3-legacy-transfer-authorization-0001/a","text_sha256":"271eabac73e111da0e164e7e6e20159d34a804e80a51e6b8f89070689358ca68","row_sha256":"565d874a5f4062e94be13e8decce2b6ba07075ebfca990e7be6f60b5fc0906c8","group_id":"v3-legacy-transfer-authorization-0001","variant":"clean"}},"request_sha256":"ceef44c6f92e8d11ad7c4e6e752ea7e26fe91b363e487d2757232f5faf20bf2d","questions":[{"type":"noul","n_options":2,"row_len":95,"slot":94,"row_ids_sha256":"84625f57b1589b655999b050ba9d21de40f5f8476d2702ec30a2d3b4fda6872a","oracle":{"probs":[0.9782114717218285,0.02178852827817138],"argmax":"yes","top2_margin":0.9564229434436572,"near_tie":false,"group_logz":{"11683":-0.03121185302734375,"12447":-5.329541206359863,"2243":-3.8355541229248047,"4547":-8.611371994018555,"19598":-10.260334014892578}},"graph":{"mac_fp16":{"p":[0.9781945349972312,0.021805465002768972],"max_abs_dp":1.693672459759238e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9783920910954962,0.0216079089045038],"max_abs_dp":0.00018061937366764536,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":591,"line_sha256":"22f255d0212a39aa445031bee55288fdcb282425a7e14e665d1115079b21d6e3","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0001","pair_id":"v3-legacy-transfer-authorization-0001","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0001","id":"contrastive/authorization/v3-legacy-transfer-authorization-0001/b","text_sha256":"bec64bdb07d392ed48cb61c7bdc756a20a398395ab441ea59edd55447bed4a25","row_sha256":"de49096ff4ea76276f724473c358e2550e9cd571bad2c3c1863bcb021fb5202d","group_id":"v3-legacy-transfer-authorization-0001","variant":"clean"}},"request_sha256":"d55d2e7b0db66bc929afb4ea7ec95ae28b37d4533bbad0e8892e03c32f011aad","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"a8ace2381b4e11501980e6675bea9c6727c7d4b814454bd99f9369692ffc04e1","oracle":{"probs":[0.015060740443824218,0.9849392595561758],"argmax":"no","top2_margin":0.9698785191123516,"near_tie":false,"group_logz":{"11683":-4.201944351196289,"12447":-9.3618803024292,"2243":-0.021455764770507812,"4547":-5.483155250549316,"19598":-8.741315841674805}},"graph":{"mac_fp16":{"p":[0.01506529928646908,0.984934700713531],"max_abs_dp":4.558842644861866e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01506270450795332,0.9849372954920467],"max_abs_dp":1.9640641291118754e-06,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":592,"line_sha256":"78672942818279cd46c1c186d28a444815c43d8755a440805262ab1c9c26d32b","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0002","pair_id":"v3-legacy-transfer-authorization-0002","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0002","id":"contrastive/authorization/v3-legacy-transfer-authorization-0002/a","text_sha256":"574d7caa6a760ed64982106052c743f6376d2aa7b24326ae56be46a1a402c30c","row_sha256":"974650c9a1402f048b27152f2e4982bd2c77b7ab994b97abad81a5e5e76e0168","group_id":"v3-legacy-transfer-authorization-0002","variant":"clean"}},"request_sha256":"1dffaa4730a6365c90dff061db6aca9c27f2da64886314ab53b54eca42eb6905","questions":[{"type":"noul","n_options":2,"row_len":95,"slot":94,"row_ids_sha256":"1ee931256cdc002aad0ad86a1e385a39dac88dd8b6c10337accd844f96570d0d","oracle":{"probs":[0.981904312117697,0.01809568788230303],"argmax":"yes","top2_margin":0.963808624235394,"near_tie":false,"group_logz":{"11683":-0.02767181396484375,"12447":-5.262115478515625,"2243":-4.021492004394531,"4547":-8.673426628112793,"19598":-10.307236671447754}},"graph":{"mac_fp16":{"p":[0.9819162051447144,0.01808379485528558],"max_abs_dp":1.1893027017449709e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9819787577214059,0.018021242278594106],"max_abs_dp":7.44456037089225e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":593,"line_sha256":"43945587affbf39d501d8ecc7bb3dd819a9b2e2295b29ae1ffcd1ea8bb1166b8","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0002","pair_id":"v3-legacy-transfer-authorization-0002","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0002","id":"contrastive/authorization/v3-legacy-transfer-authorization-0002/b","text_sha256":"9033761e35fe5f8356f0d88559c800d1e36c84d38b260064d379cdacfdf1d902","row_sha256":"6266e4098a0564730723f608013b7cb52ebcaf6ad67720d848f44be3ab07fd81","group_id":"v3-legacy-transfer-authorization-0002","variant":"clean"}},"request_sha256":"57bff77107f8503f7e5544903f35e4a44c2ccc51eba38fa3a8dc9f46aad5a82b","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"fd6c6d3080146ef4c7e6ea7b04ea8f7dd56720a369378a6fe78555a9ea9a2ba0","oracle":{"probs":[0.01005066054458987,0.9899493394554101],"argmax":"no","top2_margin":0.9798986789108202,"near_tie":false,"group_logz":{"11683":-4.604658126831055,"12447":-10.001974105834961,"2243":-0.014642715454101562,"4547":-5.823759078979492,"19598":-8.936594009399414}},"graph":{"mac_fp16":{"p":[0.01004432779350936,0.9899556722064905],"max_abs_dp":6.332751080509805e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010032004611254456,0.9899679953887455],"max_abs_dp":1.865593333541353e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":594,"line_sha256":"e02d06f963ec8705c053f4d3faafcf96626362ed29180356ae3b783c44787a50","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0003","pair_id":"v3-legacy-transfer-authorization-0003","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0003","id":"contrastive/authorization/v3-legacy-transfer-authorization-0003/a","text_sha256":"90f7b4f1e4633c41259e676aa81856a60d04ca903b186a957046bf6eece8e6f1","row_sha256":"aae0096dd83398cf2389cfcf820f81ccb7f05f65c0c3ac9adafe36d78cf3fd13","group_id":"v3-legacy-transfer-authorization-0003","variant":"clean"}},"request_sha256":"3606766cfef424ed9ae6f1d1b6566fec43ef24de103454dca1be80f01d5e26e3","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"fa9cadc06f064e08b001b144cbae311fb14ae37c9e9ab62d97068f167ea7cb56","oracle":{"probs":[0.9654423672116438,0.03455763278835618],"argmax":"yes","top2_margin":0.9308847344232876,"near_tie":false,"group_logz":{"11683":-0.04519844055175781,"12447":-5.3742570877075195,"2243":-3.3751564025878906,"4547":-8.232758522033691,"19598":-9.893497467041016}},"graph":{"mac_fp16":{"p":[0.9654338597299321,0.034566140270067915],"max_abs_dp":8.507481711737819e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9634718864117157,0.03652811358828445],"max_abs_dp":0.001970480799928273,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":595,"line_sha256":"466febbd2f8af7013047682421928159a2e27e17415af39c20c97c1cb2b08502","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0003","pair_id":"v3-legacy-transfer-authorization-0003","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0003","id":"contrastive/authorization/v3-legacy-transfer-authorization-0003/b","text_sha256":"5cded64bb426ba6d4aa67e77405a9cd3d239a415437de0128e4e160e04867553","row_sha256":"4096ac0b5a6e3b84e8a9c9709ef3b22c0ea3b58e2d06e2305aa8dc3d10a7f691","group_id":"v3-legacy-transfer-authorization-0003","variant":"clean"}},"request_sha256":"ed55dd3917dfb0556f9fa7fc6f61f44d3763a25c8345dd593999589e5e93d6f5","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"731a27298cf558e867dc10bacd11a5ea3d96f40f9de418159abda4dae5eb8915","oracle":{"probs":[0.009533807881601845,0.990466192118398],"argmax":"no","top2_margin":0.9809323842367962,"near_tie":false,"group_logz":{"11683":-4.657718658447266,"12447":-10.14711856842041,"2243":-0.014387130737304688,"4547":-5.8658952713012695,"19598":-8.72094440460205}},"graph":{"mac_fp16":{"p":[0.009538613947588436,0.9904613860524116],"max_abs_dp":4.806065986590752e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.00953602652133783,0.9904639734786622],"max_abs_dp":2.2186397359846843e-06,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":596,"line_sha256":"667246ded44cd6460acfd4fc39c733cfb1fda1f7871c23c0017b3073064bd16b","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0004","pair_id":"v3-legacy-transfer-authorization-0004","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0004","id":"contrastive/authorization/v3-legacy-transfer-authorization-0004/a","text_sha256":"1c5cf01b96e9cf10fdf56ca75275cbff95d73ca7746f12d5241b23324d66f448","row_sha256":"b25cf494e3fe870d48e931c703bd8a43f1b8a9ea466dbd8a72cd9eb12030e1e0","group_id":"v3-legacy-transfer-authorization-0004","variant":"clean"}},"request_sha256":"eb4fa3b1024781d9aa22df5f8ff73b5f760139b7d05755fc3a8785c405f75518","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"2540909a22eb4bd942e3d51709f64824eae3061400e0da480db8c45d8a15c3bd","oracle":{"probs":[0.9859962024863149,0.014003797513685162],"argmax":"yes","top2_margin":0.9719924049726297,"near_tie":false,"group_logz":{"11683":-0.021425247192382812,"12447":-5.54732608795166,"2243":-4.275749206542969,"4547":-9.102014541625977,"19598":-10.660493850708008}},"graph":{"mac_fp16":{"p":[0.9859598074949817,0.0140401925050184],"max_abs_dp":3.639499133323808e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9861092731434254,0.013890726856574581],"max_abs_dp":0.00011307065711058106,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":597,"line_sha256":"e873f809e5bd86b46a1c927bb82ad1ea0063170ec2e1c59e3926aa859fcade32","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0004","pair_id":"v3-legacy-transfer-authorization-0004","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0004","id":"contrastive/authorization/v3-legacy-transfer-authorization-0004/b","text_sha256":"07c4c83c3738eb881c7eb630ff30ea7dd93c45dc7cdb5e1b07a56338ffb0245d","row_sha256":"45faa77f513bfd475dfe075cb98619a9b39d1d239f69d2aa3b0d381039b5d8a3","group_id":"v3-legacy-transfer-authorization-0004","variant":"clean"}},"request_sha256":"267dc193bd2912e12223af43045428107d0c1d560c5283e8694feb55d0d272fe","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"475f6c88548a3d2daf22770233542a919578d4c58ab526af40ee7c6f56a91e16","oracle":{"probs":[0.011212415324239423,0.9887875846757604],"argmax":"no","top2_margin":0.977575169351521,"near_tie":false,"group_logz":{"11683":-4.494716644287109,"12447":-9.9929838180542,"2243":-0.0152587890625,"4547":-6.050886154174805,"19598":-8.994832992553711}},"graph":{"mac_fp16":{"p":[0.011241386157828182,0.9887586138421718],"max_abs_dp":2.8970833588758474e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.011233361480114619,0.9887666385198854],"max_abs_dp":2.094615587519555e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":598,"line_sha256":"b134a682c4cd85a00b87069a330a04048f884ed7ed2293f4deff70f655672fb0","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0005","pair_id":"v3-legacy-transfer-authorization-0005","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0005","id":"contrastive/authorization/v3-legacy-transfer-authorization-0005/a","text_sha256":"4d3d3769d3d2191c191065c7403378cb6c30fb24ea807233305da13d7ef2e853","row_sha256":"6ef7d1426dae17368d2ce35151484d4ef411084a27d5a2cace6305e37bb68a8f","group_id":"v3-legacy-transfer-authorization-0005","variant":"clean"}},"request_sha256":"6dcad60146e92e3f0425c5791d1466f18a095a0bf9b6edbf41604d81682c6446","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"9b122e7f0e3f67afbbd5f2aa2e16c7066dcd13254745208f6262dc5a5167df42","oracle":{"probs":[0.9839642029102131,0.016035797089786943],"argmax":"yes","top2_margin":0.9679284058204262,"near_tie":false,"group_logz":{"11683":-0.023832321166992188,"12447":-5.511268615722656,"2243":-4.140598297119141,"4547":-9.016716003417969,"19598":-10.527368545532227}},"graph":{"mac_fp16":{"p":[0.9838790887680158,0.016120911231984336],"max_abs_dp":8.511414219739347e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9839690924577741,0.016030907542225907],"max_abs_dp":4.889547561035684e-06,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":599,"line_sha256":"9e17fbb24549fac85b0174f19e31cf203c6456526b4639e24cf135fb9f7a578e","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0005","pair_id":"v3-legacy-transfer-authorization-0005","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0005","id":"contrastive/authorization/v3-legacy-transfer-authorization-0005/b","text_sha256":"f3f9a528d7d2183125eedc8f5661ca883e50fca6ae9979fc0cc8c92cd23c3e64","row_sha256":"b5b1ee21464a6bac5e38fd5379565b1d88802db539d6432fd3f3092be34192fd","group_id":"v3-legacy-transfer-authorization-0005","variant":"clean"}},"request_sha256":"0ada0c3c0039197a492e8cd947ed43a18667b08d9a0112e723556ad2019ec20f","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"735d0384b2d02547b8e3764e327ff3cd71706c742df88d5e4a29a9109ea90bcf","oracle":{"probs":[0.015715697686393245,0.9842843023136068],"argmax":"no","top2_margin":0.9685686046272135,"near_tie":false,"group_logz":{"11683":-4.15948486328125,"12447":-9.296110153198242,"2243":-0.022230148315429688,"4547":-5.449737548828125,"19598":-8.673974990844727}},"graph":{"mac_fp16":{"p":[0.01569105821125184,0.9843089417887483],"max_abs_dp":2.4639475141530198e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01566908290476926,0.9843309170952307],"max_abs_dp":4.6614781623984664e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":600,"line_sha256":"e731c013ca51aee6c6428f9d1dffe8d0620f8c9a7c5f080725b0e7abdf9720ff","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0006","pair_id":"v3-legacy-transfer-authorization-0006","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0006","id":"contrastive/authorization/v3-legacy-transfer-authorization-0006/a","text_sha256":"ecd3c93d9b939d0f85444e52bd842e2d36cf0837bed11485152f5a31dd1bce58","row_sha256":"bfe78e8f9977da7242027380b90f05b448f3161150eb95d7d3ca8c91ad1d0624","group_id":"v3-legacy-transfer-authorization-0006","variant":"clean"}},"request_sha256":"7269786bfcf1538536cb3ed9aeaafa79e0b81c7375ecacabc85b61be9760ebf5","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"197ede2083deccba66d4fdca3601423dcb093393b546b2f86dafde84ec9f9f05","oracle":{"probs":[0.9833061174194555,0.016693882580544484],"argmax":"yes","top2_margin":0.966612234838911,"near_tie":false,"group_logz":{"11683":-0.02384185791015625,"12447":-5.701446533203125,"2243":-4.099720001220703,"4547":-9.133341789245605,"19598":-10.525537490844727}},"graph":{"mac_fp16":{"p":[0.9833149943033542,0.016685005696645824],"max_abs_dp":8.876883898722987e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9829142361281182,0.017085763871881914],"max_abs_dp":0.00039188129133743005,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":601,"line_sha256":"ae2f3c609dc55f99817022a5651adb8815d04df0e7940d7b1169c34d317e04cf","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0006","pair_id":"v3-legacy-transfer-authorization-0006","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0006","id":"contrastive/authorization/v3-legacy-transfer-authorization-0006/b","text_sha256":"3b017d703a12e51ab0f0b2103aa946faa9462f88375d06d8c2eb06914fcac35a","row_sha256":"8aad8b63e2ff385800a647aa1b2a64aecac781b33dd92b88670e516762c8eee0","group_id":"v3-legacy-transfer-authorization-0006","variant":"clean"}},"request_sha256":"b25c05642b05cc7f8c66a8bc8ee9f1be813de67ef36b4c22d835a4c79d9cbf58","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"0554a8e41aa4da8e15c0ea70df7aeabb587a65606b3cc41f41dd35f0eea6455f","oracle":{"probs":[0.013285834085786234,0.9867141659142137],"argmax":"no","top2_margin":0.9734283318284275,"near_tie":false,"group_logz":{"11683":-4.326086044311523,"12447":-9.81816291809082,"2243":-0.018404006958007812,"4547":-5.863295555114746,"19598":-8.768489837646484}},"graph":{"mac_fp16":{"p":[0.013271994723040876,0.986728005276959],"max_abs_dp":1.3839362745357922e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.013335399530753918,0.9866646004692461],"max_abs_dp":4.9565444967683905e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":602,"line_sha256":"e50ee51e92d2fc132e89f66162afe9a9b145fc2616a196163c86e23fa76fd957","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0007","pair_id":"v3-legacy-transfer-authorization-0007","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0007","id":"contrastive/authorization/v3-legacy-transfer-authorization-0007/a","text_sha256":"c3d41884d8a336d1d9179194fbc14c16f913d4d7f1e1629c373e5265f4cfdbff","row_sha256":"b6f039dfc5143e1695e27fb8d47a1bc920fb9fe7224d3c213e509c40580b4808","group_id":"v3-legacy-transfer-authorization-0007","variant":"clean"}},"request_sha256":"afeb53dce593fda68b9ee4afdea4583a34687742aaf330c036856513c2d58b6b","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"68526971b46ffe3ebe1135faa88b9f62423eb4259d00d028bf9119dccc1bb6bc","oracle":{"probs":[0.9695782760787084,0.030421723921291687],"argmax":"yes","top2_margin":0.9391565521574167,"near_tie":false,"group_logz":{"11683":-0.0405731201171875,"12447":-5.418776512145996,"2243":-3.502277374267578,"4547":-8.41547966003418,"19598":-9.983762741088867}},"graph":{"mac_fp16":{"p":[0.9696783249940883,0.03032167500591163],"max_abs_dp":0.00010004891538005489,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9684452769324875,0.031554723067512426],"max_abs_dp":0.0011329991462208922,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":603,"line_sha256":"a05c9cf5e1e543c8ecad809521e6b8086524f19c3f5daeab23a0c3494e3b54ad","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0007","pair_id":"v3-legacy-transfer-authorization-0007","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0007","id":"contrastive/authorization/v3-legacy-transfer-authorization-0007/b","text_sha256":"7d7fd5548cc0cc301887f2df78a4fced5caebe785afa79c7b1333a56e9845122","row_sha256":"bf9d9a17775ba2e6a31271b4b2ddca06519e04612fbb7ad76cb367f825565e3e","group_id":"v3-legacy-transfer-authorization-0007","variant":"clean"}},"request_sha256":"8ca3e0d27bb3b6ef95af812a0f4d50b048323915cbc646d9c587f9522e0a5fc4","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"ccdf2fce7c1f903bbe6b01425848ed7fdf224948c4f7b999fbbba0736d156be8","oracle":{"probs":[0.009665806152949898,0.9903341938470501],"argmax":"no","top2_margin":0.9806683876941003,"near_tie":false,"group_logz":{"11683":-4.644195556640625,"12447":-9.989404678344727,"2243":-0.01474761962890625,"4547":-5.704216003417969,"19598":-8.79353141784668}},"graph":{"mac_fp16":{"p":[0.009698244811165338,0.9903017551888347],"max_abs_dp":3.243865821545011e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009699212732125956,0.990300787267874],"max_abs_dp":3.3406579176165074e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":604,"line_sha256":"378793571c68c1fe10482e628113fd5a0d00dd8c9f0550cf31fd928e53656fcd","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0008","pair_id":"v3-legacy-transfer-authorization-0008","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0008","id":"contrastive/authorization/v3-legacy-transfer-authorization-0008/a","text_sha256":"94da7e72d1aebd9b7460a8a4084f27285ee39303051ee737bcffede7c105293b","row_sha256":"892add71eeedd8d58d01a96db49223c9c311f66851dd689df41de05fb4e14957","group_id":"v3-legacy-transfer-authorization-0008","variant":"clean"}},"request_sha256":"51dc36b6f14e72df20d2c6be688261eac2d8c117452e5f9823029f0f0a267519","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"cd9648a22e5378a21605d7af8fb7c51c1f3346681bcd96559592e4df5cdd4aba","oracle":{"probs":[0.9883694240657707,0.011630575934229316],"argmax":"yes","top2_margin":0.9767388481315413,"near_tie":false,"group_logz":{"11683":-0.018278121948242188,"12447":-5.666482925415039,"2243":-4.460697174072266,"4547":-9.397729873657227,"19598":-10.796866416931152}},"graph":{"mac_fp16":{"p":[0.9883724356117468,0.01162756438825322],"max_abs_dp":3.011545976150387e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9884623119123673,0.011537688087632697],"max_abs_dp":9.28878465966676e-05,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":605,"line_sha256":"5c47b8194d3e52e243c3ec2c151f172cefbc15899ed3b0b50b983fb4278386e8","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0008","pair_id":"v3-legacy-transfer-authorization-0008","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0008","id":"contrastive/authorization/v3-legacy-transfer-authorization-0008/b","text_sha256":"aa7487423f79b550238ef577886098b9268f0c4e25725b7ba1ca9383c024ce67","row_sha256":"ab93cf1d8d870794dc335cb5fdc49119cd9784ac56e3e4101439a7577b089dc9","group_id":"v3-legacy-transfer-authorization-0008","variant":"clean"}},"request_sha256":"e6e105bbb362fe12738cf4836888c7393592a940f01c2f856569e19917c0d857","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"b08d36c8b02f6a57d5174eb788c1f8d865816c7563d2780a86a56edcf277be84","oracle":{"probs":[0.013101083130256245,0.9868989168697436],"argmax":"no","top2_margin":0.9737978337394874,"near_tie":false,"group_logz":{"11683":-4.339666366577148,"12447":-9.6512451171875,"2243":-0.017793655395507812,"4547":-5.783684730529785,"19598":-8.82848072052002}},"graph":{"mac_fp16":{"p":[0.013089296260092403,0.9869107037399076],"max_abs_dp":1.1786870164010033e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01295575550951656,0.9870442444904834],"max_abs_dp":0.0001453276207398213,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":606,"line_sha256":"8bb6ef4b844fc4387c910d1c108ee0fd468c5ec77dbb2fb8d956a48c507feaec","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0009","pair_id":"v3-legacy-transfer-authorization-0009","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0009","id":"contrastive/authorization/v3-legacy-transfer-authorization-0009/a","text_sha256":"7ce0efd4db2ef8224c841560d6b7d461fb8bbd6a20e06af9a420fa48e65ed42c","row_sha256":"a617c422ea7347aa0c746192a302f038077b92ae2aec541e76b8e901b118d571","group_id":"v3-legacy-transfer-authorization-0009","variant":"clean"}},"request_sha256":"d5825debdf677adc15c81f8cd7749c7e101ba0b4d4f91ad0004bab62872959a7","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"a194cf84a4b9809c4860d761bfa712631bdf61699b37bb9af7b6eba4e245ac4a","oracle":{"probs":[0.975114234805952,0.024885765194047945],"argmax":"yes","top2_margin":0.9502284696119041,"near_tie":false,"group_logz":{"11683":-0.03469085693359375,"12447":-5.334224700927734,"2243":-3.7029495239257812,"4547":-8.455469131469727,"19598":-10.161171913146973}},"graph":{"mac_fp16":{"p":[0.9752445704585709,0.024755429541429112],"max_abs_dp":0.0001303356526188848,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9738824597710577,0.026117540228942214],"max_abs_dp":0.0012317750348943557,"argmax_equal":true}}}]},{"id":"tv4x_legacy_holdout_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":607,"line_sha256":"88f90555873b2eb842d60196212234bfe2bd2923e2d223eb43d9a6cb1c73332c","_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0009","pair_id":"v3-legacy-transfer-authorization-0009","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0009","id":"contrastive/authorization/v3-legacy-transfer-authorization-0009/b","text_sha256":"84bf86bf9cfd6d4232bcc52fb50594e132f5bad9cd156434686c4c046766b872","row_sha256":"731cfc2a35ba9bbaba7b0af0f72e1e13cd0cfb19a13ab5d0d896cfb5228a13ee","group_id":"v3-legacy-transfer-authorization-0009","variant":"clean"}},"request_sha256":"3d9072d368a32b4d37392384aa19c7bea6e76cd2f4a6686206c98586c8cef6de","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"85303acb64a161219cd7c3f94b957cd52112f3e2c6c4f9c0b6280f372d8df012","oracle":{"probs":[0.009671157172907981,0.9903288428270921],"argmax":"no","top2_margin":0.9806576856541841,"near_tie":false,"group_logz":{"11683":-4.643705368041992,"12447":-10.03579330444336,"2243":-0.0148162841796875,"4547":-5.756818771362305,"19598":-8.646820068359375}},"graph":{"mac_fp16":{"p":[0.009658447686513827,0.9903415523134861],"max_abs_dp":1.2709486394153757e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.00970535191597617,0.9902946480840238],"max_abs_dp":3.419474306831649e-05,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_00","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":668,"line_sha256":"8920696af4b5f59c2f5c3b710e27146aa5fee9f2e7e6437290b1988607220d29","_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"9d21c218a7b00c718c076542156adb5f2d344bf3f0fef09b372948148fb73f65","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"request_sha256":"e67021d97398818ae9a5727e9728a755fb825b4df3c8621b10ed99e0d9ee8e5b","questions":[{"type":"choice","n_options":2,"row_len":162,"slot":161,"row_ids_sha256":"ffb6e7efd3c3d81e79548efb83f31eee768f6fe6e5d5fdfe47490841044c7cd2","oracle":{"probs":[0.6541787934769859,0.3458212065230141],"argmax":"reject","top2_margin":0.3083575869539718,"near_tie":false,"group_logz":{"41":-0.4363441467285156,"334":-8.999947547912598,"42":-1.0738029479980469,"378":-9.319923400878906}},"graph":{"mac_fp16":{"p":[0.653213561799197,0.34678643820080296],"max_abs_dp":0.0009652316777889469,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6599854875625992,0.34001451243740083],"max_abs_dp":0.005806694085613295,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_01","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":669,"line_sha256":"ca12b8e0ba6e7fd29b6cfefd36ed4d82ae7995477beaeb3984422643e71d0284","_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"6e759020afb08858bcc3422ac266ad43f50323c3f1268a2ce736ecc0a2677fc3","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":13,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":false}}},"request_sha256":"f4cec363001fe71c1b0e5b5488dd8feb17b08581b956e3395ac2ed151b785175","questions":[{"type":"choice","n_options":2,"row_len":162,"slot":161,"row_ids_sha256":"20a754b27ffab0f94b9ef3c54f9c68587fe7ef46b19fe343b86876d4020f31e7","oracle":{"probs":[0.9617493795779891,0.03825062042201084],"argmax":"reject","top2_margin":0.9234987591559782,"near_tie":false,"group_logz":{"41":-0.04583740234375,"334":-8.697181701660156,"42":-3.2704315185546875,"378":-11.359238624572754}},"graph":{"mac_fp16":{"p":[0.9617335990249367,0.038266400975063154],"max_abs_dp":1.5780553052402446e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9617816624096773,0.03821833759032276],"max_abs_dp":3.228283168821289e-05,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_02","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":670,"line_sha256":"1e8b24b3a915a7b094e71e3f98b320ef9df2f8596b3f854aafd3c40fd7bcd16a","_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"9d21c218a7b00c718c076542156adb5f2d344bf3f0fef09b372948148fb73f65","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"request_sha256":"e67021d97398818ae9a5727e9728a755fb825b4df3c8621b10ed99e0d9ee8e5b","questions":[{"type":"choice","n_options":2,"row_len":162,"slot":161,"row_ids_sha256":"ffb6e7efd3c3d81e79548efb83f31eee768f6fe6e5d5fdfe47490841044c7cd2","oracle":{"probs":[0.6541787934769859,0.3458212065230141],"argmax":"reject","top2_margin":0.3083575869539718,"near_tie":false,"group_logz":{"41":-0.4363441467285156,"334":-8.999947547912598,"42":-1.0738029479980469,"378":-9.319923400878906}},"graph":{"mac_fp16":{"p":[0.653213561799197,0.34678643820080296],"max_abs_dp":0.0009652316777889469,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6599854875625992,0.34001451243740083],"max_abs_dp":0.005806694085613295,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_03","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":671,"line_sha256":"652d5be5a79bb05388c2e03a17cc36d46c37c963746e43d4bcac92ed11c26578","_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"c37513c5cf15a96d63b6bcf1f7bc6c02f8ab02a7445170a07528081709f07aab","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":185},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"request_sha256":"b842465b10fe8f0668a5bc8336953bfd67d3599b091338828114e3b44c7d13dc","questions":[{"type":"choice","n_options":2,"row_len":162,"slot":161,"row_ids_sha256":"189a5f9ac7c17d430a9225b77abf0562073af89bc3179da5856ee732ebd28657","oracle":{"probs":[0.5906892836366454,0.4093107163633546],"argmax":"reject","top2_margin":0.18137856727329082,"near_tie":false,"group_logz":{"41":-0.5388393402099609,"334":-9.012537002563477,"42":-0.9056549072265625,"378":-9.124797821044922}},"graph":{"mac_fp16":{"p":[0.5896275094750797,0.41037249052492036],"max_abs_dp":0.001061774161565776,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5959145913883684,0.40408540861163156],"max_abs_dp":0.0052253077517230295,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_04","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":672,"line_sha256":"680378d73b42c9fcb2ad7d7e8b1d08c5b9a9ab5976d8dcef3024bd439fb8d952","_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e308fdfe8a88e0200c7281e2d1b6312ec2e22874a2e1165eb1088ebdf98d776a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"request_sha256":"c97f1e687b4d1cc29bc521f410ff7446445efee34baa61ca76ddc82cd7df0a7f","questions":[{"type":"choice","n_options":2,"row_len":153,"slot":152,"row_ids_sha256":"47e195c7519192cfce4247f36aaefdf62f3a87bffbb42ea840f52739717bdff4","oracle":{"probs":[0.045708211390355497,0.9542917886096446],"argmax":"reject","top2_margin":0.9085835772192891,"near_tie":false,"group_logz":{"41":-3.094414710998535,"334":-11.359395027160645,"42":-0.05572319030761719,"378":-8.24864387512207}},"graph":{"mac_fp16":{"p":[0.0457515699805549,0.954248430019445],"max_abs_dp":4.335859019954924e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.045772436233504916,0.9542275637664952],"max_abs_dp":6.422484314941995e-05,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_05","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":673,"line_sha256":"8a6a53b3e8365a3e677424d53690560de93d5a6925157af5bcdfc652790f5513","_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"c237482c05993ce384d2fe35c55853b22c11d677cecc9916466eff5498247811","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Sana","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":true}}},"request_sha256":"87e86101f07c7d8408fdaf110871a8cf81db21a7d3e0b3aa4250952f012b9e56","questions":[{"type":"choice","n_options":2,"row_len":154,"slot":153,"row_ids_sha256":"7b8784c61bd750da1c47fdb358b75f89068929c748559161112b3ac5373a64c5","oracle":{"probs":[0.9808758201340538,0.019124179865946198],"argmax":"accept","top2_margin":0.9617516402681077,"near_tie":false,"group_logz":{"41":-0.028097152709960938,"334":-8.96469497680664,"42":-3.9655895233154297,"378":-12.431669235229492}},"graph":{"mac_fp16":{"p":[0.980900501822994,0.019099498177006048],"max_abs_dp":2.4681688940209057e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9807238407340935,0.01927615926590646],"max_abs_dp":0.0001519793999603669,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_06","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":674,"line_sha256":"9c82396225ea8439e8117bb8e03413b669fe459ee7a3f69d5dea2c54b7c2cfb4","_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e308fdfe8a88e0200c7281e2d1b6312ec2e22874a2e1165eb1088ebdf98d776a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"request_sha256":"c97f1e687b4d1cc29bc521f410ff7446445efee34baa61ca76ddc82cd7df0a7f","questions":[{"type":"choice","n_options":2,"row_len":153,"slot":152,"row_ids_sha256":"47e195c7519192cfce4247f36aaefdf62f3a87bffbb42ea840f52739717bdff4","oracle":{"probs":[0.045708211390355497,0.9542917886096446],"argmax":"reject","top2_margin":0.9085835772192891,"near_tie":false,"group_logz":{"41":-3.094414710998535,"334":-11.359395027160645,"42":-0.05572319030761719,"378":-8.24864387512207}},"graph":{"mac_fp16":{"p":[0.0457515699805549,0.954248430019445],"max_abs_dp":4.335859019954924e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.045772436233504916,0.9542275637664952],"max_abs_dp":6.422484314941995e-05,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_07","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":675,"line_sha256":"1e0b2d25356b48b432de8a91fa17dbcba5221c637e5c24aecd22e423cd5b0aac","_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"79886061d045f1d5332c4dd9643caeda04a0aeba9939bcfb40315e3c3e76fc89","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":980},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"request_sha256":"c513ed9cf5e2eedbf5f731164e25b69f23ea804481659996d58a4952c93a63af","questions":[{"type":"choice","n_options":2,"row_len":153,"slot":152,"row_ids_sha256":"99bf273b03e4f9791b47b8805df6b584fd398c7a7c4e24710c302ae3f05d0c53","oracle":{"probs":[0.04572414618018977,0.9542758538198103],"argmax":"reject","top2_margin":0.9085517076396206,"near_tie":false,"group_logz":{"41":-3.094095230102539,"334":-11.360522270202637,"42":-0.05576896667480469,"378":-8.248712539672852}},"graph":{"mac_fp16":{"p":[0.04580327393102189,0.9541967260689781],"max_abs_dp":7.912775083218637e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04585137728572938,0.9541486227142707],"max_abs_dp":0.00012723110553961026,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_08","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":676,"line_sha256":"afd8184e2eae9b9204413fda9de401983eb9b3867c8958ded79e1bc61367a5d4","_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"88789ea4fc9d940247d5027641a0a57a4f4e858005a4ab6b49b6be705372a2df","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"request_sha256":"f662927580cd2a5303d0ec3e1d81bbe0e37ce06bcaf9cea106b8686999f5bcbd","questions":[{"type":"choice","n_options":2,"row_len":179,"slot":178,"row_ids_sha256":"b79b5472296630e6bb358a531805194bfc528ce929ef35828cea666a6e9be7d5","oracle":{"probs":[0.803498486528403,0.19650151347159697],"argmax":"reject","top2_margin":0.6069969730568061,"near_tie":false,"group_logz":{"41":-0.22997093200683594,"334":-8.843777656555176,"42":-1.6382761001586914,"378":-9.77705192565918}},"graph":{"mac_fp16":{"p":[0.8038736195087096,0.19612638049129033],"max_abs_dp":0.00037513298030664166,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8149404787120155,0.18505952128798456],"max_abs_dp":0.011441992183612415,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_09","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":677,"line_sha256":"40d19d56abe8b1e41d4a916ec8585d5d55debebb504c3e73dc9cbd33fee179d7","_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e37ef88d426d56212bb754e5a65f0aeafa211c282d1577867b00a2248ccd5770","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Aiko","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":false}}},"request_sha256":"faefac3b763847128f332a03312c0742d3746b9ff33b473462dd0ccb939e7d83","questions":[{"type":"choice","n_options":2,"row_len":180,"slot":179,"row_ids_sha256":"c6d496dac3380299c7726f3a5a914cbb8053ac57418f9537868c88b972de570f","oracle":{"probs":[0.9712997815975297,0.02870021840247038],"argmax":"reject","top2_margin":0.9425995631950593,"near_tie":false,"group_logz":{"41":-0.03731346130371094,"334":-8.493030548095703,"42":-3.5590438842773438,"378":-11.608511924743652}},"graph":{"mac_fp16":{"p":[0.9713045505648925,0.028695449435107507],"max_abs_dp":4.768967362872578e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9713885392175586,0.028611460782441443],"max_abs_dp":8.875762002893625e-05,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_10","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":678,"line_sha256":"ef4829c56e5bf0e2044bca04b9c67ae4417cad2de0ee036be0ec79f102f646bf","_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"88789ea4fc9d940247d5027641a0a57a4f4e858005a4ab6b49b6be705372a2df","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"request_sha256":"f662927580cd2a5303d0ec3e1d81bbe0e37ce06bcaf9cea106b8686999f5bcbd","questions":[{"type":"choice","n_options":2,"row_len":179,"slot":178,"row_ids_sha256":"b79b5472296630e6bb358a531805194bfc528ce929ef35828cea666a6e9be7d5","oracle":{"probs":[0.803498486528403,0.19650151347159697],"argmax":"reject","top2_margin":0.6069969730568061,"near_tie":false,"group_logz":{"41":-0.22997093200683594,"334":-8.843777656555176,"42":-1.6382761001586914,"378":-9.77705192565918}},"graph":{"mac_fp16":{"p":[0.8038736195087096,0.19612638049129033],"max_abs_dp":0.00037513298030664166,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8149404787120155,0.18505952128798456],"max_abs_dp":0.011441992183612415,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_11","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":679,"line_sha256":"f450d58c2778bc6c2773fa32be8c4794ea66cbdeaba5b2feaa91722edadb1707","_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"594d32ac9bd72de204403039f35eb2150741195ef36ea7e9c04551a52103eb42","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":534},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"request_sha256":"912de609dbf922be7bf2aa4b240d104f7b9de440e5ecced987329cfd90696777","questions":[{"type":"choice","n_options":2,"row_len":179,"slot":178,"row_ids_sha256":"ef5aabcd125cc59adfd498557088a0c1e59ccc658c8bcaa6a61cee4d7bbb7f52","oracle":{"probs":[0.8002664384210346,0.19973356157896544],"argmax":"reject","top2_margin":0.6005328768420692,"near_tie":false,"group_logz":{"41":-0.23403167724609375,"334":-8.836151123046875,"42":-1.6219921112060547,"378":-9.754150390625}},"graph":{"mac_fp16":{"p":[0.8009870147064211,0.1990129852935788],"max_abs_dp":0.0007205762853866315,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8119453852492741,0.18805461475072596],"max_abs_dp":0.011678946828239473,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_12","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":680,"line_sha256":"af060abb67585345c4e9732880fecbf619807bac477bb597a09f5c9c3e1acce6","_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"44333ca47dd477e539907f962281754aeba97abb57ff3fe0fc64b652bf341ec9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"request_sha256":"18a553f66c1f203c2d7d68519775d2153fa117147566075b394ff813645edb6c","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"e696f2864d54e1009fd5460b8a378f2b8ef919f61b2c593c83f3096bdeffc3d5","oracle":{"probs":[0.22782783648504007,0.7721721635149599],"argmax":"accept","top2_margin":0.5443443270299198,"near_tie":false,"group_logz":{"41":-1.4895744323730469,"334":-9.461581230163574,"42":-0.26895713806152344,"378":-8.212019920349121}},"graph":{"mac_fp16":{"p":[0.2299140756543437,0.7700859243456564],"max_abs_dp":0.002086239169303644,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2436688937236969,0.7563311062763032],"max_abs_dp":0.015841057238656836,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_13","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":681,"line_sha256":"65728cd79dddb0c482084ab2ca2490f288e7815b1f5f9506db362c4c94dd8841","_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"38b842d897276b85c54a2398f0269e9bbd01e488a08934084d0f273db092d1d9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":50,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":true}}},"request_sha256":"7c5b08d97d7fd38e7169d1b745f2949a4adf7acaf1fd4de7e73e9c07e669df8a","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"4653209981af9770ba82051712ae300361da634290e1a40a978292a6c5cc2c42","oracle":{"probs":[0.02196484222170583,0.9780351577782941],"argmax":"accept","top2_margin":0.9560703155565883,"near_tie":false,"group_logz":{"41":-3.827482223510742,"334":-12.350825309753418,"42":-0.03137969970703125,"378":-8.454890251159668}},"graph":{"mac_fp16":{"p":[0.021999635611253107,0.978000364388747],"max_abs_dp":3.479338954727809e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02207035098472432,0.9779296490152756],"max_abs_dp":0.00010550876301851364,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_14","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":682,"line_sha256":"d996759fbea27e4ed44a49747f80edfbef03018b20161c49b9338797d8e90e9c","_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"44333ca47dd477e539907f962281754aeba97abb57ff3fe0fc64b652bf341ec9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"request_sha256":"18a553f66c1f203c2d7d68519775d2153fa117147566075b394ff813645edb6c","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"e696f2864d54e1009fd5460b8a378f2b8ef919f61b2c593c83f3096bdeffc3d5","oracle":{"probs":[0.22782783648504007,0.7721721635149599],"argmax":"accept","top2_margin":0.5443443270299198,"near_tie":false,"group_logz":{"41":-1.4895744323730469,"334":-9.461581230163574,"42":-0.26895713806152344,"378":-8.212019920349121}},"graph":{"mac_fp16":{"p":[0.2299140756543437,0.7700859243456564],"max_abs_dp":0.002086239169303644,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2436688937236969,0.7563311062763032],"max_abs_dp":0.015841057238656836,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_15","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":683,"line_sha256":"fbe52b98c1ce88cf6a1182bc1bafd2c585db3ee36f9ce88d589daaba80e5dc95","_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"5ae23575dd80b3268d4923affc02a219801d66f11840e53c897244e64e0ce3e8","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":573},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"request_sha256":"dea11d3255fbb9cde5e38a6c50ba01741a2ec86f921b685d1377e7002997ad3e","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"a23ab621a2b337b659c2d1f94baa8ce5f64d65651e3bd392b5d890ea472bc542","oracle":{"probs":[0.20761795416225876,0.7923820458377413],"argmax":"accept","top2_margin":0.5847640916754826,"near_tie":false,"group_logz":{"41":-1.5824851989746094,"334":-9.567928314208984,"42":-0.24314117431640625,"378":-8.195073127746582}},"graph":{"mac_fp16":{"p":[0.21057504237022928,0.7894249576297707],"max_abs_dp":0.002957088207970582,"argmax_equal":true},"iphone_int8mlp":{"p":[0.220448601752082,0.779551398247918],"max_abs_dp":0.01283064758982333,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_16","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":684,"line_sha256":"04b01120925ac5f08ceb32122f6797ca2c708c92f08bd7cb7cc7542ca1a1c994","_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"ee1eab34afdaea0093568ecfa1a1025cb93f17c4c1cc85043e7a61da75b32c39","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"request_sha256":"7c3377a1c9ee6f72fb36ccdf76b9541fe3198435fa2fdecf1064a255acac8385","questions":[{"type":"choice","n_options":2,"row_len":166,"slot":165,"row_ids_sha256":"901439324ff9321973dca57880b74c0e011e7e7cf9e2b831377dc59b0ff2da2d","oracle":{"probs":[0.1129437999966824,0.8870562000033175],"argmax":"reject","top2_margin":0.7741124000066352,"near_tie":false,"group_logz":{"41":-2.1910400390625,"334":-10.256568908691406,"42":-0.1300220489501953,"378":-8.1936674118042}},"graph":{"mac_fp16":{"p":[0.11267633629006515,0.8873236637099349],"max_abs_dp":0.00026746370661734353,"argmax_equal":true},"iphone_int8mlp":{"p":[0.11432149667636263,0.8856785033236373],"max_abs_dp":0.0013776966796802403,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_17","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":685,"line_sha256":"bc740e6869e7b8713f09a1718664de4945a605282e8d32603e01392b90b43e7d","_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"03cfd4b4074dbbf8e3c8420fe91a93db29eda4e7a97252e1f6ce79ded694bcc6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Sana","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":true}}},"request_sha256":"5d45494ea937fcc672010cd89d8e3d78dca15f179ef35120390172783b9086a3","questions":[{"type":"choice","n_options":2,"row_len":166,"slot":165,"row_ids_sha256":"2cbcfd2771f6abe93f0c4972d1d02449196c4f1fd8fc98a2af392c6067b4f899","oracle":{"probs":[0.15738660186768128,0.8426133981323188],"argmax":"reject","top2_margin":0.6852267962646376,"near_tie":false,"group_logz":{"41":-1.860525131225586,"334":-9.9063081741333,"42":-0.1827220916748047,"378":-8.206663131713867}},"graph":{"mac_fp16":{"p":[0.15671605089266702,0.843283949107333],"max_abs_dp":0.0006705509750142558,"argmax_equal":true},"iphone_int8mlp":{"p":[0.15803550881721812,0.8419644911827819],"max_abs_dp":0.0006489069495368982,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_18","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":686,"line_sha256":"8c458667d9a9c459accc512613ac7bb71406b6d789950e02c4c19979e78e9525","_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"ee1eab34afdaea0093568ecfa1a1025cb93f17c4c1cc85043e7a61da75b32c39","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"request_sha256":"7c3377a1c9ee6f72fb36ccdf76b9541fe3198435fa2fdecf1064a255acac8385","questions":[{"type":"choice","n_options":2,"row_len":166,"slot":165,"row_ids_sha256":"901439324ff9321973dca57880b74c0e011e7e7cf9e2b831377dc59b0ff2da2d","oracle":{"probs":[0.1129437999966824,0.8870562000033175],"argmax":"reject","top2_margin":0.7741124000066352,"near_tie":false,"group_logz":{"41":-2.1910400390625,"334":-10.256568908691406,"42":-0.1300220489501953,"378":-8.1936674118042}},"graph":{"mac_fp16":{"p":[0.11267633629006515,0.8873236637099349],"max_abs_dp":0.00026746370661734353,"argmax_equal":true},"iphone_int8mlp":{"p":[0.11432149667636263,0.8856785033236373],"max_abs_dp":0.0013776966796802403,"argmax_equal":true}}}]},{"id":"tv4x_composition_holdout_19","source":"tv4x","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":687,"line_sha256":"c853583f4d39e1b769116030057143f29ed42f8be1c6b9884a25a5fc20fe9072","_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"b725e3446fd24db24714ad2c277ca4e2a5943831b9bd3769d41d4bf3506d02e5","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":326},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"request_sha256":"78df31d6ca98240669769972535ae73238934bef50328dd26d442237c8e9569b","questions":[{"type":"choice","n_options":2,"row_len":166,"slot":165,"row_ids_sha256":"aa8d1fea31dd4bd3c5e2e8f342130e53a135505fb7654d767d007813422bc02c","oracle":{"probs":[0.11274845997085954,0.8872515400291404],"argmax":"reject","top2_margin":0.7745030800582808,"near_tie":false,"group_logz":{"41":-2.1927871704101562,"334":-10.262438774108887,"42":-0.12981796264648438,"378":-8.195003509521484}},"graph":{"mac_fp16":{"p":[0.11250612419632416,0.8874938758036758],"max_abs_dp":0.00024233577453537658,"argmax_equal":true},"iphone_int8mlp":{"p":[0.11416430318037982,0.8858356968196202],"max_abs_dp":0.001415843209520276,"argmax_equal":true}}}]},{"id":"tv4s_00","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":628,"line_sha256":"b2b6b7970d3ed9efaa2d81a5fb80cb1d61e69ade01d103bc6b22e95bb748e56e","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0000","pair_id":"v3-legacy-transfer-deadline-0000","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0000","id":"contrastive/deadline/v3-legacy-transfer-deadline-0000/a","text_sha256":"60fc44aa009aad0ef852c67705132af2432c7a855f61b8d2010877ae53164c03","row_sha256":"79838f04b3dcbb877bab439bac73134a4715c193971c8795e28819b67e54bf2a","group_id":"v3-legacy-transfer-deadline-0000","variant":"clean"}},"request_sha256":"2b8cbc23b048dd28416bbf6fe7c0565200d418840144760d511294d58386bc72","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"bbc44b743a756fc485005b87e6becfe07afe7fa109e4535367282ee7cc589494","oracle":{"probs":[0.4232136648199876,0.5710778009385256,0.005708534241486871],"argmax":"1","top2_margin":0.14786413611853794,"near_tie":false,"group_logz":{"24":-0.8744068145751953,"25":-0.5747585296630859,"26":-5.18032169342041}},"graph":{"mac_fp16":{"p":[0.42343417428127716,0.5708573652701466,0.005708460448576206],"max_abs_dp":0.00022050946128954196,"argmax_equal":true},"iphone_int8mlp":{"p":[0.4144036871304449,0.5797728950483572,0.0058234178211978484],"max_abs_dp":0.008809977689542725,"argmax_equal":true}}}]},{"id":"tv4s_01","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":629,"line_sha256":"2635c7aebe3acc57cea7e489a1afcf6517912a4df7978991f1429ba8255e7582","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0000","pair_id":"v3-legacy-transfer-deadline-0000","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0000","id":"contrastive/deadline/v3-legacy-transfer-deadline-0000/b","text_sha256":"7ee8365ace01dd340f7eb13fde0813efb80021c429d3f53798935c0f7df199e1","row_sha256":"a37b38bd0560d4d21da5a97bf5a3ef47bf2325e3ca6e90ec5ceef9267d082c3b","group_id":"v3-legacy-transfer-deadline-0000","variant":"clean"}},"request_sha256":"2fea826244aaac1ed70d3de79370bbfdc5c9ae3163fd5b95aa997719a54b7e04","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"c302d937b94108d7f5d4d7522d268c8698619e80e2b6ab9f6122e5227d4df81a","oracle":{"probs":[0.016598922980871646,0.9769363926776917,0.006464684341436599],"argmax":"1","top2_margin":0.9603374696968201,"near_tie":false,"group_logz":{"24":-4.109986305236816,"25":-0.03490257263183594,"26":-5.052969932556152}},"graph":{"mac_fp16":{"p":[0.01649697543683095,0.9770617289783831,0.006441295584786039],"max_abs_dp":0.0001253363006913366,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01599627058766799,0.9775778467996706,0.00642588261266139],"max_abs_dp":0.0006414541219789127,"argmax_equal":true}}}]},{"id":"tv4s_02","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":630,"line_sha256":"53b8b02fcce00362ea74a1ce12a4f08092d5006d651358c338620534f1dd8324","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0001","pair_id":"v3-legacy-transfer-deadline-0001","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0001","id":"contrastive/deadline/v3-legacy-transfer-deadline-0001/a","text_sha256":"6a4c4a36c00d6c4858efd120cca9eff44df4258bc9ddc093bc4938bd94e57846","row_sha256":"594793cc8703daca99c86621a9931dee60eca526b6f1755f89051791e3bb755b","group_id":"v3-legacy-transfer-deadline-0001","variant":"clean"}},"request_sha256":"03bf990d25a8a9ffd7dd90455618cea012dd757ea52b367039f60f84a1df2709","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"767d501b360e4dd6f603946e39ec0061a0a653e6acf44fd7f63878bf50f9985a","oracle":{"probs":[0.33263560810507287,0.6625148562937819,0.004849535601145211],"argmax":"1","top2_margin":0.32987924818870906,"near_tie":false,"group_logz":{"24":-1.112802505493164,"25":-0.42380714416503906,"26":-5.340967178344727}},"graph":{"mac_fp16":{"p":[0.3321863347435049,0.6629606467456735,0.004853018510821741],"max_abs_dp":0.00044927336156797004,"argmax_equal":true},"iphone_int8mlp":{"p":[0.32786904309658804,0.6672836938604509,0.004847263042961017],"max_abs_dp":0.004768837566669015,"argmax_equal":true}}}]},{"id":"tv4s_03","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":631,"line_sha256":"21d959282f737e58addc47e449fcd9821a72420a64c4425451adcfb1cfe410fb","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0001","pair_id":"v3-legacy-transfer-deadline-0001","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0001","id":"contrastive/deadline/v3-legacy-transfer-deadline-0001/b","text_sha256":"c94f79861d12ea95869612f0e571189d58acb302119b71d00adc9b792403a66c","row_sha256":"ebddf4f71912e0a1d93a48984634f94afb16c89aaba947ec2be47b74d9597066","group_id":"v3-legacy-transfer-deadline-0001","variant":"clean"}},"request_sha256":"d258eba7fa2c2d22a8daf42f1a02ff043c36c2eddce609e3559d6b8fe9cd8f17","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"09f8a0573ac77d81f9723f3b1cbfcfeb8bd4a0d74664721b0a00be3c19c36f65","oracle":{"probs":[0.003465540964738935,0.9094229374814363,0.08711152155382472],"argmax":"1","top2_margin":0.8223114159276116,"near_tie":false,"group_logz":{"24":-5.68013858795166,"25":-0.11019706726074219,"26":-2.4558181762695312}},"graph":{"mac_fp16":{"p":[0.0034546869671605797,0.9086574662059403,0.08788784682689907],"max_abs_dp":0.0007763252730743492,"argmax_equal":true},"iphone_int8mlp":{"p":[0.003493025062397573,0.9077071529911426,0.08879982194645973],"max_abs_dp":0.001715784490293748,"argmax_equal":true}}}]},{"id":"tv4s_04","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":632,"line_sha256":"0c2793f498c54af0413852715f5bae2adc4fbd93339e08acb558421f64b73e4c","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0002","pair_id":"v3-legacy-transfer-deadline-0002","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0002","id":"contrastive/deadline/v3-legacy-transfer-deadline-0002/a","text_sha256":"e7abe027950fcd48a41c4e72bc5a7336fae563c9b8541f26ff733a2905f665e1","row_sha256":"f19aef83ea00725d0d9920c4462b46b40681657ff506ed348d8520315b004b45","group_id":"v3-legacy-transfer-deadline-0002","variant":"clean"}},"request_sha256":"77d96138d3d544838eaf580a64c5859b825d5d888b3c3c562b85b8dcdc22d4fe","questions":[{"type":"score","n_options":3,"row_len":123,"slot":122,"row_ids_sha256":"f92fecb3a29ee12e5cdf09f08e444fad1cc70f9878ef6aaef85b8a38124bdd1a","oracle":{"probs":[0.25474653956371424,0.736335394759884,0.008918065676401677],"argmax":"1","top2_margin":0.48158885519616973,"near_tie":false,"group_logz":{"24":-1.3902664184570312,"25":-0.32884979248046875,"26":-4.742456436157227}},"graph":{"mac_fp16":{"p":[0.2547244069341292,0.7363527833599398,0.00892280970593099],"max_abs_dp":2.2132629585058794e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.24663390783000835,0.7443061915359952,0.009059900633996438],"max_abs_dp":0.008112631733705883,"argmax_equal":true}}}]},{"id":"tv4s_05","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":633,"line_sha256":"f4fc3afe44e3ceaf41ffa27a63fd924bf78a03569629d74d565e15a270dea045","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0002","pair_id":"v3-legacy-transfer-deadline-0002","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0002","id":"contrastive/deadline/v3-legacy-transfer-deadline-0002/b","text_sha256":"536f00f1b4a91c96dd327d22f1d409c2ecdeb065649d011c91aa09211404b75e","row_sha256":"75b49419e6b53a9d9d04a9898c415b1793bf01d18472b2a01de7a8660f55f604","group_id":"v3-legacy-transfer-deadline-0002","variant":"clean"}},"request_sha256":"5470e7ee888d1d8cab08dae74d09ccaf3d361f97c772b31228a271b6b0076652","questions":[{"type":"score","n_options":3,"row_len":123,"slot":122,"row_ids_sha256":"52aa5c9eb0cf4af011f30953916d5189edd02cace570d5f74cb326d557528527","oracle":{"probs":[0.005783559001195753,0.9824064875133596,0.01180995348544467],"argmax":"1","top2_margin":0.9705965340279149,"near_tie":false,"group_logz":{"24":-5.16896915435791,"25":-0.03398323059082031,"26":-4.455045700073242}},"graph":{"mac_fp16":{"p":[0.005799485297682429,0.9823563508396558,0.011844163862661726],"max_abs_dp":5.013667370379338e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.005817056463643881,0.9822576713288415,0.011925272207514668],"max_abs_dp":0.00014881618451811907,"argmax_equal":true}}}]},{"id":"tv4s_06","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":634,"line_sha256":"33dbb66a760ddd24b894924618c488a497cf4fbd3f4e0a3945698c31a8c9a7b3","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0003","pair_id":"v3-legacy-transfer-deadline-0003","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0003","id":"contrastive/deadline/v3-legacy-transfer-deadline-0003/a","text_sha256":"518551635a353eb9b3666a2ad2e2a56c700352a0abe6b96235c7a9c300be1fcc","row_sha256":"85c34557c97a43a6b527f1eb5d81501fb3753466cf5d430d11814ce379883281","group_id":"v3-legacy-transfer-deadline-0003","variant":"clean"}},"request_sha256":"804d96a913c9736074bdfeceffbb076ec4c058cfe5f16370c804ee1672e83803","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"b16ec3de42a7c282d3dc00e2e24a6acecf72e51ef02e9c200ef21c047cf27bc8","oracle":{"probs":[0.43954698878513687,0.5523500806348068,0.008102930580056284],"argmax":"1","top2_margin":0.11280309184966997,"near_tie":false,"group_logz":{"24":-0.8357868194580078,"25":-0.6073493957519531,"26":-4.829305648803711}},"graph":{"mac_fp16":{"p":[0.4400725550539563,0.551851497292834,0.008075947653209561],"max_abs_dp":0.000525566268819444,"argmax_equal":true},"iphone_int8mlp":{"p":[0.4346940478778272,0.5570128395611968,0.008293112560975972],"max_abs_dp":0.004852940907309644,"argmax_equal":true}}}]},{"id":"tv4s_07","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":635,"line_sha256":"114624d7d0d4fb9882defe45b78a533ae1739a6832051a58f47cc6bd60140ed7","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0003","pair_id":"v3-legacy-transfer-deadline-0003","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0003","id":"contrastive/deadline/v3-legacy-transfer-deadline-0003/b","text_sha256":"bd59d3dc5d85b02f3cd8856ec2470b6dc904b52d2a2bb2ad74eb758e3dbde460","row_sha256":"919f3f660c4a5cf98310c25f8d46c27e2fb11b2a330231aa79c04f4bcbbc757f","group_id":"v3-legacy-transfer-deadline-0003","variant":"clean"}},"request_sha256":"e172d5c2b1edb4a06c4c64054cc9b0fc8437b733a5b26af71b8a7ce177234c36","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"7d2d294920867e9327fd8dc924ae088cb7cbeac79c433050514c037c83c165d0","oracle":{"probs":[0.04276505540521452,0.8694315837800853,0.08780336081470014],"argmax":"1","top2_margin":0.7816282229653853,"near_tie":false,"group_logz":{"24":-3.400498390197754,"25":-0.3883800506591797,"26":-2.681119918823242}},"graph":{"mac_fp16":{"p":[0.04271091760158631,0.8689689430822535,0.08832013931616027],"max_abs_dp":0.0005167785014601328,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04303723127672653,0.8702511699381328,0.08671159878514063],"max_abs_dp":0.0010917620295595037,"argmax_equal":true}}}]},{"id":"tv4s_08","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":636,"line_sha256":"a765a714967d23ff4dc39d3944f8352eafd9efd369a18e93e98bc33e1d398348","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0004","pair_id":"v3-legacy-transfer-deadline-0004","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0004","id":"contrastive/deadline/v3-legacy-transfer-deadline-0004/a","text_sha256":"3f3a8ad387844c8104129d70bdaf40a72b9270d90ca1064fd031494b11205761","row_sha256":"8bf056570b0810ee9edbbe9a8634216c9d234bd8e4693d56c0d1cf1b9c54550b","group_id":"v3-legacy-transfer-deadline-0004","variant":"clean"}},"request_sha256":"123305184c516a574bdf2286d5373851bf1a63edcb67108444bb77b4cd9397b9","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"c44728ab3e5f45339922613d115ef879f84de56b00fd6b88208e110dcbd635d9","oracle":{"probs":[0.10353186454301046,0.8876709568627331,0.008797178594256366],"argmax":"1","top2_margin":0.7841390923197227,"near_tie":false,"group_logz":{"24":-2.278238296508789,"25":-0.1295166015625,"26":-4.743686676025391}},"graph":{"mac_fp16":{"p":[0.10364968050784745,0.8875703242954268,0.008779995196725725],"max_abs_dp":0.0001178159648369953,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1021128981557456,0.8891385067883983,0.008748595055856104],"max_abs_dp":0.0014675499256652236,"argmax_equal":true}}}]},{"id":"tv4s_09","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":637,"line_sha256":"5de45ef0c7ab5bb5e65f043834bf8e3529bc53829e4a1107a8e3152bab600aab","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0004","pair_id":"v3-legacy-transfer-deadline-0004","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0004","id":"contrastive/deadline/v3-legacy-transfer-deadline-0004/b","text_sha256":"ac0c886d2f3cc92a0141cf91dea9b3936564915a88134052cbf3a900a7b1962c","row_sha256":"c45c3a04933b7389e59f16530e4238b39366230f23edd10407bee44131367d83","group_id":"v3-legacy-transfer-deadline-0004","variant":"clean"}},"request_sha256":"0ab68d300f2a164a34bf26b9b5145598cefe4eda316bbad6129a30ff9367ff67","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"5ddcd2888c92e7f93cf0789cd9b6dbfb8e5525a87b1887523f876f990951704e","oracle":{"probs":[0.005701996756747784,0.9875656705681668,0.0067323326750853875],"argmax":"1","top2_margin":0.9808333378930814,"near_tie":false,"group_logz":{"24":-5.174247741699219,"25":-0.0198211669921875,"26":-5.008142471313477}},"graph":{"mac_fp16":{"p":[0.005713505202666117,0.9875448064601742,0.0067416883371597],"max_abs_dp":2.086410799262417e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.005686260079305114,0.9876202338723779,0.006693506048317078],"max_abs_dp":5.456330421105271e-05,"argmax_equal":true}}}]},{"id":"tv4s_10","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":638,"line_sha256":"c61073b80cfc8ea149a1151d676bf3e13db2556631811137c3af0f56efa81c1b","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0005","pair_id":"v3-legacy-transfer-deadline-0005","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0005","id":"contrastive/deadline/v3-legacy-transfer-deadline-0005/a","text_sha256":"1dcf029b2f29eff1d5db377ecc9f7eaa5780943529cb6c0b6d9181e65ade83e3","row_sha256":"7f2ca1f8d91f857466e604bb40d6996f6fcc5e0bd16d44ae4dfc04af92beaf5a","group_id":"v3-legacy-transfer-deadline-0005","variant":"clean"}},"request_sha256":"161e27b58196e0accb6139336d62fdc11123edf5e656dfc330fcd4650a80ac6f","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"9f0c5bdd3ae98124d59cfe39215ed2633707e66ee6ec65d94879ee1d59e7b5f3","oracle":{"probs":[0.28630073021666574,0.7010419590735015,0.012657310709832701],"argmax":"1","top2_margin":0.4147412288568358,"near_tie":false,"group_logz":{"24":-1.264230728149414,"25":-0.36870574951171875,"26":-4.383038520812988}},"graph":{"mac_fp16":{"p":[0.2853849771312857,0.7019449010317038,0.012670121837010566],"max_abs_dp":0.0009157530853800555,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2819672053411038,0.7052459931251085,0.012786801533787749],"max_abs_dp":0.004333524875561934,"argmax_equal":true}}}]},{"id":"tv4s_11","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":639,"line_sha256":"534960fd2af36cb457fa4f6fd9c2df719245ece9077b1c07b0ee005a144ffa5e","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0005","pair_id":"v3-legacy-transfer-deadline-0005","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0005","id":"contrastive/deadline/v3-legacy-transfer-deadline-0005/b","text_sha256":"e57e6c96bf0b18642eaa06dbe450c57cce84d436022bd5342a4316459ea74c30","row_sha256":"5790faa6466eab4bc42fe96342c98c67ad9bb1f7ce190a0e8b2bfe73411bb04d","group_id":"v3-legacy-transfer-deadline-0005","variant":"clean"}},"request_sha256":"5d00db3ac11175b657a903d7664a2ad1548cc03b4241363e88b72b7fc9ac5715","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"b36fa9fbaa1d44e3c4e1e685eea1c178695e0eb702c4fd7e1f421e0a3dbeec2b","oracle":{"probs":[0.015542563893668038,0.7786464408571936,0.20581099524913837],"argmax":"1","top2_margin":0.5728354456080552,"near_tie":false,"group_logz":{"24":-4.257408142089844,"25":-0.3434333801269531,"26":-1.674032211303711}},"graph":{"mac_fp16":{"p":[0.015499368026395642,0.779039592029395,0.20546103994420942],"max_abs_dp":0.0003931511722014003,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015641599142791993,0.7872867651206703,0.1970716357365377],"max_abs_dp":0.008739359512600686,"argmax_equal":true}}}]},{"id":"tv4s_12","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":640,"line_sha256":"d89e9c1fb3287bd31df8864ffe9230682849e3b4fc858ee77985d2dd08610215","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0006","pair_id":"v3-legacy-transfer-deadline-0006","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0006","id":"contrastive/deadline/v3-legacy-transfer-deadline-0006/a","text_sha256":"d14936c609d1f5a21ffed3f0360912b15b4f17548707aa5563cd978a1c4aafb4","row_sha256":"dc10fe25f1999ebed57149c6c57398d043aa4429de09ed8ac2c67bebd111be68","group_id":"v3-legacy-transfer-deadline-0006","variant":"clean"}},"request_sha256":"a7f3b7af5b9c571c35c8cdd673c54d975761649c45af3a38cb64a9366ed3151f","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"5807232cc057c215044dda8617e61101de36e214e7747e890e44753b44a62a20","oracle":{"probs":[0.8008905293342318,0.1962838858026671,0.0028255848631011243],"argmax":"0","top2_margin":0.6046066435315647,"near_tie":false,"group_logz":{"24":-0.2361621856689453,"25":-1.642324447631836,"26":-5.883171081542969}},"graph":{"mac_fp16":{"p":[0.7999474230448184,0.19722227384065197,0.0028303031145296594],"max_abs_dp":0.000943106289413409,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7929671527986617,0.20416717598119544,0.002865671220142823],"max_abs_dp":0.00792337653557007,"argmax_equal":true}}}]},{"id":"tv4s_13","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":641,"line_sha256":"561594f1bd593e15f98950cbeb92744545b6d796f2f8c3cc3e387bfe45253cc3","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0006","pair_id":"v3-legacy-transfer-deadline-0006","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0006","id":"contrastive/deadline/v3-legacy-transfer-deadline-0006/b","text_sha256":"7b3c28f53dad88a83f61c0933cb56bfdfb3a40824db6e42baa3a7fae324798fe","row_sha256":"78a2493317239cef828d33d92f4b5a51068308871013b4dd794036288e4b0817","group_id":"v3-legacy-transfer-deadline-0006","variant":"clean"}},"request_sha256":"db86f143caa41c0db4d0b75d5e7bc7a88b9293a896bfdfe7d607282c42cb4282","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"fa34ed478d3eb4a554a0689dcda40cd63899dd67e82f2c0f41405221a78ae7c8","oracle":{"probs":[0.005144222902106952,0.9119210153364328,0.08293476176146025],"argmax":"1","top2_margin":0.8289862535749726,"near_tie":false,"group_logz":{"24":-5.293918609619141,"25":-0.11623954772949219,"26":-2.5137386322021484}},"graph":{"mac_fp16":{"p":[0.005156247001274795,0.9121310003505888,0.08271275264813645],"max_abs_dp":0.0002220091133238039,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0052434822570418165,0.9163195941514628,0.07843692359149543],"max_abs_dp":0.004497838169964824,"argmax_equal":true}}}]},{"id":"tv4s_14","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":642,"line_sha256":"ad0c3c2f83d0b7255f372845e32f0fb0468e0d7e30c28ca197e3a0862d36bf4d","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0007","pair_id":"v3-legacy-transfer-deadline-0007","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0007","id":"contrastive/deadline/v3-legacy-transfer-deadline-0007/a","text_sha256":"a8c60ecb11984dd3e71ff10ccfae662e1a0eb24e06718bc4d7e2b931cbc2079d","row_sha256":"4f170ff0f24194bfedc3d20274df3493d0e265d5405d6c6152e3aa9a3277a8b1","group_id":"v3-legacy-transfer-deadline-0007","variant":"clean"}},"request_sha256":"bf1fda25d0c4960f96c0bbd73363e372feacdb603ed87482e393f786c54caaf2","questions":[{"type":"score","n_options":3,"row_len":123,"slot":122,"row_ids_sha256":"47194b888170f37ef15dbb9e1cc0ac0bd57e28b0c3c43e98ccc14a72a6a2d046","oracle":{"probs":[0.012370437507861917,0.9765759226363191,0.011053639855819126],"argmax":"1","top2_margin":0.9642054851284572,"near_tie":false,"group_logz":{"24":-4.400759696960449,"25":-0.032016754150390625,"26":-4.513309478759766}},"graph":{"mac_fp16":{"p":[0.012391816058439068,0.9765534681789902,0.011054715762570697],"max_abs_dp":2.2454457328846544e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012005090135500534,0.9770665680398217,0.010928341824677704],"max_abs_dp":0.000490645403502632,"argmax_equal":true}}}]},{"id":"tv4s_15","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":643,"line_sha256":"14edae76354691257bb0934ba7b7c26d1e70808bf221c09a1d7fb38f267af59e","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0007","pair_id":"v3-legacy-transfer-deadline-0007","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0007","id":"contrastive/deadline/v3-legacy-transfer-deadline-0007/b","text_sha256":"13aa619399e3927530b663fc9d26e409c88b44a8c07a94eae8e93c06921e01f3","row_sha256":"9ddea24990c6956702d0a1f6fbb209c8f023cb3ad53c10fc742cb657b7511dca","group_id":"v3-legacy-transfer-deadline-0007","variant":"clean"}},"request_sha256":"040d14d0b70afb9132f9598fce575836986c02dec11e553f221a90fd78a7a6a3","questions":[{"type":"score","n_options":3,"row_len":123,"slot":122,"row_ids_sha256":"68f2c8a81bfd7a94b8634f34383cc71612b51b172d38d0492c14b3a0a5b8c8fb","oracle":{"probs":[0.01642489069769777,0.9670523414316275,0.0165227678706747],"argmax":"1","top2_margin":0.9505295735609528,"near_tie":false,"group_logz":{"24":-4.136928558349609,"25":-0.061473846435546875,"26":-4.130987167358398}},"graph":{"mac_fp16":{"p":[0.01638178329086242,0.9671442095562028,0.016474007152934865],"max_abs_dp":9.186812457528948e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016083656518152422,0.9669917489566331,0.016924594525214436],"max_abs_dp":0.00040182665453973587,"argmax_equal":true}}}]},{"id":"tv4s_16","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":644,"line_sha256":"cb5a7d9081a6c3f92c7c45f39da41cdb7ad3aee86bbb7ab0c247fe7c9fcf0c63","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0008","pair_id":"v3-legacy-transfer-deadline-0008","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0008","id":"contrastive/deadline/v3-legacy-transfer-deadline-0008/a","text_sha256":"c5742086c74a5a53a5f11ed184c3d611193196f6b8cb4ac7c3b66e84329f065a","row_sha256":"94bceb24c84ef2def72f044a3a45915a324ddac91d3f53ae01d9ac46999cb78e","group_id":"v3-legacy-transfer-deadline-0008","variant":"clean"}},"request_sha256":"cf8156454d70026c2494a46f644ae51d2d740b1959560e8cd3b094f4c4694616","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"bc968abba1f2b62bbb72e2b124999547bbb70adbe0f301b198bdb4a87ef909c1","oracle":{"probs":[0.4302137814833975,0.5660611073705676,0.0037251111460348597],"argmax":"1","top2_margin":0.13584732588717008,"near_tie":false,"group_logz":{"24":-0.8528804779052734,"25":-0.578460693359375,"26":-5.6020660400390625}},"graph":{"mac_fp16":{"p":[0.43415178300943086,0.5621426772581272,0.0037055397324420375],"max_abs_dp":0.003938001526033352,"argmax_equal":true},"iphone_int8mlp":{"p":[0.42741944125826364,0.5688428615861941,0.00373769715554229],"max_abs_dp":0.0027943402251338623,"argmax_equal":true}}}]},{"id":"tv4s_17","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":645,"line_sha256":"6ae6e1de56a3db4c782e8c8e148a3892dc45e8d61c9c43818a1fb9ee88132141","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0008","pair_id":"v3-legacy-transfer-deadline-0008","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0008","id":"contrastive/deadline/v3-legacy-transfer-deadline-0008/b","text_sha256":"b3dd1c44cae9495cd28ff312f20d319e07fabb962bc61138dade8c7adf524e53","row_sha256":"d15eca907b2a8c547bb96b02c92209e884685393cc0772fb5f0b73f26d6295bb","group_id":"v3-legacy-transfer-deadline-0008","variant":"clean"}},"request_sha256":"02c9c3d4b228907170255d936b98a44beaacb35416692a4f8f10660d8aea0751","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"c4de0de7f6e5efbafe619337fe1fc47e9974632bfc56df9bc01f3a8d7059040d","oracle":{"probs":[0.006869109001634941,0.955206032602511,0.037924858395854154],"argmax":"1","top2_margin":0.9172811742066569,"near_tie":false,"group_logz":{"24":-5.0573272705078125,"25":-0.12243461608886719,"26":-3.3487548828125}},"graph":{"mac_fp16":{"p":[0.006845830544245207,0.9552247895003279,0.03792937995542691],"max_abs_dp":2.3278457389734435e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.007057819893754109,0.9551787683211509,0.03776341178509498],"max_abs_dp":0.00018871089211916752,"argmax_equal":true}}}]},{"id":"tv4s_18","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":646,"line_sha256":"005acdd744e3968a240a71f22909468ccec057466c2cb3dd915f6c6d7a3464aa","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0009","pair_id":"v3-legacy-transfer-deadline-0009","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0009","id":"contrastive/deadline/v3-legacy-transfer-deadline-0009/a","text_sha256":"089647dcec080abecf6cfadc5cd00e0e6c03fcb94c92fb12e512539825233c2e","row_sha256":"9b334f8aef425a2b55f28ce20e2ddda7eb742cdc6d8d03cb8c06e223e9ffdbf8","group_id":"v3-legacy-transfer-deadline-0009","variant":"clean"}},"request_sha256":"3b87633347b3b74dd88a6af26be3f735fd1284a8cfc39edabb3516cc7456facd","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"23923280dbcd6fd4a2795edbcda3633287ec8ccf843a37f22c38ad2122e9bba2","oracle":{"probs":[0.9601093048250166,0.03740291907666823,0.002487776098315198],"argmax":"0","top2_margin":0.9227063857483483,"near_tie":false,"group_logz":{"24":-0.05495643615722656,"25":-3.3002548217773438,"26":-6.010614395141602}},"graph":{"mac_fp16":{"p":[0.9600412371407516,0.03747570770450632,0.002483055154742175],"max_abs_dp":7.278862783809126e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9588965729619778,0.03858895423081402,0.0025144728072081326],"max_abs_dp":0.0012127318630387185,"argmax_equal":true}}}]},{"id":"tv4s_19","source":"tv4s","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","line":647,"line_sha256":"3c27d4e6538298d765eda08c221aee2fd929076b0588bb11fa60155466644b26","_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0009","pair_id":"v3-legacy-transfer-deadline-0009","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0009","id":"contrastive/deadline/v3-legacy-transfer-deadline-0009/b","text_sha256":"f332427ac84752956e94a0a0ae03443cf06bab5a255a0a240071bbce45da78fe","row_sha256":"e13cca24963a4dba6e0ef1311192f27e8a1d9f016ab4a51493be86a18c00ecb6","group_id":"v3-legacy-transfer-deadline-0009","variant":"clean"}},"request_sha256":"6080a1b7a3c0f407eb61eb84ca9ce3f63dbb3411113f632c28b2ce5eabf3be6d","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"d5edde228b17b58f936e2c860b971d3c02eca4e597289980f2db587be4ea1bea","oracle":{"probs":[0.0065849193505234565,0.9884861990948577,0.004928881554618801],"argmax":"1","top2_margin":0.9819012797443343,"near_tie":false,"group_logz":{"24":-5.031009674072266,"25":-0.019617080688476562,"26":-5.320679664611816}},"graph":{"mac_fp16":{"p":[0.006564607271109844,0.9885215176882236,0.004913875040666572],"max_abs_dp":3.531859336591836e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0065705229127393265,0.9886091098582808,0.004820367228979905],"max_abs_dp":0.0001229107634230786,"argmax_equal":true}}}]},{"id":"semif_a3f18f3a63d45345942b","source":"semif","publication":"publish_text","request":{"state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 0 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":0,"line_sha256":"29b8ab32b468adfba83998a9f0161d285d4ad10a4dfd8fe310f40ae2f5a64567","licence":"MIT","semif_id":"a3f18f3a63d45345942b","group_id":"2fdaa8a61e6e989dd866","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":76,"slot":75,"row_ids_sha256":"64bccb2d1abf3681facee9f5a72542a36106429c6e52f97dfae1e992801b9c28","oracle":{"probs":[0.012133949356806195,0.026071893907275722,0.9617941567359181],"argmax":"contradicted","top2_margin":0.9357222628286423,"near_tie":false,"group_logz":{"41":-4.418430328369141,"334":-12.901017189025879,"42":-3.6535797119140625,"378":-11.46681022644043,"43":-0.04563713073730469,"340":-8.431144714355469}},"graph":{"mac_fp16":{"p":[0.012136207799222254,0.026050424635484536,0.9618133675652933],"max_abs_dp":2.1469271791186434e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012088734354520582,0.025948674535824997,0.9619625911096544],"max_abs_dp":0.00016843437373637027,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.012133949356806195,0.026071893907275722,0.9617941567359181]],"input_tokens":76,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9617941567359181,"probabilities":{"supported":0.012133949356806195,"insufficient":0.026071893907275722,"contradicted":0.9617941567359181}}},"usage":{"input_tokens":76,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.315}},{"id":"semif_f40beba9088c8db8bbd6","source":"semif","publication":"publish_text","request":{"state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: replacement lenses were ordered.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 1 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":1,"line_sha256":"4ac1b71277f4c327373bff0295ba8fc36c0d2efc3cf3e5b8e3aeff3b43d81d87","licence":"MIT","semif_id":"f40beba9088c8db8bbd6","group_id":"2fdaa8a61e6e989dd866","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":74,"slot":73,"row_ids_sha256":"93c8c1eee84b1f777c7ad4aa87fdd810dd204393d0baecf3d7c808843f1a1010","oracle":{"probs":[0.9607202003213008,0.02789632129349822,0.011383478385201056],"argmax":"supported","top2_margin":0.9328238790278026,"near_tie":false,"group_logz":{"41":-0.047336578369140625,"334":-8.758323669433594,"42":-3.5865249633789062,"378":-11.72152328491211,"43":-4.482856750488281,"340":-12.810026168823242}},"graph":{"mac_fp16":{"p":[0.9607169872787084,0.027884609564394897,0.011398403156896735],"max_abs_dp":1.4924771695679073e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9610653052224822,0.02760213876800682,0.011332556009510957],"max_abs_dp":0.00034510490118144777,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9607202003213008,0.02789632129349822,0.011383478385201056]],"input_tokens":74,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9607202003213008,"probabilities":{"supported":0.9607202003213008,"insufficient":0.02789632129349822,"contradicted":0.011383478385201056}}},"usage":{"input_tokens":74,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3019}},{"id":"semif_0b43ea8e24e74d621f1b","source":"semif","publication":"publish_text","request":{"state":"The workshop confirms it fitted the replacement lenses to the customer's glasses.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 2 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":2,"line_sha256":"3ec20897f955bcd3f7eafbbfc3c5046e56828a44e61bc910c2e6501a91317553","licence":"MIT","semif_id":"0b43ea8e24e74d621f1b","group_id":"2fdaa8a61e6e989dd866","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":68,"slot":67,"row_ids_sha256":"d076e38a001cd2607a72cf971b4a1a622bd1eee7c323fd89b45ccba9fdfc53cd","oracle":{"probs":[0.03331417547630556,0.9549797775959883,0.011706046927706232],"argmax":"supported","top2_margin":0.9216656021196827,"near_tie":false,"group_logz":{"41":-3.4130115509033203,"334":-11.05691909790039,"42":-0.05730438232421875,"378":-7.628443717956543,"43":-4.458889007568359,"340":-12.03843879699707}},"graph":{"mac_fp16":{"p":[0.03331372854298944,0.9549914482878533,0.011694823169157258],"max_abs_dp":1.1670691865028893e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03325470136169396,0.9550748562447832,0.011670442393522872],"max_abs_dp":9.507864879487737e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03331417547630556,0.9549797775959883,0.011706046927706232]],"input_tokens":68,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9549797775959883,"probabilities":{"insufficient":0.03331417547630556,"supported":0.9549797775959883,"contradicted":0.011706046927706232}}},"usage":{"input_tokens":68,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3018}},{"id":"semif_942ec65dac0bfb9874f9","source":"semif","publication":"publish_text","request":{"state":"The optician records a glasses prescription but no order or fitting status.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 3 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":3,"line_sha256":"0f9aaf4ac9d9718cf8bdb0d1a5f8d9a14ef51bace747b9c4c4146165d3e829cb","licence":"MIT","semif_id":"942ec65dac0bfb9874f9","group_id":"2fdaa8a61e6e989dd866","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"insufficient","row_len":68,"slot":67,"row_ids_sha256":"c9eef3e2579c6c1fd817e0fddc1340a3b2acb95dc57ed50dcddf76d008843ed6","oracle":{"probs":[0.8440112112486663,0.024290314830393903,0.13169847392093978],"argmax":"insufficient","top2_margin":0.7123127373277265,"near_tie":false,"group_logz":{"41":-0.17984771728515625,"334":-8.144991874694824,"42":-3.727935791015625,"378":-11.277973175048828,"43":-2.0374984741210938,"340":-9.436553955078125}},"graph":{"mac_fp16":{"p":[0.8438141881761142,0.02426307419696414,0.13192273762692167],"max_abs_dp":0.00022426370598188639,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8458032238266677,0.02427454691089699,0.1299222292624353],"max_abs_dp":0.001792012578001434,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8440112112486663,0.024290314830393903,0.13169847392093978]],"input_tokens":68,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8440112112486663,"probabilities":{"insufficient":0.8440112112486663,"supported":0.024290314830393903,"contradicted":0.13169847392093978}}},"usage":{"input_tokens":68,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3047}},{"id":"semif_e0c140e9222d1bce5647","source":"semif","publication":"publish_text","request":{"state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 4 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":4,"line_sha256":"ee9d8c375b6acc354f7420edd5d92646d64e60c9bc78eeb549283a1e430680ff","licence":"MIT","semif_id":"e0c140e9222d1bce5647","group_id":"f4bea3d3bcc01056740d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"supported","row_len":83,"slot":82,"row_ids_sha256":"05effdab27de6e1a06d81d29c2d61580e5ad495925e1b50a7893dff274b5ca2e","oracle":{"probs":[0.953247693049077,0.013832645018015358,0.03291966193290773],"argmax":"supported","top2_margin":0.9203280311161692,"near_tie":false,"group_logz":{"41":-0.056903839111328125,"334":-8.168096542358398,"42":-4.28974723815918,"378":-12.131593704223633,"43":-3.422708511352539,"340":-11.02421760559082}},"graph":{"mac_fp16":{"p":[0.9533288853472699,0.013816445179608327,0.032854669473121864],"max_abs_dp":8.119229819292251e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.954118662116462,0.013638012537630705,0.032243325345907375],"max_abs_dp":0.0008709690673850412,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.953247693049077,0.013832645018015358,0.03291966193290773]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.953247693049077,"probabilities":{"supported":0.953247693049077,"contradicted":0.013832645018015358,"insufficient":0.03291966193290773}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3334}},{"id":"semif_8e8c3804a3c15ebb31e3","source":"semif","publication":"publish_text","request":{"state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the guest disputes the towel statement.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 5 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":5,"line_sha256":"d04fb857b655f9545341ef8874d1ac7adb53915cda8c0a95bb83bd9e18f9c914","licence":"MIT","semif_id":"8e8c3804a3c15ebb31e3","group_id":"f4bea3d3bcc01056740d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":80,"slot":79,"row_ids_sha256":"818ae3cc3e33a9c52fc8e1598b7c08e4efe1b2f91cd94da48142d06ea18e84f7","oracle":{"probs":[0.5998779608004364,0.04013914017585696,0.35998289902370656],"argmax":"insufficient","top2_margin":0.23989506177672987,"near_tie":false,"group_logz":{"41":-0.5190811157226562,"334":-8.272184371948242,"42":-3.2234554290771484,"378":-10.447509765625,"43":-1.0297508239746094,"340":-8.451512336730957}},"graph":{"mac_fp16":{"p":[0.5983762421597949,0.04005170831547836,0.3615720495247267],"max_abs_dp":0.0015891505010201534,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5913898328124007,0.039376378912920985,0.3692337882746784],"max_abs_dp":0.009250889250971839,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5998779608004364,0.04013914017585696,0.35998289902370656]],"input_tokens":80,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5998779608004364,"probabilities":{"insufficient":0.5998779608004364,"supported":0.04013914017585696,"contradicted":0.35998289902370656}}},"usage":{"input_tokens":80,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3134}},{"id":"semif_56c419b12cd37cbe790c","source":"semif","publication":"publish_text","request":{"state":"The guest writes, \"The listing says towels are supplied, but none were provided. I dispute that statement.\"","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 6 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":6,"line_sha256":"349e0655f52d59d16af361363c7409ec2c2bbd940b1b76e2670d848b311469f2","licence":"MIT","semif_id":"56c419b12cd37cbe790c","group_id":"f4bea3d3bcc01056740d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":81,"slot":80,"row_ids_sha256":"4c1015491c35fd08d2fb79268a8bcda73d5f1d17c477ad6fe11fcf00663b6c7e","oracle":{"probs":[0.93394782514435,0.030994729258969175,0.03505744559668092],"argmax":"contradicted","top2_margin":0.8988903795476691,"near_tie":false,"group_logz":{"41":-0.07492256164550781,"334":-8.166675567626953,"42":-3.4805259704589844,"378":-11.081184387207031,"43":-3.3573551177978516,"340":-10.874034881591797}},"graph":{"mac_fp16":{"p":[0.934195153195683,0.030858198547212515,0.03494664825710447],"max_abs_dp":0.0002473280513329845,"argmax_equal":true},"iphone_int8mlp":{"p":[0.934840996388888,0.030604174676069008,0.03455482893504308],"max_abs_dp":0.0008931712445379159,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.93394782514435,0.030994729258969175,0.03505744559668092]],"input_tokens":81,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.93394782514435,"probabilities":{"contradicted":0.93394782514435,"insufficient":0.030994729258969175,"supported":0.03505744559668092}}},"usage":{"input_tokens":81,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4846}},{"id":"semif_75b470586834df8c568d","source":"semif","publication":"publish_text","request":{"state":"The guest forwards the listing but provides no comment about towels or the stay.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 7 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":7,"line_sha256":"6f393466824b6b141fa200643f214eab240c4900031248bb95dddc1de96afe85","licence":"MIT","semif_id":"75b470586834df8c568d","group_id":"f4bea3d3bcc01056740d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":73,"slot":72,"row_ids_sha256":"77f22319e7bde88b73ea80a27e41c4241ac45aaa11c742091f57581eb40136b4","oracle":{"probs":[0.018783667857431335,0.15378733289690427,0.8274289992456645],"argmax":"insufficient","top2_margin":0.6736416663487602,"near_tie":false,"group_logz":{"41":-3.9861507415771484,"334":-11.65626335144043,"42":-1.8835678100585938,"378":-8.964201927185059,"43":-0.20081520080566406,"340":-7.546473503112793}},"graph":{"mac_fp16":{"p":[0.01880372030114218,0.15374830349567972,0.8274479762031781],"max_abs_dp":3.9029401224549565e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01875486049858581,0.15354829189512353,0.8276968476062906],"max_abs_dp":0.0002678483606260995,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.018783667857431335,0.15378733289690427,0.8274289992456645]],"input_tokens":73,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8274289992456645,"probabilities":{"supported":0.018783667857431335,"contradicted":0.15378733289690427,"insufficient":0.8274289992456645}}},"usage":{"input_tokens":73,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3076}},{"id":"semif_834f3b3e10cc61b33d25","source":"semif","publication":"publish_text","request":{"state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 8 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":8,"line_sha256":"76b1450aa3220b19b8ce53fefccda80a38b3eb8d7cbc0ca8400101e847e8db2b","licence":"MIT","semif_id":"834f3b3e10cc61b33d25","group_id":"b51b300da2936decc6b7","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":67,"slot":66,"row_ids_sha256":"408fd91c06f7ad267be9f2f521b55247736196b7f165e4d2697e492d285543c1","oracle":{"probs":[0.01021820211227544,0.014663878073798915,0.9751179198139256],"argmax":"contradicted","top2_margin":0.9604540417401267,"near_tie":false,"group_logz":{"41":-4.589244842529297,"334":-13.298940658569336,"42":-4.228028297424316,"378":-12.31968879699707,"43":-0.030857086181640625,"340":-8.685701370239258}},"graph":{"mac_fp16":{"p":[0.010224840197946955,0.0146694395084033,0.9751057202936496],"max_abs_dp":1.219952027597948e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010223952415613824,0.014639896449748748,0.9751361511346374],"max_abs_dp":2.3981624050166753e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.01021820211227544,0.014663878073798915,0.9751179198139256]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9751179198139256,"probabilities":{"supported":0.01021820211227544,"insufficient":0.014663878073798915,"contradicted":0.9751179198139256}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3482}},{"id":"semif_76c7a79b1ff2363a60c5","source":"semif","publication":"publish_text","request":{"state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: terminal Cedar is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 9 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":9,"line_sha256":"9c62ce7a7f27588def2dbb1b9ff59f23ee91ce9eb28ea07dbc63fe353ac47628","licence":"MIT","semif_id":"76c7a79b1ff2363a60c5","group_id":"b51b300da2936decc6b7","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":68,"slot":67,"row_ids_sha256":"e932ca2983fe5fd591e738b9205a0b57eab4480c8010788e916761984af870ab","oracle":{"probs":[0.9773743610443005,0.013766529447091445,0.008859109508608002],"argmax":"supported","top2_margin":0.9636078315972091,"near_tie":false,"group_logz":{"41":-0.030080795288085938,"334":-8.901388168334961,"42":-4.292710304260254,"378":-12.626955032348633,"43":-4.733504295349121,"340":-13.11411190032959}},"graph":{"mac_fp16":{"p":[0.9773776815918277,0.013764975528799234,0.008857342879373124],"max_abs_dp":3.3205475271635976e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9774320748190435,0.01372539246785622,0.008842532713100258],"max_abs_dp":5.771377474295125e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9773743610443005,0.013766529447091445,0.008859109508608002]],"input_tokens":68,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9773743610443005,"probabilities":{"supported":0.9773743610443005,"insufficient":0.013766529447091445,"contradicted":0.008859109508608002}}},"usage":{"input_tokens":68,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3002}},{"id":"semif_36c71bb649b7d5eccb1f","source":"semif","publication":"publish_text","request":{"state":"The technician confirms terminal Maple is online and terminal Cedar is offline.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 10 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":10,"line_sha256":"ae4e718b2c02a7701141321eb3b593d0affa2a859c0c4743609abb3e870737eb","licence":"MIT","semif_id":"36c71bb649b7d5eccb1f","group_id":"b51b300da2936decc6b7","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":66,"slot":65,"row_ids_sha256":"77384eeb2ffe9efe297f5d927a873e137b83c5100f4ff35f552952a5ee18a7ad","oracle":{"probs":[0.9781057630472405,0.0131055248165852,0.008788712136174192],"argmax":"supported","top2_margin":0.9650002382306553,"near_tie":false,"group_logz":{"41":-0.029188156127929688,"334":-8.994743347167969,"42":-4.341772079467773,"378":-12.770783424377441,"43":-4.741337776184082,"340":-13.211067199707031}},"graph":{"mac_fp16":{"p":[0.9780551544015423,0.013131788612404213,0.00881305698605344],"max_abs_dp":5.060864569816914e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9781849017172877,0.013043986182379659,0.00877111210033265],"max_abs_dp":7.913867004716302e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9781057630472405,0.0131055248165852,0.008788712136174192]],"input_tokens":66,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9781057630472405,"probabilities":{"supported":0.9781057630472405,"insufficient":0.0131055248165852,"contradicted":0.008788712136174192}}},"usage":{"input_tokens":66,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3072}},{"id":"semif_eac5121a4a1bf391a08c","source":"semif","publication":"publish_text","request":{"state":"The technician lists both terminals but omits their connection status.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 11 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":11,"line_sha256":"d1a5522627e5139c6006816eb3ba1c08ef1773d89f080fc5fa70fa4611f4ed34","licence":"MIT","semif_id":"eac5121a4a1bf391a08c","group_id":"b51b300da2936decc6b7","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"insufficient","row_len":64,"slot":63,"row_ids_sha256":"d36a5f069d3a1350bc3149b0594c1d9701753c50a15271412d98e3104b8e5726","oracle":{"probs":[0.04805516738485983,0.9128304776792986,0.03911435493584158],"argmax":"insufficient","top2_margin":0.8647753102944388,"near_tie":false,"group_logz":{"41":-3.0461883544921875,"334":-11.136890411376953,"42":-0.10198783874511719,"378":-7.956967353820801,"43":-3.2520484924316406,"340":-10.841327667236328}},"graph":{"mac_fp16":{"p":[0.04796409602429535,0.9129479166268055,0.039087987348899164],"max_abs_dp":0.00011743894750693151,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04850213935801705,0.9120093882897219,0.03948847235226115],"max_abs_dp":0.0008210893895767013,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.04805516738485983,0.9128304776792986,0.03911435493584158]],"input_tokens":64,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9128304776792986,"probabilities":{"contradicted":0.04805516738485983,"insufficient":0.9128304776792986,"supported":0.03911435493584158}}},"usage":{"input_tokens":64,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.2578}},{"id":"semif_44574c44e243ee1b2f52","source":"semif","publication":"publish_text","request":{"state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 12 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":12,"line_sha256":"cdddec91b355222b28583f322688f4fa77b2e8ee81412acd5b6580ab6e58f908","licence":"MIT","semif_id":"44574c44e243ee1b2f52","group_id":"e4cfd694c80e08d3e6a5","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","contradicted","supported"],"codes":["A","B","C"],"gold":"supported","row_len":77,"slot":76,"row_ids_sha256":"30aadcc77de2d3f27b94779dfd1a19730d9e158f8d0a157162c1654eb6934017","oracle":{"probs":[0.010335649118614006,0.011846332135357409,0.9778180187460286],"argmax":"supported","top2_margin":0.9659716866106711,"near_tie":false,"group_logz":{"41":-4.580345153808594,"334":-12.97797966003418,"42":-4.443925857543945,"378":-12.331560134887695,"43":-0.030620574951171875,"340":-8.32755184173584}},"graph":{"mac_fp16":{"p":[0.01034640890347426,0.011844024229873224,0.9778095668666524],"max_abs_dp":1.0759784860253988e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010340237126683149,0.011773029769177839,0.9778867331041391],"max_abs_dp":7.330236617957009e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.010335649118614006,0.011846332135357409,0.9778180187460286]],"input_tokens":77,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9778180187460286,"probabilities":{"insufficient":0.010335649118614006,"contradicted":0.011846332135357409,"supported":0.9778180187460286}}},"usage":{"input_tokens":77,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3148}},{"id":"semif_2608370d01783735c3ea","source":"semif","publication":"publish_text","request":{"state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the corrected date is in June.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 13 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":13,"line_sha256":"65202a7c669b4c8cfb51dfec194db6cc276eb53faf4d4d844d54c2172fd77e66","licence":"MIT","semif_id":"2608370d01783735c3ea","group_id":"e4cfd694c80e08d3e6a5","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":77,"slot":76,"row_ids_sha256":"65a277ec4b9b414dc812777f120cf479ebcd827d131def6d279a2aaec22df96d","oracle":{"probs":[0.02065043376820408,0.02887999971154495,0.950469566520251],"argmax":"contradicted","top2_margin":0.9215895668087061,"near_tie":false,"group_logz":{"41":-3.8871097564697266,"334":-12.269529342651367,"42":-3.55169677734375,"378":-11.363555908203125,"43":-0.05788993835449219,"340":-8.481659889221191}},"graph":{"mac_fp16":{"p":[0.02066594480894145,0.028920287192775465,0.950413767998283],"max_abs_dp":5.5798521968020154e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.020718361115308535,0.028971966775890222,0.9503096721088011],"max_abs_dp":0.00015989441144992345,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.02065043376820408,0.02887999971154495,0.950469566520251]],"input_tokens":77,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.950469566520251,"probabilities":{"supported":0.02065043376820408,"insufficient":0.02887999971154495,"contradicted":0.950469566520251}}},"usage":{"input_tokens":77,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3158}},{"id":"semif_ec94ef909f7918695ab3","source":"semif","publication":"publish_text","request":{"state":"The catalogue author corrects the date to June and expressly rejects the earlier July date.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 14 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":14,"line_sha256":"7e868b0a7eb30ff0aee625f4a03ab859682dde6dcbd30266d8eadb5e927b112e","licence":"MIT","semif_id":"ec94ef909f7918695ab3","group_id":"e4cfd694c80e08d3e6a5","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":72,"slot":71,"row_ids_sha256":"739d48cd1b0a36aeebaff0e04b2318c4428770bdc6a64e08dfcebff1728d6d36","oracle":{"probs":[0.023663147141410906,0.020848964742245978,0.9554878881163431],"argmax":"contradicted","top2_margin":0.9318247409749322,"near_tie":false,"group_logz":{"41":-3.7498435974121094,"334":-12.209550857543945,"42":-3.876458168029785,"378":-11.833864212036133,"43":-0.051540374755859375,"340":-8.628643989562988}},"graph":{"mac_fp16":{"p":[0.023701971439939507,0.02087721796124078,0.9554208105988198],"max_abs_dp":6.707751752332847e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.023653779663282755,0.02071839578948075,0.9556278245472366],"max_abs_dp":0.00013993643089349472,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.023663147141410906,0.020848964742245978,0.9554878881163431]],"input_tokens":72,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9554878881163431,"probabilities":{"supported":0.023663147141410906,"insufficient":0.020848964742245978,"contradicted":0.9554878881163431}}},"usage":{"input_tokens":72,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3064}},{"id":"semif_03d919fd0e9b3a7cd1e6","source":"semif","publication":"publish_text","request":{"state":"The catalogue mentions a date correction but gives no corrected date.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 15 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":15,"line_sha256":"f71db05caba1ba82d5dae19a020f1d1e42da9ee9e92d740c2a8916ce5f69b11d","licence":"MIT","semif_id":"03d919fd0e9b3a7cd1e6","group_id":"e4cfd694c80e08d3e6a5","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"insufficient","row_len":66,"slot":65,"row_ids_sha256":"ef85e0ab5b742c18debca159b542bfafa5e065b7401f40142b7a8bb3e52bdf8d","oracle":{"probs":[0.9245918078319716,0.02418392100942946,0.051224271158598914],"argmax":"insufficient","top2_margin":0.8733675366733726,"near_tie":false,"group_logz":{"41":-0.08588218688964844,"334":-8.451149940490723,"42":-3.729546546936035,"378":-11.60865592956543,"43":-2.9790210723876953,"340":-10.790528297424316}},"graph":{"mac_fp16":{"p":[0.9245912357287426,0.024171951319128006,0.05123681295212939],"max_abs_dp":1.2541793530475298e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9246157223789596,0.02400159127920298,0.05138268634183735],"max_abs_dp":0.00018232973022648188,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9245918078319716,0.02418392100942946,0.051224271158598914]],"input_tokens":66,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9245918078319716,"probabilities":{"insufficient":0.9245918078319716,"supported":0.02418392100942946,"contradicted":0.051224271158598914}}},"usage":{"input_tokens":66,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3086}},{"id":"semif_86a15618ba4cb90a307e","source":"semif","publication":"publish_text","request":{"state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 16 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":16,"line_sha256":"e961d35381dda6fbc265a61f30504a82a683402af31886df9587524c3a7617b2","licence":"MIT","semif_id":"86a15618ba4cb90a307e","group_id":"20460d4ae0db46091e84","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":70,"slot":69,"row_ids_sha256":"96a00476ace12c368097e3b7ad9f3022c65567051fe015ff86313495ceb7039c","oracle":{"probs":[0.9581444114119021,0.02330248985878404,0.01855309872931384],"argmax":"contradicted","top2_margin":0.9348419215531181,"near_tie":false,"group_logz":{"41":-0.04803657531738281,"334":-8.592386245727539,"42":-3.764474868774414,"378":-11.698209762573242,"43":-3.992398262023926,"340":-11.967803955078125}},"graph":{"mac_fp16":{"p":[0.9581561718864129,0.023296876186595122,0.01854695192699201],"max_abs_dp":1.1760474510791319e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9584038414271839,0.02309219629769674,0.018503962275119336],"max_abs_dp":0.0002594300152818363,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9581444114119021,0.02330248985878404,0.01855309872931384]],"input_tokens":70,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9581444114119021,"probabilities":{"contradicted":0.9581444114119021,"insufficient":0.02330248985878404,"supported":0.01855309872931384}}},"usage":{"input_tokens":70,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3385}},{"id":"semif_3dd9c3ac98f8ddf7bdd3","source":"semif","publication":"publish_text","request":{"state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the document contains weighing instructions.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 17 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":17,"line_sha256":"9562948573ca3dca83a4f1fbf27c92833506d1e25101a05f65bb8ee0babfcdf4","licence":"MIT","semif_id":"3dd9c3ac98f8ddf7bdd3","group_id":"20460d4ae0db46091e84","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","supported","insufficient"],"codes":["A","B","C"],"gold":"supported","row_len":70,"slot":69,"row_ids_sha256":"71eebe99ed8958af709c0493ce7409b68c0e67140495a5eadf81ae03ce904d7a","oracle":{"probs":[0.018872303726553925,0.9645954616364145,0.016532234637031542],"argmax":"supported","top2_margin":0.9457231579098606,"near_tie":false,"group_logz":{"41":-3.979867935180664,"334":-12.041240692138672,"42":-0.04585456848144531,"378":-7.943476676940918,"43":-4.112251281738281,"340":-11.869376182556152}},"graph":{"mac_fp16":{"p":[0.018856786534800114,0.9645874393215639,0.01655577414363605],"max_abs_dp":2.3539506604507848e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.019044086560965956,0.9643444249467233,0.016611488492310676],"max_abs_dp":0.0002510366896911753,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.018872303726553925,0.9645954616364145,0.016532234637031542]],"input_tokens":70,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9645954616364145,"probabilities":{"contradicted":0.018872303726553925,"supported":0.9645954616364145,"insufficient":0.016532234637031542}}},"usage":{"input_tokens":70,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3137}},{"id":"semif_36273dd4777876a26338","source":"semif","publication":"publish_text","request":{"state":"The intern performed the weighing and recorded the parcel's measured weight.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 18 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":18,"line_sha256":"a7d05364023ebd6b3f2345ee8a27858e9eb847ad13efd01abb622534d0c33747","licence":"MIT","semif_id":"36273dd4777876a26338","group_id":"20460d4ae0db46091e84","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","supported","insufficient"],"codes":["A","B","C"],"gold":"supported","row_len":66,"slot":65,"row_ids_sha256":"d808cb1d7f14fe42289a0346ae953fe43492216609f7fde5ec9f65764bb2bb78","oracle":{"probs":[0.01077183238088659,0.9669142651511873,0.02231390246792618],"argmax":"supported","top2_margin":0.9446003626832611,"near_tie":false,"group_logz":{"41":-4.5402984619140625,"334":-12.39756965637207,"42":-0.04312324523925781,"378":-7.622689247131348,"43":-3.812023162841797,"340":-11.19572925567627}},"graph":{"mac_fp16":{"p":[0.010798832446977906,0.9668900402630594,0.022311127289962727],"max_abs_dp":2.7000066091315694e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.010698052049047294,0.9671023321210545,0.02219961582989824],"max_abs_dp":0.0001880669698671733,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.01077183238088659,0.9669142651511873,0.02231390246792618]],"input_tokens":66,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9669142651511873,"probabilities":{"contradicted":0.01077183238088659,"supported":0.9669142651511873,"insufficient":0.02231390246792618}}},"usage":{"input_tokens":66,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3387}},{"id":"semif_07ee70cfe4b01330ce89","source":"semif","publication":"publish_text","request":{"state":"The intern supplies a parcel identifier but no procedure or measurement information.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 19 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":19,"line_sha256":"533fb6800d779dd8e7dc311c2306a94d9e677cd44869c5ad7b82811b09d3bcf8","licence":"MIT","semif_id":"07ee70cfe4b01330ce89","group_id":"20460d4ae0db46091e84","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":66,"slot":65,"row_ids_sha256":"39c8a44148702748a9268daac71eb385c668fb5a01e696f0d5b2f3f8b077f411","oracle":{"probs":[0.026919367951491562,0.21218295071584006,0.7608976813326684],"argmax":"insufficient","top2_margin":0.5487147306168284,"near_tie":false,"group_logz":{"41":-3.628298759460449,"334":-11.394659042358398,"42":-1.5636959075927734,"378":-8.782023429870605,"43":-0.28664588928222656,"340":-7.689432144165039}},"graph":{"mac_fp16":{"p":[0.026985200562276025,0.21200731001023654,0.7610074894274874],"max_abs_dp":0.00017564070560352474,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02673355372297131,0.20500130851737922,0.7682651377596494],"max_abs_dp":0.0073674564269810006,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.026919367951491562,0.21218295071584006,0.7608976813326684]],"input_tokens":66,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.7608976813326684,"probabilities":{"supported":0.026919367951491562,"contradicted":0.21218295071584006,"insufficient":0.7608976813326684}}},"usage":{"input_tokens":66,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.323}},{"id":"semif_51ceab3ce622c7c759f1","source":"semif","publication":"publish_text","request":{"state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 20 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":20,"line_sha256":"19782c41e501d7ebdbf94bdad95f8b87112cc5f7ade586d3fecb50ebb17d1534","licence":"MIT","semif_id":"51ceab3ce622c7c759f1","group_id":"51b91ba31984c5939c1c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":84,"slot":83,"row_ids_sha256":"b6a1678e2b34ab24a9eab079dc94bbfa773252c20720b811eeb2cde43ac7085a","oracle":{"probs":[0.01613858285953829,0.021602339937194407,0.9622590772032673],"argmax":"supported","top2_margin":0.9406567372660729,"near_tie":false,"group_logz":{"41":-4.137869834899902,"334":-11.997360229492188,"42":-3.846281051635742,"378":-11.144548416137695,"43":-0.04979896545410156,"340":-7.790713310241699}},"graph":{"mac_fp16":{"p":[0.01613310523520686,0.02158614608452823,0.9622807486802648],"max_abs_dp":2.167147699749794e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016069274008645107,0.02149166720517923,0.9624390587861757],"max_abs_dp":0.00017998158290843946,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.01613858285953829,0.021602339937194407,0.9622590772032673]],"input_tokens":84,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9622590772032673,"probabilities":{"contradicted":0.01613858285953829,"insufficient":0.021602339937194407,"supported":0.9622590772032673}}},"usage":{"input_tokens":84,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3485}},{"id":"semif_b8091be929b1e489fcc4","source":"semif","publication":"publish_text","request":{"state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: a real egg was discovered.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 21 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":21,"line_sha256":"5d90512f35f0acf6cc6d77b69fb8afde56ad13a2760c3a3c617eb7a21c1d699a","licence":"MIT","semif_id":"b8091be929b1e489fcc4","group_id":"51b91ba31984c5939c1c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"contradicted","row_len":84,"slot":83,"row_ids_sha256":"9148ac6dcbe1ded45c3dc8b7492945889eaf8016ed8b6403d3859037ad151fbd","oracle":{"probs":[0.013933004568171927,0.9510123573891058,0.03505463804272228],"argmax":"contradicted","top2_margin":0.9159577193463836,"near_tie":false,"group_logz":{"41":-4.282403945922852,"334":-11.991755485534668,"42":-0.05913734436035156,"378":-7.517152786254883,"43":-3.3597564697265625,"340":-10.597423553466797}},"graph":{"mac_fp16":{"p":[0.01395086831772432,0.9509108579921883,0.03513827369008734],"max_abs_dp":0.00010149939691750376,"argmax_equal":true},"iphone_int8mlp":{"p":[0.013806200322000773,0.95136584863658,0.03482795104141918],"max_abs_dp":0.00035349124747419136,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.013933004568171927,0.9510123573891058,0.03505463804272228]],"input_tokens":84,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9510123573891058,"probabilities":{"supported":0.013933004568171927,"contradicted":0.9510123573891058,"insufficient":0.03505463804272228}}},"usage":{"input_tokens":84,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3593}},{"id":"semif_c89b86e3f55123bfa506","source":"semif","publication":"publish_text","request":{"state":"The note describes a real bird egg discovered outside a performance and expressly says the discovery was not fictional.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 22 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":22,"line_sha256":"c54b46a9d2ae82f50224c67196af5e61ad3dd1ea1dfd67ba7285a3782b4f1f97","licence":"MIT","semif_id":"c89b86e3f55123bfa506","group_id":"51b91ba31984c5939c1c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"contradicted","row_len":73,"slot":72,"row_ids_sha256":"f1cdf53c9e34af80edb51988203ce89b24844f9f6e502c07905d733da62bfd15","oracle":{"probs":[0.024762119864940757,0.9472671826163139,0.02797069751874528],"argmax":"contradicted","top2_margin":0.9192964850975687,"near_tie":false,"group_logz":{"41":-3.7070236206054688,"334":-11.22244644165039,"42":-0.06275749206542969,"378":-7.52678108215332,"43":-3.58518123626709,"340":-10.805156707763672}},"graph":{"mac_fp16":{"p":[0.024735354711780747,0.947326637625112,0.02793800766310718],"max_abs_dp":5.945500879811494e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.024373668946487514,0.9477413558005783,0.027884975252934138],"max_abs_dp":0.0004741731842643482,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.024762119864940757,0.9472671826163139,0.02797069751874528]],"input_tokens":73,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9472671826163139,"probabilities":{"supported":0.024762119864940757,"contradicted":0.9472671826163139,"insufficient":0.02797069751874528}}},"usage":{"input_tokens":73,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3482}},{"id":"semif_3b0958cabc339005cf66","source":"semif","publication":"publish_text","request":{"state":"A note mentions an egg without describing a discovery or whether the account is fictional.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 23 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":23,"line_sha256":"a70dec8b593bda8561766b8997d1c67b7b77bac3b68cbfe63cce2180800d461f","licence":"MIT","semif_id":"3b0958cabc339005cf66","group_id":"51b91ba31984c5939c1c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"insufficient","row_len":69,"slot":68,"row_ids_sha256":"ebed7311694ec3478693bb5f7d0ba8e39bcc18e8f64c9eb0568549b37bd26987","oracle":{"probs":[0.9118147091533432,0.04868823638025259,0.03949705446640432],"argmax":"insufficient","top2_margin":0.8631264727730906,"near_tie":false,"group_logz":{"41":-0.10367202758789062,"334":-8.285820960998535,"42":-3.0336713790893555,"378":-10.741714477539062,"43":-3.24288272857666,"340":-10.935715675354004}},"graph":{"mac_fp16":{"p":[0.9115855529637362,0.0487748401889092,0.03963960684735453],"max_abs_dp":0.0002291561896069494,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9118484781881455,0.04841126136820957,0.039740260443644815],"max_abs_dp":0.0002769750120430181,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9118147091533432,0.04868823638025259,0.03949705446640432]],"input_tokens":69,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9118147091533432,"probabilities":{"insufficient":0.9118147091533432,"supported":0.04868823638025259,"contradicted":0.03949705446640432}}},"usage":{"input_tokens":69,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3375}},{"id":"semif_f2b4ec4930322fe45118","source":"semif","publication":"publish_text","request":{"state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"answer":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 24 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":24,"line_sha256":"6d8455b84b5dc54273b2b3339ff883c772369514195228f70588de34550b5149","licence":"MIT","semif_id":"f2b4ec4930322fe45118","group_id":"770c1607f0a11f88f81e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"prohibited","row_len":86,"slot":85,"row_ids_sha256":"235feb3b722ee35ddc44ec6a93027254efd76f761bd61ad3a13fa1a93da2d7ea","oracle":{"probs":[0.0392582699116555,0.9433214602079277,0.017420269880416683],"argmax":"prohibited","top2_margin":0.9040631902962722,"near_tie":false,"group_logz":{"41":-3.2468490600585938,"334":-10.748239517211914,"42":-0.06760406494140625,"378":-7.614670753479004,"43":-4.0593767166137695,"340":-11.438989639282227}},"graph":{"mac_fp16":{"p":[0.03923588440486526,0.943357867856025,0.017406247739109756],"max_abs_dp":3.6407648097314116e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03856382922985258,0.944228688838642,0.017207481931505446],"max_abs_dp":0.0009072286307142408,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.0392582699116555,0.9433214602079277,0.017420269880416683]],"input_tokens":86,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9433214602079277,"probabilities":{"insufficient":0.0392582699116555,"prohibited":0.9433214602079277,"permitted":0.017420269880416683}}},"usage":{"input_tokens":86,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3471}},{"id":"semif_61ac20b07a5189231e15","source":"semif","publication":"publish_text","request":{"state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"answer":{"type":"choice","instructions":"Internal training reproduction is allowed when the architect permits that use. May the plan appear in the training booklet?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 25 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":25,"line_sha256":"691bc05476ac4d932caea0932682f0b3f36fa71aec4943f4dd64b220f27a6598","licence":"MIT","semif_id":"61ac20b07a5189231e15","group_id":"770c1607f0a11f88f81e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":91,"slot":90,"row_ids_sha256":"194f7ef94d3cf2d76428675c326b96cf25da83889ef14f2930565a4e06bd1d4a","oracle":{"probs":[0.015243960736725914,0.956553660201955,0.028202379061319213],"argmax":"permitted","top2_margin":0.9283512811406358,"near_tie":false,"group_logz":{"41":-4.19643497467041,"334":-11.999608993530273,"42":-0.057281494140625,"378":-7.594764709472656,"43":-3.581212043762207,"340":-10.909177780151367}},"graph":{"mac_fp16":{"p":[0.015262802701624872,0.9565437583615224,0.028193438936852767],"max_abs_dp":1.8841964898957952e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015289691915970577,0.956440511036401,0.02826979704762834],"max_abs_dp":0.0001131491655539385,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.015243960736725914,0.956553660201955,0.028202379061319213]],"input_tokens":91,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.956553660201955,"probabilities":{"prohibited":0.015243960736725914,"permitted":0.956553660201955,"insufficient":0.028202379061319213}}},"usage":{"input_tokens":91,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3467}},{"id":"semif_4b3d794c004566610181","source":"semif","publication":"publish_text","request":{"state":"The architect expressly permits selling copies of the plan.","questions":{"answer":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 26 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":26,"line_sha256":"538c612d90b7bf9c593bbf4d6d800603851735619c737db79576f0dde0eddc5f","licence":"MIT","semif_id":"4b3d794c004566610181","group_id":"770c1607f0a11f88f81e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":77,"slot":76,"row_ids_sha256":"42e5d4b890612f0a613b28a445e741f846cbf65e2da98b93549fecb038207d89","oracle":{"probs":[0.03608568698602408,0.061292741926420785,0.9026215710875553],"argmax":"permitted","top2_margin":0.8413288291611345,"near_tie":false,"group_logz":{"41":-3.337733268737793,"334":-10.791507720947266,"42":-2.8079681396484375,"378":-9.701664924621582,"43":-0.11832618713378906,"340":-7.4884233474731445}},"graph":{"mac_fp16":{"p":[0.036045450837060684,0.06120512130729042,0.9027494278556489],"max_abs_dp":0.00012785676809368152,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0364448193834514,0.0610670278582172,0.9024881527583314],"max_abs_dp":0.00035913239742731834,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03608568698602408,0.061292741926420785,0.9026215710875553]],"input_tokens":77,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.9026215710875553,"probabilities":{"prohibited":0.03608568698602408,"insufficient":0.061292741926420785,"permitted":0.9026215710875553}}},"usage":{"input_tokens":77,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3415}},{"id":"semif_b7c4e62d07986286fce0","source":"semif","publication":"publish_text","request":{"state":"An architect's permission letter exists, but the uses it allows are omitted.","questions":{"answer":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 27 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":27,"line_sha256":"4377841ac694a8560da15ba4a1806c65d7eadae2a049789cc592ad4dc4f1ecf0","licence":"MIT","semif_id":"b7c4e62d07986286fce0","group_id":"770c1607f0a11f88f81e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","permitted","prohibited"],"codes":["A","B","C"],"gold":"insufficient","row_len":82,"slot":81,"row_ids_sha256":"2b6a5439e5266dec8addf549ca0a2ef60fb92c383960a8cce62572242d4778c0","oracle":{"probs":[0.6731627501614286,0.133775803416283,0.1930614464222884],"argmax":"insufficient","top2_margin":0.4801013037391402,"near_tie":false,"group_logz":{"41":-0.41539573669433594,"334":-7.769467353820801,"42":-2.031217575073242,"378":-8.976231575012207,"43":-1.6643743515014648,"340":-8.663263320922852}},"graph":{"mac_fp16":{"p":[0.6733884435764005,0.13397325551956474,0.19263830090403486],"max_abs_dp":0.0004231455182535371,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6698201637993195,0.13360568869502976,0.19657414750565075],"max_abs_dp":0.0035127010833623518,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.6731627501614286,0.133775803416283,0.1930614464222884]],"input_tokens":82,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6731627501614286,"probabilities":{"insufficient":0.6731627501614286,"permitted":0.133775803416283,"prohibited":0.1930614464222884}}},"usage":{"input_tokens":82,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3812}},{"id":"semif_9473603b44af63beab79","source":"semif","publication":"publish_text","request":{"state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"answer":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 28 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":28,"line_sha256":"78bf3deaa931770bb9559d841f27b4bf9726a925240b3a1b419d76a8e3fc44da","licence":"MIT","semif_id":"9473603b44af63beab79","group_id":"ffd081ada3bd9be9207a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":83,"slot":82,"row_ids_sha256":"aa6bf7874cee6015dfadfea70e67be0a11e1b5ca474f16f4955043600037780c","oracle":{"probs":[0.8882316001067374,0.09385778749481022,0.017910612398452302],"argmax":"permitted","top2_margin":0.7943738126119272,"near_tie":false,"group_logz":{"41":-0.12923431396484375,"334":-7.909018516540527,"42":-2.3766860961914062,"378":-9.699871063232422,"43":-4.033073425292969,"340":-11.569986343383789}},"graph":{"mac_fp16":{"p":[0.888262299317294,0.09385759450545128,0.0178801061772548],"max_abs_dp":3.069921055653957e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8878601621879731,0.09410224702710901,0.018037590784917762],"max_abs_dp":0.00037143791876426135,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8882316001067374,0.09385778749481022,0.017910612398452302]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8882316001067374,"probabilities":{"permitted":0.8882316001067374,"insufficient":0.09385778749481022,"prohibited":0.017910612398452302}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3459}},{"id":"semif_d41072dc35d8cd14c056","source":"semif","publication":"publish_text","request":{"state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"answer":{"type":"choice","instructions":"Under the alternative rule, the badge requires both first-aid and mountain-navigation training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 29 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":29,"line_sha256":"dc9d8d209c1446787c544c6eb8084619438d667bf170c61286828a982efca928","licence":"MIT","semif_id":"d41072dc35d8cd14c056","group_id":"ffd081ada3bd9be9207a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":89,"slot":88,"row_ids_sha256":"a8097338ea76a0658a96cfac38b888380ac9a1f0bb88f8adc0ec0d04409968ff","oracle":{"probs":[0.03149320251089711,0.9059081077411554,0.06259868974794745],"argmax":"prohibited","top2_margin":0.8433094179932079,"near_tie":false,"group_logz":{"41":-3.4692907333374023,"334":-10.490440368652344,"42":-0.11012458801269531,"378":-7.020866394042969,"43":-2.782318115234375,"340":-9.431905746459961}},"graph":{"mac_fp16":{"p":[0.03148795803010232,0.9060758092826907,0.062436232687207105],"max_abs_dp":0.00016770154153533667,"argmax_equal":true},"iphone_int8mlp":{"p":[0.031135272395705873,0.9064637895475405,0.06240093805675359],"max_abs_dp":0.0005556818063850999,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03149320251089711,0.9059081077411554,0.06259868974794745]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9059081077411554,"probabilities":{"permitted":0.03149320251089711,"prohibited":0.9059081077411554,"insufficient":0.06259868974794745}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3301}},{"id":"semif_33d6e5da58f0fee2490d","source":"semif","publication":"publish_text","request":{"state":"The guide completed mountain-navigation training but not first-aid training.","questions":{"answer":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 30 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":30,"line_sha256":"96bcadb9989a5c0102c65f37f6b6faa10f1baae9b1be69436a5ec7464777ca36","licence":"MIT","semif_id":"33d6e5da58f0fee2490d","group_id":"ffd081ada3bd9be9207a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"prohibited","row_len":83,"slot":82,"row_ids_sha256":"f3c089097fb44a20ff04a08e17029fe885b6869cf85549893ed86d1296266191","oracle":{"probs":[0.5480284090006339,0.292924732062749,0.15904685893661707],"argmax":"prohibited","top2_margin":0.2551036769378849,"near_tie":false,"group_logz":{"41":-0.6203556060791016,"334":-7.762153625488281,"42":-1.2467670440673828,"378":-8.031811714172363,"43":-1.8574838638305664,"340":-8.583991050720215}},"graph":{"mac_fp16":{"p":[0.5462157439015011,0.29361806181260414,0.16016619428589482],"max_abs_dp":0.0018126650991328042,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5676886833789831,0.2839410946029395,0.1483702220180774],"max_abs_dp":0.019660274378349274,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5480284090006339,0.292924732062749,0.15904685893661707]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.5480284090006339,"probabilities":{"prohibited":0.5480284090006339,"insufficient":0.292924732062749,"permitted":0.15904685893661707}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3072}},{"id":"semif_823bf736b92e9d8fa66d","source":"semif","publication":"publish_text","request":{"state":"The guide registered for training, but completion information is not supplied.","questions":{"answer":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 31 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":31,"line_sha256":"765beadff20a28e6d0841e473392c30fb786c7a0610017fcb5a7c66fe4f3d1f0","licence":"MIT","semif_id":"823bf736b92e9d8fa66d","group_id":"ffd081ada3bd9be9207a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"insufficient","row_len":82,"slot":81,"row_ids_sha256":"667da782767362cd0a7ab42d7130b41bef949ce7520e05cf344ec213f22e8c62","oracle":{"probs":[0.07854372740856806,0.4028461173237135,0.5186101552677185],"argmax":"permitted","top2_margin":0.11576403794400497,"near_tie":false,"group_logz":{"41":-2.5694236755371094,"334":-9.646844863891602,"42":-0.9345245361328125,"378":-7.669573783874512,"43":-0.6819267272949219,"340":-7.412057876586914}},"graph":{"mac_fp16":{"p":[0.07855712518463019,0.40295457725957345,0.5184882975557964],"max_abs_dp":0.00012185771192207984,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07860345087255084,0.4069925016685935,0.5144040474588557],"max_abs_dp":0.004206107808862769,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.07854372740856806,0.4028461173237135,0.5186101552677185]],"input_tokens":82,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.5186101552677185,"probabilities":{"prohibited":0.07854372740856806,"insufficient":0.4028461173237135,"permitted":0.5186101552677185}}},"usage":{"input_tokens":82,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3097}},{"id":"semif_8a4c3de28be95c105ca1","source":"semif","publication":"publish_text","request":{"state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"answer":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 32 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":32,"line_sha256":"560f828590ab917169978b2220ad1f8706dd25fefef175d34b1e715a957492a2","licence":"MIT","semif_id":"8a4c3de28be95c105ca1","group_id":"2c3a0cb61f7bae78739e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"prohibited","row_len":84,"slot":83,"row_ids_sha256":"11d3e2c3706b24cf415c636d4c53f2bdac602ee64561454a382b65ff749a1a6d","oracle":{"probs":[0.014341024828977604,0.048768201887951515,0.9368907732830708],"argmax":"prohibited","top2_margin":0.8881225713951193,"near_tie":false,"group_logz":{"41":-4.255091667175293,"334":-12.146198272705078,"42":-3.031137466430664,"378":-10.222267150878906,"43":-0.07564926147460938,"340":-7.775973320007324}},"graph":{"mac_fp16":{"p":[0.014334209189989428,0.048988212596581586,0.936677578213429],"max_abs_dp":0.00022001070863007094,"argmax_equal":true},"iphone_int8mlp":{"p":[0.014308595891786744,0.04827051453032559,0.9374208895778877],"max_abs_dp":0.000530116294816918,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.014341024828977604,0.048768201887951515,0.9368907732830708]],"input_tokens":84,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9368907732830708,"probabilities":{"permitted":0.014341024828977604,"insufficient":0.048768201887951515,"prohibited":0.9368907732830708}}},"usage":{"input_tokens":84,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3127}},{"id":"semif_ea7ec66a5e8eac886548","source":"semif","publication":"publish_text","request":{"state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"answer":{"type":"choice","instructions":"A sample may be made when sample production is approved. May the sample be made?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 33 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":33,"line_sha256":"3dddf198bace7e3ba583635c4c62832b623c417e823edfb6e123bc7f79d68449","licence":"MIT","semif_id":"ea7ec66a5e8eac886548","group_id":"2c3a0cb61f7bae78739e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":86,"slot":85,"row_ids_sha256":"b64a717fb2f111c3d815a05d0d8e27f7e3c785a5bc1bb770e6a417fc82fd9c93","oracle":{"probs":[0.8356211223874521,0.11859858817148489,0.045780289441062924],"argmax":"permitted","top2_margin":0.7170225342159673,"near_tie":false,"group_logz":{"41":-0.19477081298828125,"334":-7.764487266540527,"42":-2.1472015380859375,"378":-9.279454231262207,"43":-3.099092483520508,"340":-10.277840614318848}},"graph":{"mac_fp16":{"p":[0.8355985960196999,0.11859935732512457,0.045802046655175566],"max_abs_dp":2.252636775224026e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.836538411217731,0.11777768532979706,0.04568390345247201],"max_abs_dp":0.0009172888302788618,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8356211223874521,0.11859858817148489,0.045780289441062924]],"input_tokens":86,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8356211223874521,"probabilities":{"permitted":0.8356211223874521,"insufficient":0.11859858817148489,"prohibited":0.045780289441062924}}},"usage":{"input_tokens":86,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3097}},{"id":"semif_25172cd6ddc49ab0ca70","source":"semif","publication":"publish_text","request":{"state":"The designer explicitly approves mass production of the badge.","questions":{"answer":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 34 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":34,"line_sha256":"36b77284c665a4d63e1e59d9374caad429406ff0cf87b2fc63d79ca6e543189b","licence":"MIT","semif_id":"25172cd6ddc49ab0ca70","group_id":"2c3a0cb61f7bae78739e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":77,"slot":76,"row_ids_sha256":"f9b1df8586e4325ff0e16d28ab4a8b1f5d2c5240cb477147f58082c2a2a28988","oracle":{"probs":[0.05551971517625919,0.08596175540202038,0.8585185294217205],"argmax":"permitted","top2_margin":0.7725567740197001,"near_tie":false,"group_logz":{"41":-2.9088497161865234,"334":-10.279293060302734,"42":-2.4716854095458984,"378":-9.307035446166992,"43":-0.170379638671875,"340":-7.37919807434082}},"graph":{"mac_fp16":{"p":[0.055463550999553336,0.08599421997369039,0.8585422290267563],"max_abs_dp":5.616417670585572e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.05355539114282612,0.0830672646110024,0.8633773442461714],"max_abs_dp":0.004858814824450941,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.05551971517625919,0.08596175540202038,0.8585185294217205]],"input_tokens":77,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8585185294217205,"probabilities":{"prohibited":0.05551971517625919,"insufficient":0.08596175540202038,"permitted":0.8585185294217205}}},"usage":{"input_tokens":77,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.297}},{"id":"semif_fb73a85d386cf59b0c51","source":"semif","publication":"publish_text","request":{"state":"The designer issued a decision but its production scope is unavailable.","questions":{"answer":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 35 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":35,"line_sha256":"f0515999b33f09466d28e49066220dbc124fe28abbc7070c088264edd06ca7c5","licence":"MIT","semif_id":"fb73a85d386cf59b0c51","group_id":"2c3a0cb61f7bae78739e","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","permitted","prohibited"],"codes":["A","B","C"],"gold":"insufficient","row_len":78,"slot":77,"row_ids_sha256":"bf9c81d89a2c5ae242cf698c0cbc5ccf55fa295de44f62cd425d187e530270b2","oracle":{"probs":[0.2793823964083356,0.022714569566389647,0.6979030340252746],"argmax":"prohibited","top2_margin":0.41852063761693903,"near_tie":false,"group_logz":{"41":-1.2863292694091797,"334":-8.514945030212402,"42":-3.7959041595458984,"378":-10.703216552734375,"43":-0.3708305358886719,"340":-7.5107221603393555}},"graph":{"mac_fp16":{"p":[0.2799015213825074,0.02272169474072508,0.6973767838767675],"max_abs_dp":0.0005262501485071569,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2782364445807105,0.02265293727683437,0.6991106181424552],"max_abs_dp":0.0012075841171805823,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.2793823964083356,0.022714569566389647,0.6979030340252746]],"input_tokens":78,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.6979030340252746,"probabilities":{"insufficient":0.2793823964083356,"permitted":0.022714569566389647,"prohibited":0.6979030340252746}}},"usage":{"input_tokens":78,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3326}},{"id":"semif_a02fd582d2881cc462b2","source":"semif","publication":"publish_text","request":{"state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"answer":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 36 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":36,"line_sha256":"cbc3172ae1e36e1c1bc71774eaa79185254051a851fbc70c6d2a68d0dbaa237f","licence":"MIT","semif_id":"a02fd582d2881cc462b2","group_id":"42ed4cbfde7f2f64274b","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":90,"slot":89,"row_ids_sha256":"ee01c9ee1db5b03a7fb9781f0c58204bc069b665b800055d91276fa26892d014","oracle":{"probs":[0.014364182150472238,0.025422821565218686,0.960212996284309],"argmax":"permitted","top2_margin":0.9347901747190903,"near_tie":false,"group_logz":{"41":-4.253437042236328,"334":-12.010326385498047,"42":-3.682527542114258,"378":-10.946203231811523,"43":-0.05101966857910156,"340":-7.824175834655762}},"graph":{"mac_fp16":{"p":[0.014366944471752404,0.025408969380010565,0.960224086148237],"max_abs_dp":1.385218520812137e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.014362666057103209,0.025179470786589636,0.9604578631563072],"max_abs_dp":0.0002448668719982061,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.014364182150472238,0.025422821565218686,0.960212996284309]],"input_tokens":90,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.960212996284309,"probabilities":{"prohibited":0.014364182150472238,"insufficient":0.025422821565218686,"permitted":0.960212996284309}}},"usage":{"input_tokens":90,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3186}},{"id":"semif_19f37d5a97034bd7d04f","source":"semif","publication":"publish_text","request":{"state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"answer":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the rooftop terrace?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 37 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":37,"line_sha256":"f392adb5e2fd2be74666008c525e6015935c1f18e12e55f5f9baa9abfd3316a1","licence":"MIT","semif_id":"19f37d5a97034bd7d04f","group_id":"42ed4cbfde7f2f64274b","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"prohibited","row_len":93,"slot":92,"row_ids_sha256":"327242bedd0b663a1f094245323a94da97935d89a07d7ad9712732504fecc0fb","oracle":{"probs":[0.03759554743785107,0.9445955583073518,0.017808894254797144],"argmax":"prohibited","top2_margin":0.9070000108695007,"near_tie":false,"group_logz":{"41":-3.2891311645507812,"334":-10.589012145996094,"42":-0.06525993347167969,"378":-7.372825622558594,"43":-4.036318778991699,"340":-11.210683822631836}},"graph":{"mac_fp16":{"p":[0.03760862605760574,0.944604734453316,0.01778663948907815],"max_abs_dp":2.2254765718994485e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03724991586562452,0.9450736099632794,0.017676474171096152],"max_abs_dp":0.0004780516559276249,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03759554743785107,0.9445955583073518,0.017808894254797144]],"input_tokens":93,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9445955583073518,"probabilities":{"insufficient":0.03759554743785107,"prohibited":0.9445955583073518,"permitted":0.017808894254797144}}},"usage":{"input_tokens":93,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3192}},{"id":"semif_fda0ca94652c6d739ecf","source":"semif","publication":"publish_text","request":{"state":"The wristband expressly excludes the sculpture garden and covers only the rooftop terrace.","questions":{"answer":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 38 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":38,"line_sha256":"b2df3c2405ec4dff81cc7fb84c3062e3b233298e2ae9101c0401509b05b5f49c","licence":"MIT","semif_id":"fda0ca94652c6d739ecf","group_id":"42ed4cbfde7f2f64274b","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":89,"slot":88,"row_ids_sha256":"b1171298190579750e24a83c4bdcbed92ee2e22244fe3578362971ef44bfd2ae","oracle":{"probs":[0.9378669041996914,0.020879702077917475,0.041253393722391],"argmax":"prohibited","top2_margin":0.8966135104773004,"near_tie":false,"group_logz":{"41":-0.07191848754882812,"334":-7.766524314880371,"42":-3.876749038696289,"378":-11.13078498840332,"43":-3.1957931518554688,"340":-10.33889389038086}},"graph":{"mac_fp16":{"p":[0.938034062223987,0.02083165163800822,0.041134286138004725],"max_abs_dp":0.00016715802429556614,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9382631742626432,0.020881114300138105,0.0408557114372188],"max_abs_dp":0.0003976822851722017,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9378669041996914,0.020879702077917475,0.041253393722391]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9378669041996914,"probabilities":{"prohibited":0.9378669041996914,"permitted":0.020879702077917475,"insufficient":0.041253393722391}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3136}},{"id":"semif_d0e36c4cb10a29b3191c","source":"semif","publication":"publish_text","request":{"state":"The visitor has a wristband but its covered areas are not supplied.","questions":{"answer":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 39 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":39,"line_sha256":"72cc14e30f353d2d774556d9ce81674ed3cfbc77dded3451f23e5115af6faae3","licence":"MIT","semif_id":"d0e36c4cb10a29b3191c","group_id":"42ed4cbfde7f2f64274b","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","permitted","prohibited"],"codes":["A","B","C"],"gold":"insufficient","row_len":85,"slot":84,"row_ids_sha256":"586c46c10e01fd9495f217508bf04886d23c16ddfa7f1f1bac223172ba63fede","oracle":{"probs":[0.40532614006050927,0.038015963355997495,0.5566578965834932],"argmax":"prohibited","top2_margin":0.15133175652298392,"near_tie":false,"group_logz":{"41":-0.9171142578125,"334":-8.266411781311035,"42":-3.2838001251220703,"378":-10.301095008850098,"43":-0.5998554229736328,"340":-7.76055908203125}},"graph":{"mac_fp16":{"p":[0.40518655529182374,0.03805294540579169,0.5567604993023846],"max_abs_dp":0.00013958476868553005,"argmax_equal":true},"iphone_int8mlp":{"p":[0.40434558671930426,0.038147231148936936,0.5575071821317588],"max_abs_dp":0.0009805533412050038,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.40532614006050927,0.038015963355997495,0.5566578965834932]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.5566578965834932,"probabilities":{"insufficient":0.40532614006050927,"permitted":0.038015963355997495,"prohibited":0.5566578965834932}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3063}},{"id":"semif_a0a39ace7e5337d038a7","source":"semif","publication":"publish_text","request":{"state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"answer":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 40 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":40,"line_sha256":"656219530edfc4681c9aadc7f1a955a264b4d5a5ec885b66dce867002129716e","licence":"MIT","semif_id":"a0a39ace7e5337d038a7","group_id":"6dd193172b9240a04bf9","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":89,"slot":88,"row_ids_sha256":"b24510de975dffdfd49275ffbf6d06f26a1efc822b1841e7ebfe40abc0e402df","oracle":{"probs":[0.9766490773100043,0.009471000353407266,0.013879922336588451],"argmax":"prohibited","top2_margin":0.9627691549734159,"near_tie":false,"group_logz":{"41":-0.02923583984375,"334":-8.92763900756836,"42":-4.665128707885742,"378":-12.992975234985352,"43":-4.282919883728027,"340":-12.586028099060059}},"graph":{"mac_fp16":{"p":[0.9766259590266253,0.009478316208495324,0.013895724764879398],"max_abs_dp":2.311828337897559e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9769324098817873,0.009405666915838877,0.01366192320237388],"max_abs_dp":0.00028333257178303484,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9766490773100043,0.009471000353407266,0.013879922336588451]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9766490773100043,"probabilities":{"prohibited":0.9766490773100043,"permitted":0.009471000353407266,"insufficient":0.013879922336588451}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3113}},{"id":"semif_eeae283841ce75dd25bf","source":"semif","publication":"publish_text","request":{"state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"answer":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the display name be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 41 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":41,"line_sha256":"dd3219239ebef5a1aeccb55b7e45527a66dca4a5fdd1584d526267221e7ff317","licence":"MIT","semif_id":"eeae283841ce75dd25bf","group_id":"6dd193172b9240a04bf9","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","permitted","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":89,"slot":88,"row_ids_sha256":"ad5d63c224d43040ce0fb809b5e31b9bb002813799fae6726a7e271211bb2b0e","oracle":{"probs":[0.013624903907824091,0.9757917520144408,0.010583344077735084],"argmax":"permitted","top2_margin":0.9621668481066167,"near_tie":false,"group_logz":{"41":-4.304120063781738,"334":-12.273296356201172,"42":-0.03277015686035156,"378":-7.834596633911133,"43":-4.556737899780273,"340":-12.213696479797363}},"graph":{"mac_fp16":{"p":[0.013656234330689576,0.9757536034004749,0.01059016226883553],"max_abs_dp":3.814861396589375e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01367421568310739,0.9757093294042103,0.010616454912682262],"max_abs_dp":8.242261023050368e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.013624903907824091,0.9757917520144408,0.010583344077735084]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.9757917520144408,"probabilities":{"insufficient":0.013624903907824091,"permitted":0.9757917520144408,"prohibited":0.010583344077735084}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3088}},{"id":"semif_00bb4cc043f7d2b288d4","source":"semif","publication":"publish_text","request":{"state":"The owner expressly permits changing the recovery email.","questions":{"answer":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 42 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":42,"line_sha256":"6b3d818c9910289316980359d766a984aede09c2fb8340f43b616a7c42131903","licence":"MIT","semif_id":"00bb4cc043f7d2b288d4","group_id":"6dd193172b9240a04bf9","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":81,"slot":80,"row_ids_sha256":"4fe59871b2581edc3c50375da5a44d3d906e8f404715fb04dbaefc98fb20313a","oracle":{"probs":[0.012905986907737293,0.0088276647204948,0.978266348371768],"argmax":"permitted","top2_margin":0.9653603614640307,"near_tie":false,"group_logz":{"41":-4.358186721801758,"334":-12.28799057006836,"42":-4.737987518310547,"378":-12.24251937866211,"43":-0.030096054077148438,"340":-8.023372650146484}},"graph":{"mac_fp16":{"p":[0.012864858352462706,0.00881883940392609,0.9783163022436112],"max_abs_dp":4.9953871843255726e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012808548190575761,0.008789035574641285,0.978402416234783],"max_abs_dp":0.00013606786301501028,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.012905986907737293,0.0088276647204948,0.978266348371768]],"input_tokens":81,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.978266348371768,"probabilities":{"insufficient":0.012905986907737293,"prohibited":0.0088276647204948,"permitted":0.978266348371768}}},"usage":{"input_tokens":81,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3006}},{"id":"semif_1a81ead69e93e5d0c86a","source":"semif","publication":"publish_text","request":{"state":"The owner requested an account update, but the fields to be changed are omitted.","questions":{"answer":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 43 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":43,"line_sha256":"231636717ff1e5be74db376f1f853588cb0d45d16ecef89598310be521faf0e9","licence":"MIT","semif_id":"1a81ead69e93e5d0c86a","group_id":"6dd193172b9240a04bf9","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":88,"slot":87,"row_ids_sha256":"3708eee958ee7d85ac686eca677b80289589aa77ac456d5eeae23e0f74872fae","oracle":{"probs":[0.0890625017999338,0.31081183910634697,0.6001256590937193],"argmax":"insufficient","top2_margin":0.2893138199873723,"near_tie":false,"group_logz":{"41":-2.4381160736083984,"334":-9.654348373413086,"42":-1.1882667541503906,"378":-8.027667999267578,"43":-0.5303153991699219,"340":-7.394813537597656}},"graph":{"mac_fp16":{"p":[0.08924484928875517,0.31000926215474917,0.6007458885564957],"max_abs_dp":0.0008025769515977998,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0877345094747073,0.316846873157966,0.5954186173673267],"max_abs_dp":0.006035034051619037,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.0890625017999338,0.31081183910634697,0.6001256590937193]],"input_tokens":88,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6001256590937193,"probabilities":{"permitted":0.0890625017999338,"prohibited":0.31081183910634697,"insufficient":0.6001256590937193}}},"usage":{"input_tokens":88,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3106}},{"id":"semif_4444dad25e438f128673","source":"semif","publication":"publish_text","request":{"state":"The applicant lives in the valley but works outside it.","questions":{"answer":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 44 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":44,"line_sha256":"604bcae1c371242120a51a3c2d4669647190da2376be505243cdcf352e5eeba7","licence":"MIT","semif_id":"4444dad25e438f128673","group_id":"3200eb252d6737e03027","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":82,"slot":81,"row_ids_sha256":"cb10c172915b5faaa70173f6674e4c47b74aa739d94370ff3793eae918f30d19","oracle":{"probs":[0.35778233736602233,0.1276523950419761,0.5145652675920016],"argmax":"insufficient","top2_margin":0.15678293022597928,"near_tie":false,"group_logz":{"41":-1.0450248718261719,"334":-8.169977188110352,"42":-2.075638771057129,"378":-8.855682373046875,"43":-0.6816272735595703,"340":-7.591317176818848}},"graph":{"mac_fp16":{"p":[0.35842185908414925,0.12740665307934193,0.5141714878365088],"max_abs_dp":0.000639521718126923,"argmax_equal":true},"iphone_int8mlp":{"p":[0.36821249060250666,0.12587913423021901,0.5059083751672744],"max_abs_dp":0.01043015323648433,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.35778233736602233,0.1276523950419761,0.5145652675920016]],"input_tokens":82,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5145652675920016,"probabilities":{"prohibited":0.35778233736602233,"permitted":0.1276523950419761,"insufficient":0.5145652675920016}}},"usage":{"input_tokens":82,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3058}},{"id":"semif_3226202ba1ada2570e9f","source":"semif","publication":"publish_text","request":{"state":"The applicant lives in the valley but works outside it.","questions":{"answer":{"type":"choice","instructions":"Under the alternative rule, a pass requires working in the valley; residence alone is insufficient. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 45 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":45,"line_sha256":"8448947ddecb0c2c87a7fd50a811d70585cbdc6df53e026bccd046fde96c41a2","licence":"MIT","semif_id":"3226202ba1ada2570e9f","group_id":"3200eb252d6737e03027","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":85,"slot":84,"row_ids_sha256":"f2b5336d9dd1f2531b307efc5d1c223ad78752f59f9ab6ceeca771bb8e79541a","oracle":{"probs":[0.536255318150447,0.16171446769410755,0.3020302141554455],"argmax":"prohibited","top2_margin":0.23422510399500152,"near_tie":false,"group_logz":{"41":-0.6399345397949219,"334":-7.609220504760742,"42":-1.8387126922607422,"378":-8.431269645690918,"43":-1.2140178680419922,"340":-7.817389488220215}},"graph":{"mac_fp16":{"p":[0.536421784926457,0.16180355722802803,0.3017746578455151],"max_abs_dp":0.0002555563099304048,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5380512957578316,0.1608776464676023,0.3010710577745661],"max_abs_dp":0.0017959776073845912,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.536255318150447,0.16171446769410755,0.3020302141554455]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.536255318150447,"probabilities":{"prohibited":0.536255318150447,"permitted":0.16171446769410755,"insufficient":0.3020302141554455}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3124}},{"id":"semif_3f783eec5c90812479bc","source":"semif","publication":"publish_text","request":{"state":"The applicant lives outside the valley and works inside it.","questions":{"answer":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 46 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":46,"line_sha256":"2526d42b7f34c2f6e44909233ca4699ed4610ebaed96f20088d39efa29b7b4fd","licence":"MIT","semif_id":"3f783eec5c90812479bc","group_id":"3200eb252d6737e03027","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":82,"slot":81,"row_ids_sha256":"720b4a77dd8d4a37b858c3dd81a604010f423d8b5b78c086687a1b22cb85585f","oracle":{"probs":[0.04918735527571979,0.7575993129714638,0.1932133317528165],"argmax":"prohibited","top2_margin":0.5643859812186472,"near_tie":false,"group_logz":{"41":-3.0262107849121094,"334":-10.090672492980957,"42":-0.29169273376464844,"378":-7.2320709228515625,"43":-1.6580524444580078,"340":-8.355112075805664}},"graph":{"mac_fp16":{"p":[0.049198331278722016,0.7574456536993446,0.19335601502193348],"max_abs_dp":0.00015365927211918873,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04855018020035195,0.7650614540416181,0.18638836575803006],"max_abs_dp":0.0074621410701543045,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.04918735527571979,0.7575993129714638,0.1932133317528165]],"input_tokens":82,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.7575993129714638,"probabilities":{"permitted":0.04918735527571979,"prohibited":0.7575993129714638,"insufficient":0.1932133317528165}}},"usage":{"input_tokens":82,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.321}},{"id":"semif_ffee861474a9665a093a","source":"semif","publication":"publish_text","request":{"state":"The applicant supplies a name but no home or work location.","questions":{"answer":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 47 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":47,"line_sha256":"61fec49012f547f2567bf5ef3d5608aa3fe75bfd7783e64ce8b402788b44c9f3","licence":"MIT","semif_id":"ffee861474a9665a093a","group_id":"3200eb252d6737e03027","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":83,"slot":82,"row_ids_sha256":"4fd4a333e81b57b4c4e552aeec1820661267e7895f2736df9df02352dbb74ae1","oracle":{"probs":[0.4855174958826581,0.09201828331068561,0.4224642208066564],"argmax":"prohibited","top2_margin":0.06305327507600167,"near_tie":false,"group_logz":{"41":-0.7430381774902344,"334":-7.931419372558594,"42":-2.406266212463379,"378":-9.248885154724121,"43":-0.8821487426757812,"340":-7.799282073974609}},"graph":{"mac_fp16":{"p":[0.4847263824808066,0.09229983286213984,0.42297378465705354],"max_abs_dp":0.0007911134018515065,"argmax_equal":true},"iphone_int8mlp":{"p":[0.48856747949365603,0.0931446950334696,0.4182878254728745],"max_abs_dp":0.00417639533378189,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.4855174958826581,0.09201828331068561,0.4224642208066564]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.4855174958826581,"probabilities":{"prohibited":0.4855174958826581,"permitted":0.09201828331068561,"insufficient":0.4224642208066564}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3234}},{"id":"semif_8c992e17e4fb54b044f5","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 48 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":48,"line_sha256":"89cf2043f353e2a8425b92e116152968bca6c14d724124bfe12e0d632d23c409","licence":"MIT","semif_id":"8c992e17e4fb54b044f5","group_id":"c0cbeb97315307af8111","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":75,"slot":74,"row_ids_sha256":"815f6ce258310324b5d0df6d11c482b2306b36c1bac43987ed9d6acaa13e78c7","oracle":{"probs":[0.012374051895505675,0.9691825447823647,0.018443403322129653],"argmax":"A","top2_margin":0.950739141460235,"near_tie":false,"group_logz":{"41":-4.399213790893555,"334":-12.551030158996582,"42":-0.03836250305175781,"378":-8.05233383178711,"43":-4.00010871887207,"340":-11.701364517211914}},"graph":{"mac_fp16":{"p":[0.012372895687156665,0.9691753920073765,0.018451712305466945],"max_abs_dp":8.308983337292075e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012455081154695295,0.969096569924361,0.018448348920943724],"max_abs_dp":8.597485800365678e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.012374051895505675,0.9691825447823647,0.018443403322129653]],"input_tokens":75,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9691825447823647,"probabilities":{"B":0.012374051895505675,"A":0.9691825447823647,"insufficient":0.018443403322129653}}},"usage":{"input_tokens":75,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.319}},{"id":"semif_695dd61195ca025c4f4c","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate offering a future delivery.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 49 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":49,"line_sha256":"9346a5d170f0cf1d535dc73ba483ad49c2eace8e9ec58084f20dbd7411757ffe","licence":"MIT","semif_id":"695dd61195ca025c4f4c","group_id":"c0cbeb97315307af8111","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","insufficient","A"],"codes":["A","B","C"],"gold":"B","row_len":76,"slot":75,"row_ids_sha256":"cdc6ab09c0f7535cec1576e690f4401a535776e9e9995048d1f2447f1b372a60","oracle":{"probs":[0.9461416028975701,0.04475464292202054,0.009103754180409375],"argmax":"B","top2_margin":0.9013869599755495,"near_tie":false,"group_logz":{"41":-0.06073570251464844,"334":-8.361231803894043,"42":-3.1119327545166016,"378":-10.763705253601074,"43":-4.704441070556641,"340":-12.725615501403809}},"graph":{"mac_fp16":{"p":[0.9462622226420212,0.04466109081650877,0.009076686541469993],"max_abs_dp":0.00012061974445110035,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9458685938333279,0.04497408250555635,0.009157323661115845],"max_abs_dp":0.0002730090642422134,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9461416028975701,0.04475464292202054,0.009103754180409375]],"input_tokens":76,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.9461416028975701,"probabilities":{"B":0.9461416028975701,"insufficient":0.04475464292202054,"A":0.009103754180409375}}},"usage":{"input_tokens":76,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3092}},{"id":"semif_e6155431ab7b0f8170d4","source":"semif","publication":"publish_text","request":{"state":"Candidate A: I can deliver a wreath tomorrow. Candidate B: The wreath was delivered.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 50 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":50,"line_sha256":"038c2b872fd328c23e1db158bcd0903e5a399e96b1775c5feecf8a9d4f31c33c","licence":"MIT","semif_id":"e6155431ab7b0f8170d4","group_id":"c0cbeb97315307af8111","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":67,"slot":66,"row_ids_sha256":"3dca67dbb7098a3c74d1517adca7089d5d3b88f195bae76fc236b94f8ec955f2","oracle":{"probs":[0.25677684871387574,0.5624119083169443,0.1808112429691799],"argmax":"B","top2_margin":0.3056350596030686,"near_tie":false,"group_logz":{"41":-1.3763751983642578,"334":-8.85008430480957,"42":-0.5923480987548828,"378":-7.857341766357422,"43":-1.7271289825439453,"340":-8.942142486572266}},"graph":{"mac_fp16":{"p":[0.25700942078991557,0.5620324733303775,0.18095810587970698],"max_abs_dp":0.00037943498656678454,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2558640217155707,0.5628851994143266,0.18125077887010269],"max_abs_dp":0.0009128269983050274,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.25677684871387574,0.5624119083169443,0.1808112429691799]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.5624119083169443,"probabilities":{"insufficient":0.25677684871387574,"B":0.5624119083169443,"A":0.1808112429691799}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3141}},{"id":"semif_b49ecda75086d8c68d2a","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The florist gives an address. Candidate B: The wreath contains lilies.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 51 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":51,"line_sha256":"ecf68f5df455aa43da7818fe2e7ad713f439c6eecdf7b799f02dfb4df301bbaf","licence":"MIT","semif_id":"b49ecda75086d8c68d2a","group_id":"c0cbeb97315307af8111","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":67,"slot":66,"row_ids_sha256":"b3d72a5f9d05bb8d1e0eb899be79ce250543ff50de94b3a3e41ca27b593aa1a1","oracle":{"probs":[0.0703543488323607,0.22836128890646912,0.7012843622611702],"argmax":"insufficient","top2_margin":0.4729230733547011,"near_tie":false,"group_logz":{"41":-2.6715097427368164,"334":-10.24275016784668,"42":-1.4941253662109375,"378":-8.637580871582031,"43":-0.37214088439941406,"340":-7.7266130447387695}},"graph":{"mac_fp16":{"p":[0.0702605699537122,0.22765504869830525,0.7020843813479826],"max_abs_dp":0.0008000190868123758,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0703833994195136,0.2281025840251839,0.7015140165553025],"max_abs_dp":0.00025870488128520996,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.0703543488323607,0.22836128890646912,0.7012843622611702]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.7012843622611702,"probabilities":{"B":0.0703543488323607,"A":0.22836128890646912,"insufficient":0.7012843622611702}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3132}},{"id":"semif_587ced261c6092cd1325","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 52 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":52,"line_sha256":"624cd699d50eecad754dd8457fcf13a651ee61cbdf61bf4ae96722ee00d40248","licence":"MIT","semif_id":"587ced261c6092cd1325","group_id":"428176cd8ad2c7a74eea","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":84,"slot":83,"row_ids_sha256":"f5e245131bb2f6c70096124b0c6b2fc609586512b06b8bb0af92b59aef5cc58b","oracle":{"probs":[0.13157705046921636,0.8327274264109636,0.035695523119820126],"argmax":"B","top2_margin":0.7011503759417472,"near_tie":false,"group_logz":{"41":-2.0371570587158203,"334":-9.73331356048584,"42":-0.19204330444335938,"378":-7.712823867797852,"43":-3.341724395751953,"340":-10.891783714294434}},"graph":{"mac_fp16":{"p":[0.13174482670274731,0.8325001198390057,0.03575505345824683],"max_abs_dp":0.00022730657195790815,"argmax_equal":true},"iphone_int8mlp":{"p":[0.12532066842713568,0.8399480756149412,0.03473125595792315],"max_abs_dp":0.0072206492039775805,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.13157705046921636,0.8327274264109636,0.035695523119820126]],"input_tokens":84,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8327274264109636,"probabilities":{"insufficient":0.13157705046921636,"B":0.8327274264109636,"A":0.035695523119820126}}},"usage":{"input_tokens":84,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3398}},{"id":"semif_533d4423d311de82b27d","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate asking only for information while withholding authorization.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 53 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":53,"line_sha256":"284139b9925f90df569c901d8f500e2cd7eccef7ed8960e094ee1f4457ffadbd","licence":"MIT","semif_id":"533d4423d311de82b27d","group_id":"428176cd8ad2c7a74eea","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":89,"slot":88,"row_ids_sha256":"b492744a55d696ebd4c9818f2507b6e770c66c6717573950fd85613429f40503","oracle":{"probs":[0.02372646342596839,0.9675625133310349,0.008711023242996762],"argmax":"A","top2_margin":0.9438360499050665,"near_tie":false,"group_logz":{"41":-3.747415542602539,"334":-11.95398998260498,"42":-0.039226531982421875,"378":-8.15483570098877,"43":-4.749417304992676,"340":-12.727203369140625}},"graph":{"mac_fp16":{"p":[0.0236400998756841,0.9676577429719666,0.008702157152349397],"max_abs_dp":9.522964093167108e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.023707434745758555,0.9675629434645502,0.008729621789691208],"max_abs_dp":1.9028680209833698e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.02372646342596839,0.9675625133310349,0.008711023242996762]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9675625133310349,"probabilities":{"insufficient":0.02372646342596839,"A":0.9675625133310349,"B":0.008711023242996762}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3334}},{"id":"semif_2bf540d515a51755b25c","source":"semif","publication":"publish_text","request":{"state":"Candidate A: Please engrave the ring. Candidate B: Show me how it would look, but do not engrave it yet.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 54 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":54,"line_sha256":"14c169b4741e827edb44afefd9bc380cbffb1a0c9fd006aad047633da4057d31","licence":"MIT","semif_id":"2bf540d515a51755b25c","group_id":"428176cd8ad2c7a74eea","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"A","row_len":75,"slot":74,"row_ids_sha256":"ba083046fe7874a6d08cbd22a818fa2bd3d46d19e9789796c252f19a0868c506","oracle":{"probs":[0.18312121502729023,0.0665199322381184,0.7503588527345915],"argmax":"A","top2_margin":0.5672376377073012,"near_tie":false,"group_logz":{"41":-1.7135400772094727,"334":-8.755772590637207,"42":-2.726186752319336,"378":-9.456400871276855,"43":-0.30313682556152344,"340":-7.31011962890625}},"graph":{"mac_fp16":{"p":[0.18355226669852656,0.06684266460436256,0.749605068697111],"max_abs_dp":0.0007537840374804983,"argmax_equal":true},"iphone_int8mlp":{"p":[0.18054940702192795,0.06491450355594379,0.7545360894221282],"max_abs_dp":0.004177236687536778,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.18312121502729023,0.0665199322381184,0.7503588527345915]],"input_tokens":75,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.7503588527345915,"probabilities":{"insufficient":0.18312121502729023,"B":0.0665199322381184,"A":0.7503588527345915}}},"usage":{"input_tokens":75,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3638}},{"id":"semif_fdb9fc214aea2710763a","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The customer gives a name. Candidate B: The ring is silver.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 55 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":55,"line_sha256":"333f1885b1fcad000c7909789d7a5218ebad42cfe0f8a92b97a094aa29d294dd","licence":"MIT","semif_id":"fdb9fc214aea2710763a","group_id":"428176cd8ad2c7a74eea","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":65,"slot":64,"row_ids_sha256":"6180c99eb252cc72e324c2995156d35db7f6516c355e05056c94d03520c4b384","oracle":{"probs":[0.10502221464075659,0.25812645044583465,0.6368513349134087],"argmax":"insufficient","top2_margin":0.3787248844675741,"near_tie":false,"group_logz":{"41":-2.275186538696289,"334":-9.572162628173828,"42":-1.3759088516235352,"378":-8.34677505493164,"43":-0.4728221893310547,"340":-7.562140464782715}},"graph":{"mac_fp16":{"p":[0.10511431636394453,0.25855894325665135,0.6363267403794042],"max_abs_dp":0.0005245945340045521,"argmax_equal":true},"iphone_int8mlp":{"p":[0.10622223659367531,0.25666628882026155,0.6371114745860631],"max_abs_dp":0.0014601616255731087,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.10502221464075659,0.25812645044583465,0.6368513349134087]],"input_tokens":65,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6368513349134087,"probabilities":{"B":0.10502221464075659,"A":0.25812645044583465,"insufficient":0.6368513349134087}}},"usage":{"input_tokens":65,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3558}},{"id":"semif_f6d68126b43f8c9ed183","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 56 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":56,"line_sha256":"c1f17ff7f97d4730b49ea049d68795296e425330b4cc28714606ab0259987312","licence":"MIT","semif_id":"f6d68126b43f8c9ed183","group_id":"389e65bf1ba709b84b12","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":78,"slot":77,"row_ids_sha256":"e84508016d3ef7389edd0cd7c1cd94854735fc51f6ba687d5f879e6f02fd80df","oracle":{"probs":[0.02687944833321064,0.9608021139951738,0.012318437671615499],"argmax":"A","top2_margin":0.9339226656619631,"near_tie":false,"group_logz":{"41":-3.6224231719970703,"334":-11.870092391967773,"42":-0.046016693115234375,"378":-8.26144027709961,"43":-4.402688026428223,"340":-12.468117713928223}},"graph":{"mac_fp16":{"p":[0.026836099721224003,0.9608515044282198,0.012312395850556266],"max_abs_dp":4.939043304597046e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.026810677423715068,0.9608290043568014,0.012360318219483484],"max_abs_dp":6.877090949557185e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.02687944833321064,0.9608021139951738,0.012318437671615499]],"input_tokens":78,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9608021139951738,"probabilities":{"insufficient":0.02687944833321064,"A":0.9608021139951738,"B":0.012318437671615499}}},"usage":{"input_tokens":78,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3798}},{"id":"semif_44bae0c8a1615a096c43","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate containing only an unexecuted test procedure.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 57 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":57,"line_sha256":"73ecb16d7d1766027826cdef506368c3a0fdebacc5a24208389586b776a22252","licence":"MIT","semif_id":"44bae0c8a1615a096c43","group_id":"389e65bf1ba709b84b12","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"B","row_len":80,"slot":79,"row_ids_sha256":"e20a9609e9345c1f647131ce86b0ca8fc57f54351a266a7e36016437e398accf","oracle":{"probs":[0.895268335938415,0.02811867497209637,0.07661298908948846],"argmax":"B","top2_margin":0.8186553468489266,"near_tie":false,"group_logz":{"41":-0.11712837219238281,"334":-8.310896873474121,"42":-3.577817916870117,"378":-11.40929889678955,"43":-2.5754852294921875,"340":-10.322588920593262}},"graph":{"mac_fp16":{"p":[0.8954638052457854,0.02804410089416873,0.07649209386004589],"max_abs_dp":0.00019546930737035506,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8942240696032184,0.028074507067078147,0.07770142332970338],"max_abs_dp":0.0010884342402149166,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.895268335938415,0.02811867497209637,0.07661298908948846]],"input_tokens":80,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.895268335938415,"probabilities":{"B":0.895268335938415,"A":0.02811867497209637,"insufficient":0.07661298908948846}}},"usage":{"input_tokens":80,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3714}},{"id":"semif_2175379f4c69e6207626","source":"semif","publication":"publish_text","request":{"state":"Candidate A: Here are insulation-test steps; no test was run. Candidate B: The completed insulation test passed.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 58 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":58,"line_sha256":"a66d7270af284da26d40590eeeddf810d1611af432d556a9645076a9f6013841","licence":"MIT","semif_id":"2175379f4c69e6207626","group_id":"389e65bf1ba709b84b12","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":72,"slot":71,"row_ids_sha256":"68de47a6762de7f18c3baa1fb2a26583772fbea66952edd2379dadbebd7c3903","oracle":{"probs":[0.5959216391187232,0.28709578522489054,0.11698257565638631],"argmax":"insufficient","top2_margin":0.30882585389383266,"near_tie":false,"group_logz":{"41":-0.5263614654541016,"334":-8.43985652923584,"42":-1.2566547393798828,"378":-8.6923828125,"43":-2.1544456481933594,"340":-9.729626655578613}},"graph":{"mac_fp16":{"p":[0.5954457271533037,0.28825188289866915,0.11630238994802714],"max_abs_dp":0.001156097673778611,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5908041797592872,0.28990336098208785,0.11929245925862499],"max_abs_dp":0.005117459359435972,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5959216391187232,0.28709578522489054,0.11698257565638631]],"input_tokens":72,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5959216391187232,"probabilities":{"insufficient":0.5959216391187232,"B":0.28709578522489054,"A":0.11698257565638631}}},"usage":{"input_tokens":72,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.322}},{"id":"semif_def9fe59d262357a673a","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The cable is red. Candidate B: The tester is in a drawer.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 59 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":59,"line_sha256":"6b011ff841294db3944fc693427f048aea9605aa0d0c050bb2d41c0fe518a597","licence":"MIT","semif_id":"def9fe59d262357a673a","group_id":"389e65bf1ba709b84b12","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"insufficient","row_len":68,"slot":67,"row_ids_sha256":"67801cfb782c0717647ac5460dd902df01f4ede329eb9f1495c70d9ae76df1fa","oracle":{"probs":[0.4211092930764633,0.44792492907641585,0.13096577784712085],"argmax":"A","top2_margin":0.02681563599995257,"near_tie":false,"group_logz":{"41":-0.8774890899658203,"334":-8.308467864990234,"42":-0.8157558441162109,"378":-8.037887573242188,"43":-2.045445442199707,"340":-9.142110824584961}},"graph":{"mac_fp16":{"p":[0.42002200046409005,0.4491874634477848,0.13079053608812508],"max_abs_dp":0.001262534371368973,"argmax_equal":true},"iphone_int8mlp":{"p":[0.42133201540700665,0.447414307334959,0.13125367725803433],"max_abs_dp":0.0005106217414568293,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.4211092930764633,0.44792492907641585,0.13096577784712085]],"input_tokens":68,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.44792492907641585,"probabilities":{"insufficient":0.4211092930764633,"A":0.44792492907641585,"B":0.13096577784712085}}},"usage":{"input_tokens":68,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3969}},{"id":"semif_712f6ffb763eef7c9a12","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 60 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":60,"line_sha256":"b1b66ff0d760812af5cf48a4e5966e5c231a54ef441925fc9a49d1f01b0d720f","licence":"MIT","semif_id":"712f6ffb763eef7c9a12","group_id":"45f8ead2522a14384cdc","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"B","row_len":78,"slot":77,"row_ids_sha256":"154ad2964ab2d90b8c299d373ed6224d345713c45349d71510183b749e38e1cd","oracle":{"probs":[0.19447473282189937,0.15432798239942214,0.6511972847786786],"argmax":"insufficient","top2_margin":0.4567225519567792,"near_tie":false,"group_logz":{"41":-1.652491569519043,"334":-9.041769027709961,"42":-1.883713722229004,"378":-8.89842414855957,"43":-0.4439811706542969,"340":-7.780498504638672}},"graph":{"mac_fp16":{"p":[0.1941702737152666,0.15431946116665807,0.6515102651180753],"max_abs_dp":0.0003129803393967512,"argmax_equal":true},"iphone_int8mlp":{"p":[0.19361043059200228,0.15399312636551463,0.652396443042483],"max_abs_dp":0.0011991582638044562,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.19447473282189937,0.15432798239942214,0.6511972847786786]],"input_tokens":78,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6511972847786786,"probabilities":{"B":0.19447473282189937,"A":0.15432798239942214,"insufficient":0.6511972847786786}}},"usage":{"input_tokens":78,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4129}},{"id":"semif_e1d610bd14e3d16c09b9","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate opposing a paywall.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 61 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":61,"line_sha256":"0b5013b84e46f56ca742a2903fd935c164d33c469dabc04dee5dff2316e7f530","licence":"MIT","semif_id":"e1d610bd14e3d16c09b9","group_id":"45f8ead2522a14384cdc","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","insufficient","B"],"codes":["A","B","C"],"gold":"A","row_len":78,"slot":77,"row_ids_sha256":"cc4deb79e3ca35b0d50374bf2245748c4c2ce24b96dec67b245924f7d278d90c","oracle":{"probs":[0.8580291853820817,0.11630160751802525,0.025669207099893083],"argmax":"A","top2_margin":0.7417275778640565,"near_tie":false,"group_logz":{"41":-0.16324806213378906,"334":-7.871221542358398,"42":-2.1616992950439453,"378":-9.415143966674805,"43":-3.6725940704345703,"340":-10.994503021240234}},"graph":{"mac_fp16":{"p":[0.857529322868921,0.116802495066494,0.025668182064584984],"max_abs_dp":0.0005008875484687481,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8556291261009532,0.11827257847206031,0.026098295426986363],"max_abs_dp":0.0024000592811285193,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8580291853820817,0.11630160751802525,0.025669207099893083]],"input_tokens":78,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.8580291853820817,"probabilities":{"A":0.8580291853820817,"insufficient":0.11630160751802525,"B":0.025669207099893083}}},"usage":{"input_tokens":78,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3937}},{"id":"semif_22ec57822db50ec121cf","source":"semif","publication":"publish_text","request":{"state":"Candidate A: I endorse introducing a paywall. Candidate B: I reject introducing a paywall.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 62 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":62,"line_sha256":"e68cc16bb90711db2a49e6cf906b29d3e9034a46087857aa800061034a5c1f75","licence":"MIT","semif_id":"22ec57822db50ec121cf","group_id":"45f8ead2522a14384cdc","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":68,"slot":67,"row_ids_sha256":"a695494816eb940ff39f4eb4d32ae3ae354c32048f6597b89340bb452dbfbb4f","oracle":{"probs":[0.022524972274203737,0.9676217778435989,0.00985324988219736],"argmax":"A","top2_margin":0.9450968055693952,"near_tie":false,"group_logz":{"41":-3.8012075424194336,"334":-11.93655014038086,"42":-0.04099082946777344,"378":-8.183320999145508,"43":-4.628030776977539,"340":-12.595096588134766}},"graph":{"mac_fp16":{"p":[0.022465394591519035,0.9676992942927827,0.009835311115698268],"max_abs_dp":7.751644918374545e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02249645730015372,0.9676610444864154,0.009842498213430776],"max_abs_dp":3.9266642816504493e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.022524972274203737,0.9676217778435989,0.00985324988219736]],"input_tokens":68,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9676217778435989,"probabilities":{"insufficient":0.022524972274203737,"A":0.9676217778435989,"B":0.00985324988219736}}},"usage":{"input_tokens":68,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3928}},{"id":"semif_3e75f2b84e8e08e3d3f2","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The editor names the magazine. Candidate B: The website has a blue header.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 63 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":63,"line_sha256":"c902eff9d1fd2f98073510179439fee5c04d0f1d6d4eb995792ad6bd1f68cd22","licence":"MIT","semif_id":"3e75f2b84e8e08e3d3f2","group_id":"45f8ead2522a14384cdc","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":67,"slot":66,"row_ids_sha256":"2aecca3389d9d23ef82b409e40a132fe369bc23c1d008dbfd51b6af7d34b8c2e","oracle":{"probs":[0.09711749308497455,0.06762284329723603,0.8352596636177894],"argmax":"insufficient","top2_margin":0.7381421705328148,"near_tie":false,"group_logz":{"41":-2.3471622467041016,"334":-9.889741897583008,"42":-2.7091379165649414,"378":-9.823394775390625,"43":-0.1953411102294922,"340":-7.6715240478515625}},"graph":{"mac_fp16":{"p":[0.09715686415938468,0.06765379686624787,0.8351893389743675],"max_abs_dp":7.032464342182632e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.09668049134089175,0.06708367979150935,0.8362358288675988],"max_abs_dp":0.0009761652498094797,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.09711749308497455,0.06762284329723603,0.8352596636177894]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8352596636177894,"probabilities":{"B":0.09711749308497455,"A":0.06762284329723603,"insufficient":0.8352596636177894}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3951}},{"id":"semif_ebd74aa05324a1556288","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 64 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":64,"line_sha256":"29373afd1f31e46bd9131be7bff97948583ae454e2800e284d92c7e095445ae7","licence":"MIT","semif_id":"ebd74aa05324a1556288","group_id":"3ecec8e2c166e397ad07","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":83,"slot":82,"row_ids_sha256":"3fb02af3c7bd21d4aebf88f3bc2c2986d287ccc3dbc7ee7b4758090d195fe6be","oracle":{"probs":[0.03161491631037675,0.9575687818154073,0.010816301874215879],"argmax":"A","top2_margin":0.9259538655050306,"near_tie":false,"group_logz":{"41":-3.4620361328125,"334":-11.448640823364258,"42":-0.05126762390136719,"378":-8.014652252197266,"43":-4.534610748291016,"340":-12.423125267028809}},"graph":{"mac_fp16":{"p":[0.031615134239200184,0.9575750897066786,0.010809776054121185],"max_abs_dp":6.525820094693824e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.031521491788831284,0.9576147326417164,0.010863775569452309],"max_abs_dp":9.342452154546821e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03161491631037675,0.9575687818154073,0.010816301874215879]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9575687818154073,"probabilities":{"insufficient":0.03161491631037675,"A":0.9575687818154073,"B":0.010816301874215879}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4227}},{"id":"semif_d7e84e4245a29817da1f","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate asking for complete replacement.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 65 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":65,"line_sha256":"e1333a822e60c3e20bbbaba0a8a13128ad420437e533f2ee6560ae397c9af6af","licence":"MIT","semif_id":"d7e84e4245a29817da1f","group_id":"3ecec8e2c166e397ad07","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"B","row_len":81,"slot":80,"row_ids_sha256":"3289cb56d36ac1edd83cb14cc9864239013c9f6f1bcc42c8780cd8199361135b","oracle":{"probs":[0.04042053226994511,0.011343209759730076,0.9482362579703247],"argmax":"B","top2_margin":0.9078157257003796,"near_tie":false,"group_logz":{"41":-3.2138595581054688,"334":-10.980572700500488,"42":-4.4845781326293945,"378":-11.957805633544922,"43":-0.05859375,"340":-8.004362106323242}},"graph":{"mac_fp16":{"p":[0.040634424387509076,0.01136700807160691,0.947998567540884],"max_abs_dp":0.00023769042944077334,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03991878549474366,0.011211154253838795,0.9488700602514176],"max_abs_dp":0.0006338022810928878,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.04042053226994511,0.011343209759730076,0.9482362579703247]],"input_tokens":81,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.9482362579703247,"probabilities":{"insufficient":0.04042053226994511,"A":0.011343209759730076,"B":0.9482362579703247}}},"usage":{"input_tokens":81,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.407}},{"id":"semif_f41ae66b3c09630c7a42","source":"semif","publication":"publish_text","request":{"state":"Candidate A: Remove the window and install a replacement. Candidate B: Reseal the existing window and keep its frame.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 66 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":66,"line_sha256":"12e922b40631c0eabc53fd463a9a122a76b793eb1aff308ff547b4ddd2597061","licence":"MIT","semif_id":"f41ae66b3c09630c7a42","group_id":"3ecec8e2c166e397ad07","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"B","row_len":74,"slot":73,"row_ids_sha256":"caca156c03dfc468818260510a625aff57ac9b61e09d50159ffa0f034aec4505","oracle":{"probs":[0.059997627945172806,0.03125250584380487,0.9087498662110224],"argmax":"B","top2_margin":0.8487522382658496,"near_tie":false,"group_logz":{"41":-2.8218774795532227,"334":-10.899801254272461,"42":-3.4740829467773438,"378":-11.128374099731445,"43":-0.10411262512207031,"340":-8.187650680541992}},"graph":{"mac_fp16":{"p":[0.059742418522600545,0.031154811147407174,0.9091027703299923],"max_abs_dp":0.00035290411896993223,"argmax_equal":true},"iphone_int8mlp":{"p":[0.060180870558292195,0.030915526665040733,0.9089036027766672],"max_abs_dp":0.00033697917876414005,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.059997627945172806,0.03125250584380487,0.9087498662110224]],"input_tokens":74,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.9087498662110224,"probabilities":{"insufficient":0.059997627945172806,"A":0.03125250584380487,"B":0.9087498662110224}}},"usage":{"input_tokens":74,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4128}},{"id":"semif_82c2cd97c160ad27951b","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The window faces a courtyard. Candidate B: The owner gives a postcode.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 67 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":67,"line_sha256":"5e827dfcaef587f3b69c77190c027364342d2447960deba7094c78ad6f374f90","licence":"MIT","semif_id":"82c2cd97c160ad27951b","group_id":"3ecec8e2c166e397ad07","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"insufficient","row_len":69,"slot":68,"row_ids_sha256":"2304cf619f34593408793f723e78a31c1f6b03f184591a1c85bc51f5cd56f833","oracle":{"probs":[0.6800425073639721,0.14839058882644535,0.17156690380958262],"argmax":"insufficient","top2_margin":0.5084756035543895,"near_tie":false,"group_logz":{"41":-0.3999500274658203,"334":-8.000347137451172,"42":-1.922257423400879,"378":-9.057952880859375,"43":-1.7771320343017578,"340":-8.983474731445312}},"graph":{"mac_fp16":{"p":[0.6802166236076296,0.14799593230813707,0.17178744408423333],"max_abs_dp":0.0003946565183082784,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6794374000683251,0.14679553617171298,0.17376706375996193],"max_abs_dp":0.00220015995037931,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.6800425073639721,0.14839058882644535,0.17156690380958262]],"input_tokens":69,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6800425073639721,"probabilities":{"insufficient":0.6800425073639721,"B":0.14839058882644535,"A":0.17156690380958262}}},"usage":{"input_tokens":69,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4183}},{"id":"semif_7ada97b6f99f4788e5c7","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 68 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":68,"line_sha256":"062d86d202a44054d2ca7fd9d5df1cba61b83dd6e66b9674f6759072ba0d3574","licence":"MIT","semif_id":"7ada97b6f99f4788e5c7","group_id":"a71a17a287f3d30db2aa","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"B","row_len":81,"slot":80,"row_ids_sha256":"a89a7d3063a8c5a1873665f4c5ec3c85d65758265a4c55be20cde319879aa970","oracle":{"probs":[0.8086398582661839,0.049413510214385566,0.14194663151943057],"argmax":"B","top2_margin":0.6666932267467534,"near_tie":false,"group_logz":{"41":-0.21779823303222656,"334":-8.748779296875,"42":-3.012928009033203,"378":-11.061542510986328,"43":-1.9577007293701172,"340":-10.226492881774902}},"graph":{"mac_fp16":{"p":[0.8087129513294926,0.04927220841222164,0.14201484025828567],"max_abs_dp":0.00014130180216392896,"argmax_equal":true},"iphone_int8mlp":{"p":[0.810685152351461,0.049553866634577906,0.1397609810139612],"max_abs_dp":0.002185650505469372,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8086398582661839,0.049413510214385566,0.14194663151943057]],"input_tokens":81,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8086398582661839,"probabilities":{"B":0.8086398582661839,"A":0.049413510214385566,"insufficient":0.14194663151943057}}},"usage":{"input_tokens":81,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4341}},{"id":"semif_4d7a003342db5b0a17e3","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting receipt with the award still undecided.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 69 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":69,"line_sha256":"3126261538009017470f7ac26ee182a3da3f78109e70f36c838825289a31b8d8","licence":"MIT","semif_id":"4d7a003342db5b0a17e3","group_id":"a71a17a287f3d30db2aa","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":85,"slot":84,"row_ids_sha256":"e2728b2e00bddedbb6fadb0134ca25d950f8883a9b551da0109573e840001e7b","oracle":{"probs":[0.017279844150232367,0.9721044228473125,0.010615733002455207],"argmax":"A","top2_margin":0.9548245786970802,"near_tie":false,"group_logz":{"41":-4.064971923828125,"334":-12.61410140991211,"42":-0.0350494384765625,"378":-8.457202911376953,"43":-4.552175521850586,"340":-12.846561431884766}},"graph":{"mac_fp16":{"p":[0.017279392456111608,0.9721035989489094,0.010617008594978909],"max_abs_dp":1.2755925237022125e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.017276740099180437,0.9721166458846209,0.010606614016198685],"max_abs_dp":1.222303730841201e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.017279844150232367,0.9721044228473125,0.010615733002455207]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9721044228473125,"probabilities":{"insufficient":0.017279844150232367,"A":0.9721044228473125,"B":0.010615733002455207}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4189}},{"id":"semif_19d3d8d0bf773e88be7b","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The grant was awarded. Candidate B: The application was received; the award remains undecided.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 70 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":70,"line_sha256":"f4f1112105587e7d1c76b01cdef209a4f8368a56ad9b98e6e40634bd17be02c2","licence":"MIT","semif_id":"19d3d8d0bf773e88be7b","group_id":"a71a17a287f3d30db2aa","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","B","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":70,"slot":69,"row_ids_sha256":"a7eb415877a675df9ef38b805c59676a927e44e06a46961ab164bda1927715b4","oracle":{"probs":[0.9746528699282783,0.009726332374244542,0.015620797697477167],"argmax":"A","top2_margin":0.9590320722308011,"near_tie":false,"group_logz":{"41":-0.03139495849609375,"334":-8.947011947631836,"42":-4.638639450073242,"378":-13.000232696533203,"43":-4.164873123168945,"340":-12.512444496154785}},"graph":{"mac_fp16":{"p":[0.974709491896997,0.009708000124052051,0.015582507978951084],"max_abs_dp":5.662196871869529e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9747685417825035,0.009673812393132315,0.015557645824364169],"max_abs_dp":0.00011567185422522996,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9746528699282783,0.009726332374244542,0.015620797697477167]],"input_tokens":70,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9746528699282783,"probabilities":{"A":0.9746528699282783,"B":0.009726332374244542,"insufficient":0.015620797697477167}}},"usage":{"input_tokens":70,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3889}},{"id":"semif_d6de731cca52359b1c87","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The sponsor gives its name. Candidate B: The application has a reference number.","questions":{"answer":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 71 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":71,"line_sha256":"fe82e361e4b6f004e986e61119470b7af9b0c1e5e8371efa7611d7a17b87fb7f","licence":"MIT","semif_id":"d6de731cca52359b1c87","group_id":"a71a17a287f3d30db2aa","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"insufficient","row_len":67,"slot":66,"row_ids_sha256":"818346b40097a4c552acb95833339a1eb8d11e5fd891fdeaffcc24264ea8ff4f","oracle":{"probs":[0.4572126902237533,0.3091567675251679,0.2336305422510788],"argmax":"insufficient","top2_margin":0.14805592269858542,"near_tie":false,"group_logz":{"41":-0.8009414672851562,"334":-8.021728515625,"42":-1.1922416687011719,"378":-8.18297290802002,"43":-1.4723491668701172,"340":-8.359762191772461}},"graph":{"mac_fp16":{"p":[0.45711075980907134,0.3091389750160811,0.2337502651748476],"max_abs_dp":0.00011972292376880023,"argmax_equal":true},"iphone_int8mlp":{"p":[0.4585043202016542,0.30589992025249696,0.2355957595458489],"max_abs_dp":0.0032568472726709485,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.4572126902237533,0.3091567675251679,0.2336305422510788]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.4572126902237533,"probabilities":{"insufficient":0.4572126902237533,"A":0.3091567675251679,"B":0.2336305422510788}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3958}},{"id":"semif_b45c63fb92729af27f05","source":"semif","publication":"publish_text","request":{"state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 72 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":72,"line_sha256":"3a9217aa64137c66dd3201087f0a5caa53c2f155b5ee7d3af92a1768fa0f7477","licence":"MIT","semif_id":"b45c63fb92729af27f05","group_id":"986d7eaba0c7e9738e79","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","supported","insufficient"],"codes":["A","B","C"],"gold":"contradicted","row_len":89,"slot":88,"row_ids_sha256":"21bd8e6afda674c340156c9b08d41803781a07cbd85f4d238bcb2d40e406d4e6","oracle":{"probs":[0.9634552577353703,0.021418958285116366,0.015125783979513288],"argmax":"contradicted","top2_margin":0.9420362994502539,"near_tie":false,"group_logz":{"41":-0.042171478271484375,"334":-8.461179733276367,"42":-3.848421096801758,"378":-11.672673225402832,"43":-4.196296691894531,"340":-12.083277702331543}},"graph":{"mac_fp16":{"p":[0.9634540284224903,0.02142100547055134,0.015124966106958435],"max_abs_dp":2.04718543497448e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9635687756986752,0.021324482305522988,0.015106741995801828],"max_abs_dp":0.00011351796330483221,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9634552577353703,0.021418958285116366,0.015125783979513288]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9634552577353703,"probabilities":{"contradicted":0.9634552577353703,"supported":0.021418958285116366,"insufficient":0.015125783979513288}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.424}},{"id":"semif_46b7029b9a704138b77a","source":"semif","publication":"publish_text","request":{"state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the owner gave permission on Monday.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 73 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":73,"line_sha256":"619b2ef40bdcdac22a59d18d980f7175c8ad4284075897d1f6fc4d99387f1721","licence":"MIT","semif_id":"46b7029b9a704138b77a","group_id":"986d7eaba0c7e9738e79","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","contradicted","supported"],"codes":["A","B","C"],"gold":"supported","row_len":87,"slot":86,"row_ids_sha256":"b61fcf71e1c72e74629e7bce1c60163919771ef3a4345570f6a165ff4fbfa146","oracle":{"probs":[0.025317726092723817,0.01671174742598226,0.957970526481294],"argmax":"supported","top2_margin":0.9326528003885701,"near_tie":false,"group_logz":{"41":-3.6876792907714844,"334":-11.776144027709961,"42":-4.103072166442871,"378":-11.81431770324707,"43":-0.0543670654296875,"340":-8.106739044189453}},"graph":{"mac_fp16":{"p":[0.025265089783023192,0.016721743444472786,0.958013166772504],"max_abs_dp":5.263630970062452e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.025056308100309142,0.016511821584612223,0.9584318703150787],"max_abs_dp":0.0004613438337847109,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.025317726092723817,0.01671174742598226,0.957970526481294]],"input_tokens":87,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.957970526481294,"probabilities":{"insufficient":0.025317726092723817,"contradicted":0.01671174742598226,"supported":0.957970526481294}}},"usage":{"input_tokens":87,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4221}},{"id":"semif_caf773b23934fa7362a1","source":"semif","publication":"publish_text","request":{"state":"The owner authorized scanning the notebook on Monday. The complete permission record confirms that she has not withdrawn it and it remains valid.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 74 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":74,"line_sha256":"a8382e8657752bcf95d155ca6412344348fec7c4969f90d53b823b904eb2d14a","licence":"MIT","semif_id":"caf773b23934fa7362a1","group_id":"986d7eaba0c7e9738e79","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"supported","row_len":81,"slot":80,"row_ids_sha256":"99e12adcf10159bbf73e04f04252801c23657023149f3585ab1482f630de5375","oracle":{"probs":[0.9668168993846954,0.017841342203697028,0.015341758411607525],"argmax":"supported","top2_margin":0.9489755571809984,"near_tie":false,"group_logz":{"41":-0.04068565368652344,"334":-8.719072341918945,"42":-4.033176422119141,"378":-12.255001068115234,"43":-4.184116363525391,"340":-12.340970039367676}},"graph":{"mac_fp16":{"p":[0.9668676313481896,0.017806525853644756,0.01532584279816558],"max_abs_dp":5.0731963494254906e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9670477226193419,0.017667722838373385,0.015284554542284574],"max_abs_dp":0.00023082323464651555,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9668168993846954,0.017841342203697028,0.015341758411607525]],"input_tokens":81,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9668168993846954,"probabilities":{"supported":0.9668168993846954,"contradicted":0.017841342203697028,"insufficient":0.015341758411607525}}},"usage":{"input_tokens":81,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4322}},{"id":"semif_f46f392ef9e9e9df564b","source":"semif","publication":"publish_text","request":{"state":"The owner authorized scanning on Monday. The supplied record omits all later permission changes and does not establish current validity.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 75 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":75,"line_sha256":"02184e34650daf91d0424f9bbd739aab20ca996fdaef6777e3cae78204f2bc0d","licence":"MIT","semif_id":"f46f392ef9e9e9df564b","group_id":"986d7eaba0c7e9738e79","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"insufficient","row_len":79,"slot":78,"row_ids_sha256":"c3424b3a61807250955bdc6229f2fa02ce8927a3c43976b5a143ae08ee0755d1","oracle":{"probs":[0.016253714826364233,0.2529656381796184,0.7307806469940173],"argmax":"contradicted","top2_margin":0.4778150088143989,"near_tie":false,"group_logz":{"41":-4.129108428955078,"334":-12.0639009475708,"42":-1.384176254272461,"378":-8.758102416992188,"43":-0.3233165740966797,"340":-7.904387474060059}},"graph":{"mac_fp16":{"p":[0.01626296994645718,0.25409205623688014,0.7296449738166627],"max_abs_dp":0.0011356731773546347,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016146689843829297,0.2582889414543652,0.7255643687018055],"max_abs_dp":0.005323303274746782,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.016253714826364233,0.2529656381796184,0.7307806469940173]],"input_tokens":79,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.7307806469940173,"probabilities":{"supported":0.016253714826364233,"insufficient":0.2529656381796184,"contradicted":0.7307806469940173}}},"usage":{"input_tokens":79,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4042}},{"id":"semif_005dcf6d2c0f21d64796","source":"semif","publication":"publish_text","request":{"state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"answer":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 76 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":76,"line_sha256":"9440dc64a7f53830326cb2d7def27686e51501f48458adea5983b3761d658a95","licence":"MIT","semif_id":"005dcf6d2c0f21d64796","group_id":"ca2e0b043985fa49bcd2","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":104,"slot":103,"row_ids_sha256":"a98bc0445d94ed55192b1ce57d597639f34189e827c6ac1b36d6df41bfb9734a","oracle":{"probs":[0.10163346967233246,0.6402632759612489,0.25810325436641873],"argmax":"prohibited","top2_margin":0.38216002159483015,"near_tie":false,"group_logz":{"41":-2.3094587326049805,"334":-8.83523178100586,"42":-0.4689521789550781,"378":-6.896539688110352,"43":-1.377471923828125,"340":-7.565008163452148}},"graph":{"mac_fp16":{"p":[0.10151572781020499,0.6400054879247236,0.2584787842650715],"max_abs_dp":0.0003755298986527489,"argmax_equal":true},"iphone_int8mlp":{"p":[0.10157360983132298,0.6459490147010405,0.25247737546763654],"max_abs_dp":0.005685738739791613,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.10163346967233246,0.6402632759612489,0.25810325436641873]],"input_tokens":104,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.6402632759612489,"probabilities":{"permitted":0.10163346967233246,"prohibited":0.6402632759612489,"insufficient":0.25810325436641873}}},"usage":{"input_tokens":104,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.478}},{"id":"semif_edb708bd0fe5fd67d6ee","source":"semif","publication":"publish_text","request":{"state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"answer":{"type":"choice","instructions":"For this fictional protocol, the first recorded decision is binding and later withdrawals are ignored. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 77 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":77,"line_sha256":"6a2e4901524287679f7aad9cd65a4974a4ac68bbaa22a775ec76f8646d2fe69a","licence":"MIT","semif_id":"edb708bd0fe5fd67d6ee","group_id":"ca2e0b043985fa49bcd2","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":106,"slot":105,"row_ids_sha256":"00a1230738467f817fffa0c48ceba84730f0bea98b6b2ab6d698af49044a8bf5","oracle":{"probs":[0.13635662284499458,0.6555743973746637,0.20806897978034178],"argmax":"prohibited","top2_margin":0.44750541759432194,"near_tie":false,"group_logz":{"41":-2.012422561645508,"334":-8.787347793579102,"42":-0.4421844482421875,"378":-7.120844841003418,"43":-1.5898265838623047,"340":-8.018938064575195}},"graph":{"mac_fp16":{"p":[0.13598083124296936,0.6558461171589416,0.20817305159808905],"max_abs_dp":0.00037579160202522166,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1388825004846176,0.655880023713887,0.20523747580149537],"max_abs_dp":0.002831503978846406,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.13635662284499458,0.6555743973746637,0.20806897978034178]],"input_tokens":106,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.6555743973746637,"probabilities":{"permitted":0.13635662284499458,"prohibited":0.6555743973746637,"insufficient":0.20806897978034178}}},"usage":{"input_tokens":106,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4304}},{"id":"semif_97a35499bd8717f3e786","source":"semif","publication":"publish_text","request":{"state":"The homeowner first refused attic entry, then explicitly granted it before the visit. No later decision exists.","questions":{"answer":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 78 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":78,"line_sha256":"6322c2cbf3063dc0dbd11c78d1e2242b014cdc3455ebabbc00861f2cfe53c71c","licence":"MIT","semif_id":"97a35499bd8717f3e786","group_id":"ca2e0b043985fa49bcd2","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":98,"slot":97,"row_ids_sha256":"872f7abbc066fae581d3a88f9a8ab230fd274c608a6ba86debb32df116c87682","oracle":{"probs":[0.027703689771763466,0.09146626457517126,0.8808300456530653],"argmax":"permitted","top2_margin":0.7893637810778941,"near_tie":false,"group_logz":{"41":-3.598651885986328,"334":-11.182821273803711,"42":-2.404247283935547,"378":-9.431766510009766,"43":-0.13935279846191406,"340":-7.505960464477539}},"graph":{"mac_fp16":{"p":[0.027685719323045536,0.09105373471364955,0.8812605459633049],"max_abs_dp":0.0004305003102396121,"argmax_equal":true},"iphone_int8mlp":{"p":[0.028151752196940463,0.09416702242525529,0.8776812253778042],"max_abs_dp":0.0031488202752610883,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.027703689771763466,0.09146626457517126,0.8808300456530653]],"input_tokens":98,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8808300456530653,"probabilities":{"prohibited":0.027703689771763466,"insufficient":0.09146626457517126,"permitted":0.8808300456530653}}},"usage":{"input_tokens":98,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3911}},{"id":"semif_31eb4533d2c89959ec46","source":"semif","publication":"publish_text","request":{"state":"The homeowner gave an early permission decision, but neither its content nor any later decisions are available.","questions":{"answer":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 79 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":79,"line_sha256":"24b44df64d8382c2ff23b0df84aee7ce8c5d61b09c20f63c49f961cad232f3a3","licence":"MIT","semif_id":"31eb4533d2c89959ec46","group_id":"ca2e0b043985fa49bcd2","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"insufficient","row_len":96,"slot":95,"row_ids_sha256":"3d1ad82ce51613061e24853088d7ed4e6d24658df2757c6a54c06d66e681638f","oracle":{"probs":[0.8166333936350116,0.11432308782787776,0.06904351853711069],"argmax":"insufficient","top2_margin":0.7023103058071338,"near_tie":false,"group_logz":{"41":-0.2146015167236328,"334":-7.90870475769043,"42":-2.1807632446289062,"378":-9.395078659057617,"43":-2.6850547790527344,"340":-9.749147415161133}},"graph":{"mac_fp16":{"p":[0.8169575205780701,0.11407505838138732,0.06896742104054247],"max_abs_dp":0.0003241269430585003,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8162764290509662,0.11416176576620375,0.06956180518282995],"max_abs_dp":0.0005182866457192625,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8166333936350116,0.11432308782787776,0.06904351853711069]],"input_tokens":96,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8166333936350116,"probabilities":{"insufficient":0.8166333936350116,"prohibited":0.11432308782787776,"permitted":0.06904351853711069}}},"usage":{"input_tokens":96,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3738}},{"id":"semif_0aa24a6701ac9c56c4a0","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"answer":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 80 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":80,"line_sha256":"4017dd15d93c91ab811e456dae7c7912599096e1b69b0f484093e6013e841a0c","licence":"MIT","semif_id":"0aa24a6701ac9c56c4a0","group_id":"85051ba3c30277824ebe","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":90,"slot":89,"row_ids_sha256":"77a8fd3d906d0d0cf11562c04892c0a3a2c112881eaa5608c21af02922fac73d","oracle":{"probs":[0.4406316144318769,0.4885140026884189,0.07085438287970432],"argmax":"B","top2_margin":0.04788238825654201,"near_tie":false,"group_logz":{"41":-0.829498291015625,"334":-8.524250030517578,"42":-0.7263393402099609,"378":-8.084783554077148,"43":-2.65708065032959,"340":-10.058631896972656}},"graph":{"mac_fp16":{"p":[0.4389773519018687,0.4903552204270444,0.070667427671087],"max_abs_dp":0.0018412177386255002,"argmax_equal":true},"iphone_int8mlp":{"p":[0.4477302442738626,0.48018144719580824,0.07208830853032906],"max_abs_dp":0.008332555492610638,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.4406316144318769,0.4885140026884189,0.07085438287970432]],"input_tokens":90,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.4885140026884189,"probabilities":{"insufficient":0.4406316144318769,"B":0.4885140026884189,"A":0.07085438287970432}}},"usage":{"input_tokens":90,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3286}},{"id":"semif_796d5c0da6eaadae6a3f","source":"semif","publication":"publish_text","request":{"state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"answer":{"type":"choice","instructions":"Select the candidate documenting an authorization that was later withdrawn.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 81 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":81,"line_sha256":"4487fc039fd3b2e9fc1bf18a7936cea404cb6a4cce2ad0f2df4737a0450ed833","licence":"MIT","semif_id":"796d5c0da6eaadae6a3f","group_id":"85051ba3c30277824ebe","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":85,"slot":84,"row_ids_sha256":"4754db535bf94be99bd4d2473fa575e3a3d4ebd87ec0beeec0fc98bd06c2f6fa","oracle":{"probs":[0.012820504368046244,0.9715533037611729,0.015626191870780935],"argmax":"A","top2_margin":0.9559271118903919,"near_tie":false,"group_logz":{"41":-4.364389419555664,"334":-12.702878952026367,"42":-0.03653907775878906,"378":-8.14625358581543,"43":-4.166486740112305,"340":-12.097513198852539}},"graph":{"mac_fp16":{"p":[0.012825834472526122,0.9715706445007734,0.015603521026700444],"max_abs_dp":2.2670844080490207e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.012849312801917995,0.9716410960757016,0.015509591122380364],"max_abs_dp":0.00011660074840057037,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.012820504368046244,0.9715533037611729,0.015626191870780935]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9715533037611729,"probabilities":{"B":0.012820504368046244,"A":0.9715533037611729,"insufficient":0.015626191870780935}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3434}},{"id":"semif_599b3092b6fd18312d23","source":"semif","publication":"publish_text","request":{"state":"Candidate A: Streaming was authorized and the complete record says permission remains in force. Candidate B: Streaming was authorized, then explicitly withdrawn.","questions":{"answer":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 82 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":82,"line_sha256":"0b1fc7b68e5d20226c4fd87d7cf2f0af2c21ce1cac1e2bb755068f34550e9d4a","licence":"MIT","semif_id":"599b3092b6fd18312d23","group_id":"85051ba3c30277824ebe","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","insufficient","A"],"codes":["A","B","C"],"gold":"A","row_len":85,"slot":84,"row_ids_sha256":"b751ecff13a032027d38015266f9943a4546c8ae963a4c6a8c515d6c87184fc1","oracle":{"probs":[0.013130677586245667,0.02253673755802401,0.9643325848557304],"argmax":"A","top2_margin":0.9417958472977064,"near_tie":false,"group_logz":{"41":-4.33945369720459,"334":-12.724178314208984,"42":-3.799258232116699,"378":-11.586756706237793,"43":-0.04296875,"340":-8.427610397338867}},"graph":{"mac_fp16":{"p":[0.013100606630511942,0.022513521505244345,0.9643858718642437],"max_abs_dp":5.328700851336965e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.013005392607172641,0.02218697711340202,0.9648076302794253],"max_abs_dp":0.00047504542369491976,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.013130677586245667,0.02253673755802401,0.9643325848557304]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9643325848557304,"probabilities":{"B":0.013130677586245667,"insufficient":0.02253673755802401,"A":0.9643325848557304}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3278}},{"id":"semif_960a4bcd27363e6db4b9","source":"semif","publication":"publish_text","request":{"state":"Candidate A: A permission form is mentioned but its terms and later changes are missing. Candidate B: The event name is supplied without permission history.","questions":{"answer":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 83 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":83,"line_sha256":"341b018144f4d27060f78e9d9cb5adc9744bab1c184cf172b1afeca20f6813ed","licence":"MIT","semif_id":"960a4bcd27363e6db4b9","group_id":"85051ba3c30277824ebe","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":85,"slot":84,"row_ids_sha256":"0561ccbf3748366b44c7625e91216ce76397ff62915471b81ac70ac240e70e5f","oracle":{"probs":[0.140967761550587,0.10385373432569021,0.7551785041237228],"argmax":"insufficient","top2_margin":0.6142107425731358,"near_tie":false,"group_logz":{"41":-1.9798946380615234,"334":-9.63957405090332,"42":-2.285442352294922,"378":-9.522225379943848,"43":-0.3014717102050781,"340":-7.881350517272949}},"graph":{"mac_fp16":{"p":[0.14064842068616462,0.10373918945224653,0.755612389861589],"max_abs_dp":0.0004338857378661798,"argmax_equal":true},"iphone_int8mlp":{"p":[0.13899216629872432,0.10371412642872348,0.7572937072725522],"max_abs_dp":0.002115203148829403,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.140967761550587,0.10385373432569021,0.7551785041237228]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.7551785041237228,"probabilities":{"B":0.140967761550587,"A":0.10385373432569021,"insufficient":0.7551785041237228}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3437}},{"id":"semif_c6a411e7f2bbdf377173","source":"semif","publication":"publish_text","request":{"state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 84 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":84,"line_sha256":"4d437323bdea93fd769fb2f2ab69d14aefa976c7cdd814e1217d17209937f25a","licence":"MIT","semif_id":"c6a411e7f2bbdf377173","group_id":"e7798dab5f4f121cab77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":98,"slot":97,"row_ids_sha256":"40d336f3284fb45905dee870a0f8ae52eca0021548d41f40b92835c3234debe1","oracle":{"probs":[0.9122624549291998,0.058177113815911244,0.029560431254889014],"argmax":"contradicted","top2_margin":0.8540853411132885,"near_tie":false,"group_logz":{"41":-0.09815597534179688,"334":-8.12082290649414,"42":-2.8505916595458984,"378":-10.2654447555542,"43":-3.527647018432617,"340":-10.957615852355957}},"graph":{"mac_fp16":{"p":[0.9120814288428417,0.058316463951971986,0.02960210720518633],"max_abs_dp":0.0001810260863580604,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9132313124816692,0.05738636338281876,0.029382324135512113],"max_abs_dp":0.0009688575524694487,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9122624549291998,0.058177113815911244,0.029560431254889014]],"input_tokens":98,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9122624549291998,"probabilities":{"contradicted":0.9122624549291998,"insufficient":0.058177113815911244,"supported":0.029560431254889014}}},"usage":{"input_tokens":98,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3829}},{"id":"semif_e332aa927d0c2ed044aa","source":"semif","publication":"publish_text","request":{"state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: Lina has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 85 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":85,"line_sha256":"95318b27890681c64b5fcd1446106c7127f9cb3ec2f90d89dfeba0d33d984298","licence":"MIT","semif_id":"e332aa927d0c2ed044aa","group_id":"e7798dab5f4f121cab77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":99,"slot":98,"row_ids_sha256":"a76bc30c0a933ff00917245a3d404c75cf82ab2c7c3b2aa45ac3b7e833a2faec","oracle":{"probs":[0.140227116969893,0.06478723311950586,0.7949856499106012],"argmax":"supported","top2_margin":0.6547585329407082,"near_tie":false,"group_logz":{"41":-1.9800233840942383,"334":-9.608638763427734,"42":-2.752178192138672,"378":-10.047652244567871,"43":-0.2449626922607422,"340":-7.8398590087890625}},"graph":{"mac_fp16":{"p":[0.13966294563931833,0.0646455365436371,0.7956915178170446],"max_abs_dp":0.000705867906443447,"argmax_equal":true},"iphone_int8mlp":{"p":[0.13763236086126965,0.06348990509710738,0.798877734041623],"max_abs_dp":0.0038920841310218224,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.140227116969893,0.06478723311950586,0.7949856499106012]],"input_tokens":99,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.7949856499106012,"probabilities":{"contradicted":0.140227116969893,"insufficient":0.06478723311950586,"supported":0.7949856499106012}}},"usage":{"input_tokens":99,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3846}},{"id":"semif_a0128e551d095f3449f8","source":"semif","publication":"publish_text","request":{"state":"The director gives Mo direct authority to approve exhibit labels; the record confirms this grant is valid.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 86 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":86,"line_sha256":"c561f9f9e2a0289e139b78c42155a5e356c368effcf01d8b07096760c52428bc","licence":"MIT","semif_id":"a0128e551d095f3449f8","group_id":"e7798dab5f4f121cab77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":74,"slot":73,"row_ids_sha256":"a01e1551b96d97bc65b127bb6269c8aed31fe4b8d09303b30eb108ba96759884","oracle":{"probs":[0.9571298754939811,0.03005492042516889,0.012815204080850054],"argmax":"supported","top2_margin":0.9270749550688122,"near_tie":false,"group_logz":{"41":-0.052730560302734375,"334":-8.57982349395752,"42":-3.513643264770508,"378":-11.584104537963867,"43":-4.366037368774414,"340":-12.566411972045898}},"graph":{"mac_fp16":{"p":[0.957129919330281,0.030055085629810133,0.012814995039908926],"max_abs_dp":2.0904094112823846e-07,"argmax_equal":true},"iphone_int8mlp":{"p":[0.957006154056187,0.03013372070037132,0.012860125243441562],"max_abs_dp":0.00012372143779404432,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9571298754939811,0.03005492042516889,0.012815204080850054]],"input_tokens":74,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9571298754939811,"probabilities":{"supported":0.9571298754939811,"insufficient":0.03005492042516889,"contradicted":0.012815204080850054}}},"usage":{"input_tokens":74,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3103}},{"id":"semif_46d6a0731bd288c370f0","source":"semif","publication":"publish_text","request":{"state":"Mo is named in correspondence about labels, but no authority grants or delegation terms are supplied.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 87 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":87,"line_sha256":"1c46f276c92e30a59a18eb317ef01a3370f18840f7f945f3f77d42310f04dffe","licence":"MIT","semif_id":"46d6a0731bd288c370f0","group_id":"e7798dab5f4f121cab77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","contradicted","supported"],"codes":["A","B","C"],"gold":"insufficient","row_len":73,"slot":72,"row_ids_sha256":"e04a2c7b0a55cb81a2049a43529c599ade6aa63fa6fdaf943acb5c1e45b91dbb","oracle":{"probs":[0.5965094515037042,0.3770391883510236,0.026451360145272194],"argmax":"insufficient","top2_margin":0.21947026315268053,"near_tie":false,"group_logz":{"41":-0.5271720886230469,"334":-8.122379302978516,"42":-0.9859180450439453,"378":-8.300433158874512,"43":-3.6429595947265625,"340":-10.872095108032227}},"graph":{"mac_fp16":{"p":[0.596686985144864,0.3768375991936467,0.026475415661489383],"max_abs_dp":0.00020158915737694683,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5945466547525275,0.3788453437582673,0.02660800148920515],"max_abs_dp":0.00196279675117661,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5965094515037042,0.3770391883510236,0.026451360145272194]],"input_tokens":73,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5965094515037042,"probabilities":{"insufficient":0.5965094515037042,"contradicted":0.3770391883510236,"supported":0.026451360145272194}}},"usage":{"input_tokens":73,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.317}},{"id":"semif_21ca7a8b970e36b8e20f","source":"semif","publication":"publish_text","request":{"state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"answer":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 88 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":88,"line_sha256":"cb107f8ae09d2978fe7d4edf4b4553479e26fad548859ed32a1819b9c51dd144","licence":"MIT","semif_id":"21ca7a8b970e36b8e20f","group_id":"928ebc7817a8f1cac3e4","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"prohibited","row_len":106,"slot":105,"row_ids_sha256":"19315a7746aac115a007c065bfb11b93d47c63e0de3cb1bee1d4bd9b925a281b","oracle":{"probs":[0.9067816237177496,0.06423170799249191,0.02898666828975844],"argmax":"prohibited","top2_margin":0.8425499157252577,"near_tie":false,"group_logz":{"41":-0.10588264465332031,"334":-7.8498077392578125,"42":-2.7532873153686523,"378":-9.91721248626709,"43":-3.548948287963867,"340":-10.734498023986816}},"graph":{"mac_fp16":{"p":[0.9068322983805063,0.06419150861530143,0.028976193004192276],"max_abs_dp":5.0674662756655664e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9065304774182613,0.06422047513532929,0.029249047446409368],"max_abs_dp":0.0002623791566509284,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9067816237177496,0.06423170799249191,0.02898666828975844]],"input_tokens":106,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9067816237177496,"probabilities":{"prohibited":0.9067816237177496,"insufficient":0.06423170799249191,"permitted":0.02898666828975844}}},"usage":{"input_tokens":106,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.38}},{"id":"semif_4aa83af95bb69e90bc2d","source":"semif","publication":"publish_text","request":{"state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"answer":{"type":"choice","instructions":"The stage manager may delegate prop approval to an assistant, and doing so grants the assistant approval authority. May this assistant approve?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 89 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":89,"line_sha256":"67d4bfc2d38d76f11ee3e41c1171ef5598772d2c0d7b0de1c9e49017c81dc2de","licence":"MIT","semif_id":"4aa83af95bb69e90bc2d","group_id":"928ebc7817a8f1cac3e4","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":104,"slot":103,"row_ids_sha256":"d52a149b3c9832d45789fd7060a48f1af974ee6eadc1cec0148e04fef6961b1b","oracle":{"probs":[0.21019431806480404,0.2587289115683538,0.5310767703668421],"argmax":"insufficient","top2_margin":0.2723478587984883,"near_tie":false,"group_logz":{"41":-1.5811519622802734,"334":-8.588186264038086,"42":-1.373403549194336,"378":-8.095871925354004,"43":-0.6542778015136719,"340":-7.437939643859863}},"graph":{"mac_fp16":{"p":[0.20983348717557732,0.2589595046817893,0.5312070081426334],"max_abs_dp":0.00036083088922672646,"argmax_equal":true},"iphone_int8mlp":{"p":[0.20506056938019998,0.2615117933357212,0.5334276372840787],"max_abs_dp":0.005133748684604061,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.21019431806480404,0.2587289115683538,0.5310767703668421]],"input_tokens":104,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5310767703668421,"probabilities":{"permitted":0.21019431806480404,"prohibited":0.2587289115683538,"insufficient":0.5310767703668421}}},"usage":{"input_tokens":104,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3888}},{"id":"semif_cdfa9eed51fafcf09b72","source":"semif","publication":"publish_text","request":{"state":"The producer directly authorizes the assistant to approve props.","questions":{"answer":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 90 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":90,"line_sha256":"0a2fc850c6525063e8e7f27c6442e1316283526b31e36295fc66530eaa71435b","licence":"MIT","semif_id":"cdfa9eed51fafcf09b72","group_id":"928ebc7817a8f1cac3e4","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":89,"slot":88,"row_ids_sha256":"e15b69897503064a51d57fff4f294d636ed695f67ee9f6fecf1714c220e09113","oracle":{"probs":[0.2685018487923648,0.6261517789937134,0.10534637221392172],"argmax":"prohibited","top2_margin":0.35764993020134866,"near_tie":false,"group_logz":{"41":-1.3289012908935547,"334":-8.629948616027832,"42":-0.4821662902832031,"378":-7.660808563232422,"43":-2.264505386352539,"340":-9.21031379699707}},"graph":{"mac_fp16":{"p":[0.26826622172652076,0.6266115892541144,0.10512218901936483],"max_abs_dp":0.0004598102604009835,"argmax_equal":true},"iphone_int8mlp":{"p":[0.26554307410748906,0.6311261947051153,0.10333073118739569],"max_abs_dp":0.004974415711401892,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.2685018487923648,0.6261517789937134,0.10534637221392172]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.6261517789937134,"probabilities":{"insufficient":0.2685018487923648,"prohibited":0.6261517789937134,"permitted":0.10534637221392172}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3279}},{"id":"semif_d51b05bfb9d6d7f09331","source":"semif","publication":"publish_text","request":{"state":"An assistant works on the production, but no approval grants or delegation history are provided.","questions":{"answer":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 91 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":91,"line_sha256":"a33d0facd74007f56a160a7e12d893d9ecab43b60a5ec0ce3b05523c820bd036","licence":"MIT","semif_id":"d51b05bfb9d6d7f09331","group_id":"928ebc7817a8f1cac3e4","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","insufficient","permitted"],"codes":["A","B","C"],"gold":"insufficient","row_len":95,"slot":94,"row_ids_sha256":"f3881bb5fac7c644795a4daffc2075993983bd3074dec7263500a09eac7f12de","oracle":{"probs":[0.5489752327797446,0.40998565644846174,0.041039110771793647],"argmax":"prohibited","top2_margin":0.13898957633128284,"near_tie":false,"group_logz":{"41":-0.6116161346435547,"334":-8.013121604919434,"42":-0.9035472869873047,"378":-8.030211448669434,"43":-3.205143928527832,"340":-10.182268142700195}},"graph":{"mac_fp16":{"p":[0.5467922752725698,0.4121448916872784,0.0410628330401519],"max_abs_dp":0.0021829575071747875,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5541266102652984,0.40415210617517583,0.041721283559525736],"max_abs_dp":0.005833550273285915,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5489752327797446,0.40998565644846174,0.041039110771793647]],"input_tokens":95,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.5489752327797446,"probabilities":{"prohibited":0.5489752327797446,"insufficient":0.40998565644846174,"permitted":0.041039110771793647}}},"usage":{"input_tokens":95,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.5258}},{"id":"semif_fdd902cc6e65fad7cb4f","source":"semif","publication":"publish_text","request":{"state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"answer":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 92 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":92,"line_sha256":"01ceabcdf17606074dcc6242e11c9ad7d76b21e95ae96653c2ef513e1bb11fb2","licence":"MIT","semif_id":"fdd902cc6e65fad7cb4f","group_id":"53b0770ec36835b81d3a","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":98,"slot":97,"row_ids_sha256":"a26258e9681c77c486661a202d522437bc8a1e1d77f3003d36c581a71113a74c","oracle":{"probs":[0.03788929741773759,0.9101205199608678,0.051990182621394467],"argmax":"A","top2_margin":0.8581303373394734,"near_tie":false,"group_logz":{"41":-3.283766746520996,"334":-10.9194917678833,"42":-0.1048583984375,"378":-7.645013809204102,"43":-2.9673805236816406,"340":-10.247072219848633}},"graph":{"mac_fp16":{"p":[0.03798713750549632,0.9100041537709611,0.052008708723542574],"max_abs_dp":0.00011636618990673586,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03782431063400553,0.9101639169007593,0.052011772465235244],"max_abs_dp":6.49867837320614e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03788929741773759,0.9101205199608678,0.051990182621394467]],"input_tokens":98,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9101205199608678,"probabilities":{"B":0.03788929741773759,"A":0.9101205199608678,"insufficient":0.051990182621394467}}},"usage":{"input_tokens":98,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3788}},{"id":"semif_42384af9ae56beb37ff5","source":"semif","publication":"publish_text","request":{"state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"answer":{"type":"choice","instructions":"Select the candidate whose claimed authority comes only through a prohibited second delegation.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 93 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":93,"line_sha256":"2813af05a60a0fb7d63dacf7d4cdadd5144d59573a06e5f02d9fa86ee214d7b4","licence":"MIT","semif_id":"42384af9ae56beb37ff5","group_id":"53b0770ec36835b81d3a","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"B","row_len":98,"slot":97,"row_ids_sha256":"bbc275ce00408a4227b4dc5575d70872565921f6129505bd26a462671c4fce75","oracle":{"probs":[0.11450524801529813,0.06458278002705178,0.8209119719576501],"argmax":"B","top2_margin":0.7064067239423519,"near_tie":false,"group_logz":{"41":-2.177976608276367,"334":-9.59929084777832,"42":-2.7506494522094727,"378":-9.792638778686523,"43":-0.20818138122558594,"340":-7.619632720947266}},"graph":{"mac_fp16":{"p":[0.11431768898554304,0.06435128909931365,0.8213310219151433],"max_abs_dp":0.0004190499574932005,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1119812307881701,0.06271826735079813,0.8253005018610319],"max_abs_dp":0.004388529903381788,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.11450524801529813,0.06458278002705178,0.8209119719576501]],"input_tokens":98,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8209119719576501,"probabilities":{"insufficient":0.11450524801529813,"A":0.06458278002705178,"B":0.8209119719576501}}},"usage":{"input_tokens":98,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3777}},{"id":"semif_cdf7f428857e1b841cb3","source":"semif","publication":"publish_text","request":{"state":"The rule permits direct delegation by the director but forbids further delegation. Candidate A: My only claimed authority came from B. Candidate B: The director directly granted me prop approval authority.","questions":{"answer":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 94 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":94,"line_sha256":"03bc0db267eff459bb736b3c3b40dad41880dc42af3ccdb8b0a4ced9b900989e","licence":"MIT","semif_id":"cdf7f428857e1b841cb3","group_id":"53b0770ec36835b81d3a","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","insufficient","B"],"codes":["A","B","C"],"gold":"B","row_len":91,"slot":90,"row_ids_sha256":"3554386bb13b7f8c776c9d838014390aa02863cd78a3f33a59743bc17d56a565","oracle":{"probs":[0.1370019981652202,0.2592888246297449,0.6037091772050348],"argmax":"B","top2_margin":0.3444203525752899,"near_tie":false,"group_logz":{"41":-2.0043582916259766,"334":-9.442790985107422,"42":-1.3664112091064453,"378":-8.47912311553955,"43":-0.5212612152099609,"340":-7.777520179748535}},"graph":{"mac_fp16":{"p":[0.13694980320977543,0.25874835649154143,0.6043018402986832],"max_abs_dp":0.0005926630936483601,"argmax_equal":true},"iphone_int8mlp":{"p":[0.13801981469874222,0.2587715388245943,0.6032086464766636],"max_abs_dp":0.001017816533522009,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.1370019981652202,0.2592888246297449,0.6037091772050348]],"input_tokens":91,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.6037091772050348,"probabilities":{"A":0.1370019981652202,"insufficient":0.2592888246297449,"B":0.6037091772050348}}},"usage":{"input_tokens":91,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3375}},{"id":"semif_774f7f9245f83c796d44","source":"semif","publication":"publish_text","request":{"state":"The director's delegation rule is supplied, but Candidate A and Candidate B provide only their names and no authority histories.","questions":{"answer":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 95 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":95,"line_sha256":"62b3518676733167a27524fa094c74cb9467666a6440cb3c3d771d18831ae001","licence":"MIT","semif_id":"774f7f9245f83c796d44","group_id":"53b0770ec36835b81d3a","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":77,"slot":76,"row_ids_sha256":"a3bb9ba8d249336e68fd8e3500f802078ea341c6b8e25a2e030d7bb367757275","oracle":{"probs":[0.025427175977262212,0.018237800700862925,0.9563350233218749],"argmax":"insufficient","top2_margin":0.9309078473446127,"near_tie":false,"group_logz":{"41":-3.683262825012207,"334":-12.039041519165039,"42":-4.015584945678711,"378":-11.877917289733887,"43":-0.055973052978515625,"340":-8.315942764282227}},"graph":{"mac_fp16":{"p":[0.025475637221016464,0.01829942293597267,0.9562249398430107],"max_abs_dp":0.0001100834788642091,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02491961957282977,0.017935941653531778,0.9571444387736384],"max_abs_dp":0.0008094154517634955,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.025427175977262212,0.018237800700862925,0.9563350233218749]],"input_tokens":77,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9563350233218749,"probabilities":{"B":0.025427175977262212,"A":0.018237800700862925,"insufficient":0.9563350233218749}}},"usage":{"input_tokens":77,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3168}},{"id":"semif_92ec2ffa3dd46ada1335","source":"semif","publication":"publish_text","request":{"state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 96 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":96,"line_sha256":"f4e473cd8feb10ea7874d3eae00efa068c094e2d17551c15dc7a0dffbf286d29","licence":"MIT","semif_id":"92ec2ffa3dd46ada1335","group_id":"056cc5b726d6e8300a77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":96,"slot":95,"row_ids_sha256":"e104f830481d5111cc9024b794472d4eed62303dd1e6041c073782c0a2d7dd2c","oracle":{"probs":[0.7766075079866954,0.07898456130056934,0.14440793071273514],"argmax":"contradicted","top2_margin":0.6321995772739603,"near_tie":false,"group_logz":{"41":-0.26546287536621094,"334":-7.760525703430176,"42":-2.551145553588867,"378":-9.65746784210205,"43":-1.9477558135986328,"340":-8.914478302001953}},"graph":{"mac_fp16":{"p":[0.7766072642838301,0.07887994576821436,0.14451278994795547],"max_abs_dp":0.00010485923522032814,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7774728695499854,0.0782620147634337,0.14426511568658087],"max_abs_dp":0.0008653615632899792,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.7766075079866954,0.07898456130056934,0.14440793071273514]],"input_tokens":96,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.7766075079866954,"probabilities":{"contradicted":0.7766075079866954,"insufficient":0.07898456130056934,"supported":0.14440793071273514}}},"usage":{"input_tokens":96,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3306}},{"id":"semif_6149a17bc154f9c5b4a0","source":"semif","publication":"publish_text","request":{"state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the lamp has a fault.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 97 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":97,"line_sha256":"c8abb18dd997b9c8e790fc26c1e2d63aba6d7abe2dab8fd4545323f7ef2e1998","licence":"MIT","semif_id":"6149a17bc154f9c5b4a0","group_id":"056cc5b726d6e8300a77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":91,"slot":90,"row_ids_sha256":"71c6ca58484905eeb99d6cc4f014c82a8ffaa7a17c853805c6a26cf56910dcd0","oracle":{"probs":[0.9490568215763293,0.027025413045799758,0.023917765377870884],"argmax":"supported","top2_margin":0.9220314085305296,"near_tie":false,"group_logz":{"41":-0.06248283386230469,"334":-8.28081226348877,"42":-3.621173858642578,"378":-11.503377914428711,"43":-3.7433300018310547,"340":-11.56765079498291}},"graph":{"mac_fp16":{"p":[0.9491041655145257,0.026982449033272352,0.023913385452201993],"max_abs_dp":4.7343938196386937e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9493651780905343,0.02693590036169984,0.02369892154776579],"max_abs_dp":0.0003083565142050215,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9490568215763293,0.027025413045799758,0.023917765377870884]],"input_tokens":91,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9490568215763293,"probabilities":{"supported":0.9490568215763293,"insufficient":0.027025413045799758,"contradicted":0.023917765377870884}}},"usage":{"input_tokens":91,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3322}},{"id":"semif_be866e1dbec5d42d9423","source":"semif","publication":"publish_text","request":{"state":"The return policy accepts faulty lamps despite opened packaging, but excludes water damage even when faulty. This opened lamp has a manufacturing wiring fault, with no water damage or other exclusion.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 98 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":98,"line_sha256":"da3acbab2ca934c780a3e0c349d61d6bc6f34f93ade5e3c39d1a03f0784f785e","licence":"MIT","semif_id":"be866e1dbec5d42d9423","group_id":"056cc5b726d6e8300a77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":93,"slot":92,"row_ids_sha256":"c9b67666195ec099e77ec8419dd51f702034430accc8e90a5b8536d34bfc647a","oracle":{"probs":[0.02871191220751386,0.016964489256584232,0.954323598535902],"argmax":"supported","top2_margin":0.9256116863283881,"near_tie":false,"group_logz":{"41":-3.561159133911133,"334":-11.31085205078125,"42":-4.087348937988281,"378":-11.505402565002441,"43":-0.057468414306640625,"340":-7.749194145202637}},"graph":{"mac_fp16":{"p":[0.028712359531779117,0.01696046785694828,0.9543271726112725],"max_abs_dp":4.021399635952688e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.028518793873025812,0.016895251316125896,0.9545859548108483],"max_abs_dp":0.0002623562749463071,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.02871191220751386,0.016964489256584232,0.954323598535902]],"input_tokens":93,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.954323598535902,"probabilities":{"contradicted":0.02871191220751386,"insufficient":0.016964489256584232,"supported":0.954323598535902}}},"usage":{"input_tokens":93,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3316}},{"id":"semif_97d2ed2591fc7638806c","source":"semif","publication":"publish_text","request":{"state":"The return policy accepts faulty lamps despite opened packaging but excludes water damage even when faulty. The lamp is faulty, but the cause and presence of water damage are omitted.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 99 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":99,"line_sha256":"58b1bebbb4f24da0739f2f867e9b48cc84a0c3f8a15d9638d95145db5ddff591","licence":"MIT","semif_id":"97d2ed2591fc7638806c","group_id":"056cc5b726d6e8300a77","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"insufficient","row_len":91,"slot":90,"row_ids_sha256":"7ddd0dac0c704943d83ee81eb1b3855dc1f2b2c0776153c9f21658e061ebb7e0","oracle":{"probs":[0.22356297563091476,0.6992692470126346,0.07716777735645057],"argmax":"supported","top2_margin":0.47570627138171984,"near_tie":false,"group_logz":{"41":-1.5136985778808594,"334":-8.78387451171875,"42":-0.3733558654785156,"378":-7.607934951782227,"43":-2.5774097442626953,"340":-9.628934860229492}},"graph":{"mac_fp16":{"p":[0.22367419952941292,0.6993182389081732,0.07700756156241396],"max_abs_dp":0.00016021579403660557,"argmax_equal":true},"iphone_int8mlp":{"p":[0.21818842497532095,0.7040662293615425,0.07774534566313648],"max_abs_dp":0.005374550655593807,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.22356297563091476,0.6992692470126346,0.07716777735645057]],"input_tokens":91,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.6992692470126346,"probabilities":{"insufficient":0.22356297563091476,"supported":0.6992692470126346,"contradicted":0.07716777735645057}}},"usage":{"input_tokens":91,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3382}},{"id":"semif_4533dd91267b4d815690","source":"semif","publication":"publish_text","request":{"state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"answer":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 100 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":100,"line_sha256":"b2c8e57b970d4a9a6e9f986d47e8ff0645cfc0f0f3c3b693ace5427831314e40","licence":"MIT","semif_id":"4533dd91267b4d815690","group_id":"e3ce8ec496dbc9974f94","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"prohibited","row_len":100,"slot":99,"row_ids_sha256":"e5d1fd606064721f9b223d313f3839b2790d4dcd6f8572fcc94d4768c5c00abe","oracle":{"probs":[0.044351632935348356,0.042946673911479265,0.9127016931531724],"argmax":"prohibited","top2_margin":0.868350060217824,"near_tie":false,"group_logz":{"41":-3.1274280548095703,"334":-10.606481552124023,"42":-3.159618377685547,"378":-10.244626998901367,"43":-0.10316848754882812,"340":-7.541219711303711}},"graph":{"mac_fp16":{"p":[0.04429409231856542,0.042904445321005985,0.9128014623604285],"max_abs_dp":9.976920725607297e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0446821128741758,0.04307441332469313,0.9122434738011311],"max_abs_dp":0.00045821935204126696,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.044351632935348356,0.042946673911479265,0.9127016931531724]],"input_tokens":100,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.9127016931531724,"probabilities":{"permitted":0.044351632935348356,"insufficient":0.042946673911479265,"prohibited":0.9127016931531724}}},"usage":{"input_tokens":100,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3706}},{"id":"semif_d880e65fef5e2993fdd0","source":"semif","publication":"publish_text","request":{"state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"answer":{"type":"choice","instructions":"Accept defective paint even if opened or mixed with solvent; neither condition overrides the defect exception. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 101 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":101,"line_sha256":"eebb6da982496d47bb110182f50520ab215bd53576b9c9a14cd646d36c6e83c0","licence":"MIT","semif_id":"d880e65fef5e2993fdd0","group_id":"e3ce8ec496dbc9974f94","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":95,"slot":94,"row_ids_sha256":"a2adceb5499e307eb023224cdb0f6844e12698579d26794476575c716470778c","oracle":{"probs":[0.49572117631364243,0.26657614507505045,0.23770267861130703],"argmax":"permitted","top2_margin":0.22914503123859198,"near_tie":false,"group_logz":{"41":-0.7287483215332031,"334":-7.520839691162109,"42":-1.3491020202636719,"378":-7.862666130065918,"43":-1.4637413024902344,"340":-7.906989097595215}},"graph":{"mac_fp16":{"p":[0.49748486247677937,0.2659530080157717,0.2365621295074489],"max_abs_dp":0.001763686163136935,"argmax_equal":true},"iphone_int8mlp":{"p":[0.49658406163398316,0.26682135108992183,0.236594587276095],"max_abs_dp":0.0011080913352120303,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.49572117631364243,0.26657614507505045,0.23770267861130703]],"input_tokens":95,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.49572117631364243,"probabilities":{"permitted":0.49572117631364243,"insufficient":0.26657614507505045,"prohibited":0.23770267861130703}}},"usage":{"input_tokens":95,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.341}},{"id":"semif_90494c76078ffc3c717a","source":"semif","publication":"publish_text","request":{"state":"The opened tin has a manufacturing defect and the inspection confirms no solvent was added.","questions":{"answer":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 102 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":102,"line_sha256":"3b79d2949f80573b5048e271419c1654d57a5e5ee94f59449d009714b0ba92d0","licence":"MIT","semif_id":"90494c76078ffc3c717a","group_id":"e3ce8ec496dbc9974f94","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":97,"slot":96,"row_ids_sha256":"c45d3e7c39f150c5e156b96dc75ff4a6816521f3988ef8b3f8ff67ddf0809414","oracle":{"probs":[0.23341537011322386,0.5709808030163245,0.19560382687045172],"argmax":"insufficient","top2_margin":0.3375654329031006,"near_tie":false,"group_logz":{"41":-1.473464012145996,"334":-8.383761405944824,"42":-0.5789279937744141,"378":-7.398436546325684,"43":-1.6501922607421875,"340":-8.231147766113281}},"graph":{"mac_fp16":{"p":[0.23231255967039177,0.5725890369272681,0.19509840340234005],"max_abs_dp":0.0016082339109436683,"argmax_equal":true},"iphone_int8mlp":{"p":[0.23539786328304102,0.5675518291565125,0.1970503075604465],"max_abs_dp":0.003428973859812001,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.23341537011322386,0.5709808030163245,0.19560382687045172]],"input_tokens":97,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.5709808030163245,"probabilities":{"permitted":0.23341537011322386,"insufficient":0.5709808030163245,"prohibited":0.19560382687045172}}},"usage":{"input_tokens":97,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3778}},{"id":"semif_9ac5cc82c4126c91174a","source":"semif","publication":"publish_text","request":{"state":"The opened tin is defective, but whether solvent was added is not known.","questions":{"answer":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 103 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":103,"line_sha256":"2efefa757e37f35491fdb1760e1a33670eec1133282603d095dcac6fa5c6025d","licence":"MIT","semif_id":"9ac5cc82c4126c91174a","group_id":"e3ce8ec496dbc9974f94","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"insufficient","row_len":96,"slot":95,"row_ids_sha256":"fa7f7e17d9958013d7064bbc754f9e9a579a7d6dba6f4f4b06f7bcb0812ddb52","oracle":{"probs":[0.07222282999206803,0.8247040567531045,0.10307311325482747],"argmax":"insufficient","top2_margin":0.721630943498277,"near_tie":false,"group_logz":{"41":-2.6408824920654297,"334":-9.821174621582031,"42":-0.2056140899658203,"378":-7.314765930175781,"43":-2.2852001190185547,"340":-9.051528930664062}},"graph":{"mac_fp16":{"p":[0.07230979040262354,0.8242675139754155,0.10342269562196095],"max_abs_dp":0.0004365427776890085,"argmax_equal":true},"iphone_int8mlp":{"p":[0.07309451608317055,0.8216328246334483,0.10527265928338103],"max_abs_dp":0.0030712321196562087,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.07222282999206803,0.8247040567531045,0.10307311325482747]],"input_tokens":96,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8247040567531045,"probabilities":{"permitted":0.07222282999206803,"insufficient":0.8247040567531045,"prohibited":0.10307311325482747}}},"usage":{"input_tokens":96,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.339}},{"id":"semif_386e10b1c1ce906473a4","source":"semif","publication":"publish_text","request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"answer":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 104 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":104,"line_sha256":"f850047e7afe4504ef239b613b8bbf29a52335e69db396d9d2c33ab1b8b2f4e7","licence":"MIT","semif_id":"386e10b1c1ce906473a4","group_id":"a5b234c2e4fb304f1071","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":104,"slot":103,"row_ids_sha256":"217f46e2eed87bf42ca318c5b8db860f17e93ec23b067b8c4c9f0143ebc671bc","oracle":{"probs":[0.014835197790991475,0.9674507141234867,0.017714088085521787],"argmax":"A","top2_margin":0.9497366260379649,"near_tie":false,"group_logz":{"41":-4.217350006103516,"334":-12.668209075927734,"42":-0.0396881103515625,"378":-8.393699645996094,"43":-4.039992332458496,"340":-12.132036209106445}},"graph":{"mac_fp16":{"p":[0.014823402612892868,0.967494834219165,0.017681763167941986],"max_abs_dp":4.412009567833497e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.014747450532831032,0.9676068463370366,0.017645703130132306],"max_abs_dp":0.000156132213549931,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.014835197790991475,0.9674507141234867,0.017714088085521787]],"input_tokens":104,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9674507141234867,"probabilities":{"B":0.014835197790991475,"A":0.9674507141234867,"insufficient":0.017714088085521787}}},"usage":{"input_tokens":104,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3821}},{"id":"semif_4da6aedc1139224eeff2","source":"semif","publication":"publish_text","request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"answer":{"type":"choice","instructions":"Select the submission disqualified specifically by the signature exclusion despite the outage exception.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 105 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":105,"line_sha256":"dd3378780bc7ad688a83f47c1dbf5e86c213ca3cfc7550e79e6b938922f21b97","licence":"MIT","semif_id":"4da6aedc1139224eeff2","group_id":"a5b234c2e4fb304f1071","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":110,"slot":109,"row_ids_sha256":"0328d4fb318225a98afb8494e64f153836b1c7d7f77fa2a5474301cf0a90488b","oracle":{"probs":[0.009220356119364684,0.9831675412734896,0.00761210260714581],"argmax":"B","top2_margin":0.9739471851541249,"near_tie":false,"group_logz":{"41":-4.691586494445801,"334":-13.15929889678955,"42":-0.022220611572265625,"378":-8.314470291137695,"43":-4.883260726928711,"340":-13.0211820602417}},"graph":{"mac_fp16":{"p":[0.009234747475482214,0.9831449988343024,0.007620253690215462],"max_abs_dp":2.254243918720089e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009224508429073575,0.983175878726839,0.007599612844087511],"max_abs_dp":1.2489763058298915e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.009220356119364684,0.9831675412734896,0.00761210260714581]],"input_tokens":110,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.9831675412734896,"probabilities":{"insufficient":0.009220356119364684,"B":0.9831675412734896,"A":0.00761210260714581}}},"usage":{"input_tokens":110,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.4197}},{"id":"semif_a22dab8ba073ff6cdb8d","source":"semif","publication":"publish_text","request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My late outage-related submission lacks the signature. Candidate B: My late outage-related submission has the signature. Both outages are documented.","questions":{"answer":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 106 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":106,"line_sha256":"786e99c38e1c8fa4ad1ac4fa772f7fa79e85c3ce07e190fbedb69fa98153db19","licence":"MIT","semif_id":"a22dab8ba073ff6cdb8d","group_id":"a5b234c2e4fb304f1071","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":100,"slot":99,"row_ids_sha256":"32ed3bd2c08b651d880d02282d0669ca2d43c63066eb40735051e45cc5aa8ca8","oracle":{"probs":[0.055498502541988876,0.8813767102915385,0.06312478716647262],"argmax":"B","top2_margin":0.8182519231250659,"near_tie":false,"group_logz":{"41":-2.8989105224609375,"334":-10.965217590332031,"42":-0.13378143310546875,"378":-8.069723129272461,"43":-2.770153045654297,"340":-10.585244178771973}},"graph":{"mac_fp16":{"p":[0.05544275968866657,0.8814512144969024,0.06310602581443099],"max_abs_dp":7.450420536392333e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.055009639895853814,0.8819774797273721,0.06301288037677417],"max_abs_dp":0.0006007694358335369,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.055498502541988876,0.8813767102915385,0.06312478716647262]],"input_tokens":100,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8813767102915385,"probabilities":{"insufficient":0.055498502541988876,"B":0.8813767102915385,"A":0.06312478716647262}}},"usage":{"input_tokens":100,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3552}},{"id":"semif_42c54fa2ad45743e24d8","source":"semif","publication":"publish_text","request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A and Candidate B report late submissions but provide neither outage documentation nor signature information.","questions":{"answer":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 107 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":107,"line_sha256":"4e686a729a1cd636c46b8915083c9e773501c55305d2107f455a4a2c0e16f842","licence":"MIT","semif_id":"42c54fa2ad45743e24d8","group_id":"a5b234c2e4fb304f1071","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","insufficient","B"],"codes":["A","B","C"],"gold":"insufficient","row_len":86,"slot":85,"row_ids_sha256":"0f33585156730e0e5a3054bb0f74b3f5c6146ea13e1fedacfd2cff0d25d1d00b","oracle":{"probs":[0.03314994995680174,0.934563949100916,0.032286100942282354],"argmax":"insufficient","top2_margin":0.9014139991441142,"near_tie":false,"group_logz":{"41":-3.415599822998047,"334":-11.72950553894043,"42":-0.07656097412109375,"378":-8.135361671447754,"43":-3.4420042037963867,"340":-11.332996368408203}},"graph":{"mac_fp16":{"p":[0.03312770969061111,0.9346002487126521,0.03227204159673683],"max_abs_dp":3.6299611736123616e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03316011202896957,0.9347099032698353,0.0321299847011951],"max_abs_dp":0.00015611624108725214,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03314994995680174,0.934563949100916,0.032286100942282354]],"input_tokens":86,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.934563949100916,"probabilities":{"A":0.03314994995680174,"insufficient":0.934563949100916,"B":0.032286100942282354}}},"usage":{"input_tokens":86,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3146}},{"id":"semif_f3d13cf184f916677def","source":"semif","publication":"publish_text","request":{"state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 108 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":108,"line_sha256":"fe1bd700290ca4326b420a6bcc56d29ff68b13059139ad0f0d4f3f852cf80583","licence":"MIT","semif_id":"f3d13cf184f916677def","group_id":"d142f7fd0efa0eddc788","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":99,"slot":98,"row_ids_sha256":"7cba9a1b219862bcbc7d849e45e54ebeb8cd9564de8fbeeb398281c5f10318d0","oracle":{"probs":[0.9496976364361237,0.0161724266630653,0.03412993690081103],"argmax":"contradicted","top2_margin":0.9155676995353127,"near_tie":false,"group_logz":{"41":-0.05587196350097656,"334":-8.548249244689941,"42":-4.1287078857421875,"378":-12.067581176757812,"43":-3.381840705871582,"340":-11.277971267700195}},"graph":{"mac_fp16":{"p":[0.9497775709925588,0.01615648613469075,0.03406594287275054],"max_abs_dp":7.993455643506842e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9500797566053143,0.015964018468683932,0.03395622492600173],"max_abs_dp":0.0003821201691905918,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9496976364361237,0.0161724266630653,0.03412993690081103]],"input_tokens":99,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.9496976364361237,"probabilities":{"contradicted":0.9496976364361237,"insufficient":0.0161724266630653,"supported":0.03412993690081103}}},"usage":{"input_tokens":99,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.361}},{"id":"semif_1105577da4c8dad4609d","source":"semif","publication":"publish_text","request":{"state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the desk log reports the crate in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 109 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":109,"line_sha256":"8cbd882135506ead376a5a0c54728802f4dbefc47e8c031dc0c191470c88585c","licence":"MIT","semif_id":"1105577da4c8dad4609d","group_id":"d142f7fd0efa0eddc788","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":96,"slot":95,"row_ids_sha256":"c7e00998defd0d6eb3318bf1d4778c0e42cbe216c9176afa37c143006320212c","oracle":{"probs":[0.011407460792647239,0.009986668070278735,0.9786058711370741],"argmax":"supported","top2_margin":0.9671984103444269,"near_tie":false,"group_logz":{"41":-4.481322288513184,"334":-13.156651496887207,"42":-4.6143388748168945,"378":-12.753986358642578,"43":-0.029460906982421875,"340":-8.702664375305176}},"graph":{"mac_fp16":{"p":[0.011398053158212105,0.009977736240421748,0.9786242106013661],"max_abs_dp":1.8339464291972085e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.011258299329175069,0.009895977259533935,0.9788457234112911],"max_abs_dp":0.0002398522742169673,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.011407460792647239,0.009986668070278735,0.9786058711370741]],"input_tokens":96,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9786058711370741,"probabilities":{"contradicted":0.011407460792647239,"insufficient":0.009986668070278735,"supported":0.9786058711370741}}},"usage":{"input_tokens":96,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3156}},{"id":"semif_0ebd299739302d8b057b","source":"semif","publication":"publish_text","request":{"state":"The desk log says the crate has left storage; the warehouse inventory says it is in storage. Both reports are current, and the inventory controls conflicts under the protocol.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 110 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":110,"line_sha256":"a9d78891e36ebb920e80ede8c7fb8e78dbffa16049ddb4298545dff4041ba15c","licence":"MIT","semif_id":"0ebd299739302d8b057b","group_id":"d142f7fd0efa0eddc788","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":92,"slot":91,"row_ids_sha256":"5c07638f4088e7b48d8045c7b6660e3ff0fb95b1c810a232372569cf3617caec","oracle":{"probs":[0.43736484882972826,0.23321631076747587,0.3294188404027959],"argmax":"contradicted","top2_margin":0.10794600842693236,"near_tie":false,"group_logz":{"41":-0.8455810546875,"334":-7.974902153015137,"42":-1.4743824005126953,"378":-8.285425186157227,"43":-1.129018783569336,"340":-7.941165924072266}},"graph":{"mac_fp16":{"p":[0.43782917448904096,0.23303612578940439,0.3291346997215546],"max_abs_dp":0.0004643256593127032,"argmax_equal":true},"iphone_int8mlp":{"p":[0.4396890374185556,0.23274832891156672,0.3275626336698777],"max_abs_dp":0.002324188588827325,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.43736484882972826,0.23321631076747587,0.3294188404027959]],"input_tokens":92,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.43736484882972826,"probabilities":{"contradicted":0.43736484882972826,"insufficient":0.23321631076747587,"supported":0.3294188404027959}}},"usage":{"input_tokens":92,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3133}},{"id":"semif_eafc22c8c40df3932a8e","source":"semif","publication":"publish_text","request":{"state":"The protocol gives the inventory priority, but the current inventory and desk-log entries for the crate are omitted.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 111 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":111,"line_sha256":"75e8f28b7a1b0ae9f2b4347d08e6c3c9a68376450ec7dac560e80004ba6bd2a8","licence":"MIT","semif_id":"eafc22c8c40df3932a8e","group_id":"d142f7fd0efa0eddc788","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","contradicted","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":80,"slot":79,"row_ids_sha256":"05ab4304d9da6fa72af1783c0ce42176aaeb11c839a7415c5634ec1136c555de","oracle":{"probs":[0.04691419046654155,0.053631262248352726,0.8994545472851058],"argmax":"insufficient","top2_margin":0.8458232850367531,"near_tie":false,"group_logz":{"41":-3.0713119506835938,"334":-10.787970542907715,"42":-2.9375,"378":-10.197360038757324,"43":-0.1178436279296875,"340":-7.725437164306641}},"graph":{"mac_fp16":{"p":[0.046951507611546685,0.05368215525140148,0.8993663371370518],"max_abs_dp":8.821014805393546e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.04684000714673626,0.05274013719095992,0.9004198556623039],"max_abs_dp":0.0009653083771981041,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.04691419046654155,0.053631262248352726,0.8994545472851058]],"input_tokens":80,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8994545472851058,"probabilities":{"supported":0.04691419046654155,"contradicted":0.053631262248352726,"insufficient":0.8994545472851058}}},"usage":{"input_tokens":80,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3129}},{"id":"semif_0aa1447680e4f25862bb","source":"semif","publication":"publish_text","request":{"state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"answer":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 112 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":112,"line_sha256":"663d344f68cc3d1b1d93b2049cf605521821227f0f4e69f84ed910f38d6ffc2d","licence":"MIT","semif_id":"0aa1447680e4f25862bb","group_id":"3476d3f4f32290fe57c3","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"prohibited","row_len":103,"slot":102,"row_ids_sha256":"558d8185b4989a3e6cd48437ef0169ca2fc14e1f8c0f291d44dbc70172bde8cc","oracle":{"probs":[0.34101632886021804,0.4293904678492762,0.2295932032905058],"argmax":"prohibited","top2_margin":0.08837413898905816,"near_tie":false,"group_logz":{"41":-1.0932140350341797,"334":-7.99578857421875,"42":-0.8627777099609375,"378":-7.594734191894531,"43":-1.488835334777832,"340":-8.041593551635742}},"graph":{"mac_fp16":{"p":[0.34097097306960067,0.4287571450774988,0.23027188185290057],"max_abs_dp":0.0006786785623947578,"argmax_equal":true},"iphone_int8mlp":{"p":[0.34276550200828204,0.4270797519486802,0.23015474604303773],"max_abs_dp":0.0023107159005960054,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.34101632886021804,0.4293904678492762,0.2295932032905058]],"input_tokens":103,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.4293904678492762,"probabilities":{"insufficient":0.34101632886021804,"prohibited":0.4293904678492762,"permitted":0.2295932032905058}}},"usage":{"input_tokens":103,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3629}},{"id":"semif_980e76145c925747220c","source":"semif","publication":"publish_text","request":{"state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"answer":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; under this alternative protocol, the desk clerk overrides the ledger. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 113 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":113,"line_sha256":"4af81ff328aa5061c5a8ce8fbe728a292965828a2b94e13f1d220db3d244c8c9","licence":"MIT","semif_id":"980e76145c925747220c","group_id":"3476d3f4f32290fe57c3","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"permitted","row_len":104,"slot":103,"row_ids_sha256":"27a6aaffeb04940ccffd07fb17a6d508cfbdcac6b102183ea893dfd6b5f4b19b","oracle":{"probs":[0.38330960349993953,0.3530442437358038,0.2636461527642566],"argmax":"insufficient","top2_margin":0.030265359764135746,"near_tie":false,"group_logz":{"41":-0.9852619171142578,"334":-7.698137283325195,"42":-1.0675115585327148,"378":-7.566944122314453,"43":-1.3594970703125,"340":-7.737491607666016}},"graph":{"mac_fp16":{"p":[0.38311082122703793,0.3522396992531187,0.26464947951984336],"max_abs_dp":0.001003326755586731,"argmax_equal":true},"iphone_int8mlp":{"p":[0.38355565541706127,0.3452433572442066,0.2712009873387322],"max_abs_dp":0.007800886491597214,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.38330960349993953,0.3530442437358038,0.2636461527642566]],"input_tokens":104,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.38330960349993953,"probabilities":{"insufficient":0.38330960349993953,"prohibited":0.3530442437358038,"permitted":0.2636461527642566}}},"usage":{"input_tokens":104,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3606}},{"id":"semif_4aee37a6a155747766aa","source":"semif","publication":"publish_text","request":{"state":"The desk clerk reports the deposit unpaid, while the signed ledger reports it paid. Both reports are current.","questions":{"answer":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 114 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":114,"line_sha256":"1df9d665b797e8cc467e4860bc57e75e7db135844396f0d930021c218b6c122e","licence":"MIT","semif_id":"4aee37a6a155747766aa","group_id":"3476d3f4f32290fe57c3","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":101,"slot":100,"row_ids_sha256":"0982e87d041a622c33e1a94c8df402afd11c0695f67ff2d5f2c8ef799e9155a2","oracle":{"probs":[0.34901909938955433,0.4474635338527967,0.20351736675764912],"argmax":"prohibited","top2_margin":0.09844443446324236,"near_tie":false,"group_logz":{"41":-1.0723037719726562,"334":-7.87946891784668,"42":-0.8238353729248047,"378":-7.458995819091797,"43":-1.6116790771484375,"340":-8.057966232299805}},"graph":{"mac_fp16":{"p":[0.3481319816035348,0.4482367967664131,0.20363122163005207],"max_abs_dp":0.0008871177860195201,"argmax_equal":true},"iphone_int8mlp":{"p":[0.3440828974157148,0.45041251054386866,0.20550459204041646],"max_abs_dp":0.004936201973839538,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.34901909938955433,0.4474635338527967,0.20351736675764912]],"input_tokens":101,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.4474635338527967,"probabilities":{"permitted":0.34901909938955433,"prohibited":0.4474635338527967,"insufficient":0.20351736675764912}}},"usage":{"input_tokens":101,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3789}},{"id":"semif_6db17cc22bd656d78558","source":"semif","publication":"publish_text","request":{"state":"The record mentions the desk clerk and ledger but supplies neither deposit status.","questions":{"answer":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 115 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":115,"line_sha256":"4eed9d2417217f0e18e4a9fdb0ece6f6b28c85cdad8fbf6c8bbc717fe6812208","licence":"MIT","semif_id":"6db17cc22bd656d78558","group_id":"3476d3f4f32290fe57c3","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":94,"slot":93,"row_ids_sha256":"3eaeab785a7d97ca8701d4d6c7989eedf7386f77df3029b05fb96f0758a9344b","oracle":{"probs":[0.08495207607131035,0.056506585225498195,0.8585413387031914],"argmax":"insufficient","top2_margin":0.773589262631881,"near_tie":false,"group_logz":{"41":-2.478207588195801,"334":-9.908602714538574,"42":-2.8859376907348633,"378":-9.895598411560059,"43":-0.16506004333496094,"340":-7.556756019592285}},"graph":{"mac_fp16":{"p":[0.08479507185475607,0.056371837836352474,0.8588330903088914],"max_abs_dp":0.00029175160570005243,"argmax_equal":true},"iphone_int8mlp":{"p":[0.08500505273950973,0.05591756337521379,0.8590773838852765],"max_abs_dp":0.0005890218502844075,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.08495207607131035,0.056506585225498195,0.8585413387031914]],"input_tokens":94,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8585413387031914,"probabilities":{"prohibited":0.08495207607131035,"permitted":0.056506585225498195,"insufficient":0.8585413387031914}}},"usage":{"input_tokens":94,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3163}},{"id":"semif_ef3e0f8e657bf59f8a29","source":"semif","publication":"publish_text","request":{"state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"answer":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 116 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":116,"line_sha256":"e6e3d005323ea445c880706c2b7c483f8103e545dd7bd5863d542f151aabb700","licence":"MIT","semif_id":"ef3e0f8e657bf59f8a29","group_id":"c979db7416b206c1eb09","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","insufficient","B"],"codes":["A","B","C"],"gold":"B","row_len":97,"slot":96,"row_ids_sha256":"9e51a596c64c1631814ee968e340b61c339293f1a1b4ee69f2f974014aeeeea7","oracle":{"probs":[0.5931406805295101,0.28623222234270324,0.1206270971277867],"argmax":"A","top2_margin":0.30690845818680684,"near_tie":false,"group_logz":{"41":-0.5388431549072266,"334":-8.017668724060059,"42":-1.2674713134765625,"378":-8.381443977355957,"43":-2.131570816040039,"340":-9.218770980834961}},"graph":{"mac_fp16":{"p":[0.5938092468147607,0.2858045402792887,0.12038621290595068],"max_abs_dp":0.0006685662852505825,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5954034718110613,0.2835050237315586,0.1210915044573801],"max_abs_dp":0.002727198611144621,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.5931406805295101,0.28623222234270324,0.1206270971277867]],"input_tokens":97,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.5931406805295101,"probabilities":{"A":0.5931406805295101,"insufficient":0.28623222234270324,"B":0.1206270971277867}}},"usage":{"input_tokens":97,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3588}},{"id":"semif_f756e1cb04e11766341e","source":"semif","publication":"publish_text","request":{"state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"answer":{"type":"choice","instructions":"Under the alternative protocol, chat controls conflicts. Select the candidate ready according to chat.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 117 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":117,"line_sha256":"f0a758f87a1afcca4202f14b9184e7071208f6ff36e3e527f42896fffddbd8b2","licence":"MIT","semif_id":"f756e1cb04e11766341e","group_id":"c979db7416b206c1eb09","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","A","B"],"codes":["A","B","C"],"gold":"A","row_len":103,"slot":102,"row_ids_sha256":"8b24842d037d23914e8254a8c7f541df107703e0ee36bf594d3f055668ea1005","oracle":{"probs":[0.09975157244859856,0.14437702022203439,0.7558714073293671],"argmax":"B","top2_margin":0.6114943871073327,"near_tie":false,"group_logz":{"41":-2.3194007873535156,"334":-9.458216667175293,"42":-1.949655532836914,"378":-8.719223022460938,"43":-0.29421234130859375,"340":-7.342550277709961}},"graph":{"mac_fp16":{"p":[0.09949215884886234,0.1446067306795685,0.755901110471569],"max_abs_dp":0.000259413599736219,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0988738993947726,0.1442773789246789,0.7568487216805485],"max_abs_dp":0.0009773143511814242,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.09975157244859856,0.14437702022203439,0.7558714073293671]],"input_tokens":103,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.7558714073293671,"probabilities":{"insufficient":0.09975157244859856,"A":0.14437702022203439,"B":0.7558714073293671}}},"usage":{"input_tokens":103,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3628}},{"id":"semif_d167e51bb1f499099645","source":"semif","publication":"publish_text","request":{"state":"The signed checklist controls conflicts. Candidate A: The checklist says ready and chat says not ready. Candidate B: The checklist says not ready and chat says ready.","questions":{"answer":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 118 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":118,"line_sha256":"49a29277e3e839829dc1ecb8418663edbb291ac8002d1c0780e226d41bba02c5","licence":"MIT","semif_id":"d167e51bb1f499099645","group_id":"c979db7416b206c1eb09","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","B","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":83,"slot":82,"row_ids_sha256":"6f2855a340287b2646330c79e555e82bccf83a466208afe9c1f84e822674960e","oracle":{"probs":[0.2662250934795979,0.23754183467088785,0.49623307184951415],"argmax":"insufficient","top2_margin":0.23000797836991627,"near_tie":false,"group_logz":{"41":-1.3458900451660156,"334":-8.403592109680176,"42":-1.4598884582519531,"378":-8.183752059936523,"43":-0.7231864929199219,"340":-7.568159103393555}},"graph":{"mac_fp16":{"p":[0.2655598675878728,0.23624955610702877,0.49819057630509833],"max_abs_dp":0.0019575044555841803,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2594365234387829,0.2407383736031458,0.4998251029580713],"max_abs_dp":0.006788570040815012,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.2662250934795979,0.23754183467088785,0.49623307184951415]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.49623307184951415,"probabilities":{"A":0.2662250934795979,"B":0.23754183467088785,"insufficient":0.49623307184951415}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3103}},{"id":"semif_674ca4e9346216255466","source":"semif","publication":"publish_text","request":{"state":"The signed checklist controls conflicts, but Candidate A and Candidate B provide only project names without checklist or chat status.","questions":{"answer":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 119 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":119,"line_sha256":"1877cfb79eb571e504d769e4959804ebd0113011382ae0d3b27e477bf730ee1e","licence":"MIT","semif_id":"674ca4e9346216255466","group_id":"c979db7416b206c1eb09","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"insufficient","row_len":73,"slot":72,"row_ids_sha256":"4c419c120b1dd64613969ba06b653bd841ae3e559f6a6b29f0764214045c3e55","oracle":{"probs":[0.8648216624666657,0.0723288413009042,0.06284949623243019],"argmax":"insufficient","top2_margin":0.7924928211657615,"near_tie":false,"group_logz":{"41":-0.15842819213867188,"334":-8.393386840820312,"42":-2.639728546142578,"378":-10.239158630371094,"43":-2.7802085876464844,"340":-10.400901794433594}},"graph":{"mac_fp16":{"p":[0.8647135948645508,0.07238960142159578,0.06289680371385348],"max_abs_dp":0.00010806760211490296,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8666377624052677,0.07153646070439738,0.061825776890334985],"max_abs_dp":0.0018160999386019894,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8648216624666657,0.0723288413009042,0.06284949623243019]],"input_tokens":73,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.8648216624666657,"probabilities":{"insufficient":0.8648216624666657,"B":0.0723288413009042,"A":0.06284949623243019}}},"usage":{"input_tokens":73,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3045}},{"id":"semif_1df851a202994a07f123","source":"semif","publication":"publish_text","request":{"state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 120 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":120,"line_sha256":"5ae9cfe52b97a1b316faa13cf482bc8aae9db46079822741de886e04157b3a98","licence":"MIT","semif_id":"1df851a202994a07f123","group_id":"f7403e39d6796072491c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"contradicted","row_len":85,"slot":84,"row_ids_sha256":"0de817c18086a8dfd9fe6c00fd9a931f2672ec057930c8c30619e94263d717e2","oracle":{"probs":[0.964610340737576,0.022496560062365563,0.01289309920005844],"argmax":"contradicted","top2_margin":0.9421137806752105,"near_tie":false,"group_logz":{"41":-0.04071998596191406,"334":-8.702596664428711,"42":-3.799081802368164,"378":-11.810079574584961,"43":-4.355751991271973,"340":-12.450296401977539}},"graph":{"mac_fp16":{"p":[0.9646200358877469,0.022476301801941292,0.012903662310311866],"max_abs_dp":2.025826042427112e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9649076678515881,0.02226728661711953,0.012825045531292233],"max_abs_dp":0.00029732711401209144,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.964610340737576,0.022496560062365563,0.01289309920005844]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.964610340737576,"probabilities":{"contradicted":0.964610340737576,"insufficient":0.022496560062365563,"supported":0.01289309920005844}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3157}},{"id":"semif_b5e6788d95b8395d2888","source":"semif","publication":"publish_text","request":{"state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the source files were delivered.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 121 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":121,"line_sha256":"e9be942d75be4631aa06905bf1326d2b4de5c8ccf9d81587fbe67169144c74f5","licence":"MIT","semif_id":"b5e6788d95b8395d2888","group_id":"f7403e39d6796072491c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","contradicted","supported"],"codes":["A","B","C"],"gold":"supported","row_len":84,"slot":83,"row_ids_sha256":"d6708a6fa367aa1bd813c4e793268dbd37bac78034421ef66cba7bd02250041d","oracle":{"probs":[0.011370581475253522,0.014814809767343139,0.9738146087574033],"argmax":"supported","top2_margin":0.9589997989900602,"near_tie":false,"group_logz":{"41":-4.485023498535156,"334":-13.063350677490234,"42":-4.220425605773926,"378":-12.324193000793457,"43":-0.034832000732421875,"340":-8.535737991333008}},"graph":{"mac_fp16":{"p":[0.011348203805160547,0.014804590831495804,0.9738472053633437],"max_abs_dp":3.25966059403493e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.011330490751548404,0.014675584941865943,0.9739939243065856],"max_abs_dp":0.00017931554918226844,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.011370581475253522,0.014814809767343139,0.9738146087574033]],"input_tokens":84,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9738146087574033,"probabilities":{"insufficient":0.011370581475253522,"contradicted":0.014814809767343139,"supported":0.9738146087574033}}},"usage":{"input_tokens":84,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3242}},{"id":"semif_32f36cd0fbbfb3604586","source":"semif","publication":"publish_text","request":{"state":"The agreed handover requires the source files and user guide. The delivery record confirms that both arrived.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 122 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":122,"line_sha256":"41f9cc53a28c7a1bdcb503b448132898c3d541639442deba31e10980d021d985","licence":"MIT","semif_id":"32f36cd0fbbfb3604586","group_id":"f7403e39d6796072491c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":74,"slot":73,"row_ids_sha256":"1f32b44a2420915551f8a48818c4774b00e70196ccc146b37eed34c0aea4a971","oracle":{"probs":[0.9022868429519513,0.06778622623321745,0.029926930814831203],"argmax":"supported","top2_margin":0.8345006167187339,"near_tie":false,"group_logz":{"41":-0.11211395263671875,"334":-8.420111656188965,"42":-2.7006874084472656,"378":-10.467253684997559,"43":-3.5182876586914062,"340":-11.344382286071777}},"graph":{"mac_fp16":{"p":[0.9023692088820642,0.0677265489031765,0.029904242214759347],"max_abs_dp":8.236593011290427e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9029351699764201,0.06721809609079048,0.02984673393278934],"max_abs_dp":0.00064832702446882,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9022868429519513,0.06778622623321745,0.029926930814831203]],"input_tokens":74,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9022868429519513,"probabilities":{"supported":0.9022868429519513,"insufficient":0.06778622623321745,"contradicted":0.029926930814831203}}},"usage":{"input_tokens":74,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.308}},{"id":"semif_843185706adff0c60f8b","source":"semif","publication":"publish_text","request":{"state":"The agreement requires source files and a user guide, but the delivery record gives no information about either item.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 123 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":123,"line_sha256":"e5b92d781979799529c52a3a0207ce35c44e9b5da47ca44ac01a6baf87533ce1","licence":"MIT","semif_id":"843185706adff0c60f8b","group_id":"f7403e39d6796072491c","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","supported","insufficient"],"codes":["A","B","C"],"gold":"insufficient","row_len":75,"slot":74,"row_ids_sha256":"d2ccc13c3c8d0d782e33f0627ac79f9b35c30e517a3a6c2f5d341ac313f5e7f8","oracle":{"probs":[0.7671232656520918,0.01651618935437587,0.21636054499353233],"argmax":"contradicted","top2_margin":0.5507627206585595,"near_tie":false,"group_logz":{"41":-0.27431297302246094,"334":-8.004169464111328,"42":-4.112619400024414,"378":-11.505517959594727,"43":-1.5400142669677734,"340":-8.829673767089844}},"graph":{"mac_fp16":{"p":[0.7674932163519846,0.01651716378965365,0.21598961985836165],"max_abs_dp":0.0003709251351706744,"argmax_equal":true},"iphone_int8mlp":{"p":[0.7652069197159221,0.016406576253485808,0.21838650403059212],"max_abs_dp":0.0020259590370597935,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.7671232656520918,0.01651618935437587,0.21636054499353233]],"input_tokens":75,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.7671232656520918,"probabilities":{"contradicted":0.7671232656520918,"supported":0.01651618935437587,"insufficient":0.21636054499353233}}},"usage":{"input_tokens":75,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3079}},{"id":"semif_b531dd603bf080dab820","source":"semif","publication":"publish_text","request":{"state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"answer":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 124 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":124,"line_sha256":"502a957fdd6df3e2a31de0c66c4d9809bb0d3a2ee4cdd3f70b63a19d1c8fd872","licence":"MIT","semif_id":"b531dd603bf080dab820","group_id":"a060fd4621be415e9f1a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":91,"slot":90,"row_ids_sha256":"eb1f984a89da90383588a282f62d4e6e8da538bdc2d53eb50739888e69b43064","oracle":{"probs":[0.8388143902160234,0.023860596282338168,0.13732501350163837],"argmax":"prohibited","top2_margin":0.701489376714385,"near_tie":false,"group_logz":{"41":-0.18424606323242188,"334":-7.848862648010254,"42":-3.744007110595703,"378":-10.941751480102539,"43":-1.9938850402832031,"340":-9.15494441986084}},"graph":{"mac_fp16":{"p":[0.8394042083775493,0.023789738374842023,0.13680605324760867],"max_abs_dp":0.0005898181615258125,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8385075110449737,0.023739634970107823,0.13775285398491863],"max_abs_dp":0.0004278404832802618,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8388143902160234,0.023860596282338168,0.13732501350163837]],"input_tokens":91,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.8388143902160234,"probabilities":{"prohibited":0.8388143902160234,"permitted":0.023860596282338168,"insufficient":0.13732501350163837}}},"usage":{"input_tokens":91,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3187}},{"id":"semif_be2dd67bc733c1213bef","source":"semif","publication":"publish_text","request":{"state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"answer":{"type":"choice","instructions":"Approve the translation milestone when the manual has been supplied; the glossary is not required for this milestone. May this milestone be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 125 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":125,"line_sha256":"bd71f0a8e940463d081b4ad6b063dba70101621185e734e3c1d9170980f26f74","licence":"MIT","semif_id":"be2dd67bc733c1213bef","group_id":"a060fd4621be415e9f1a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":94,"slot":93,"row_ids_sha256":"461ccdc2ca1902325329fa0538618768e152af8d6373e18b6d72c2fcd90095e0","oracle":{"probs":[0.1305404296814107,0.6439743271430919,0.22548524317549748],"argmax":"permitted","top2_margin":0.41848908396759443,"near_tie":false,"group_logz":{"41":-2.0535221099853516,"334":-9.210585594177246,"42":-0.4575462341308594,"378":-7.488716125488281,"43":-1.5069503784179688,"340":-8.330883026123047}},"graph":{"mac_fp16":{"p":[0.13092710912169847,0.6435730753100151,0.22549981556828644],"max_abs_dp":0.0004012518330768211,"argmax_equal":true},"iphone_int8mlp":{"p":[0.13354397547885352,0.6370660309530914,0.22938999356805495],"max_abs_dp":0.00690829619000044,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.1305404296814107,0.6439743271430919,0.22548524317549748]],"input_tokens":94,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.6439743271430919,"probabilities":{"prohibited":0.1305404296814107,"permitted":0.6439743271430919,"insufficient":0.22548524317549748}}},"usage":{"input_tokens":94,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3169}},{"id":"semif_b8e2750307df9761e716","source":"semif","publication":"publish_text","request":{"state":"The contractor supplied both the translated manual and the glossary.","questions":{"answer":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 126 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":126,"line_sha256":"54cb789b7b1e1577718302be552b22db31fd71e109eeaddd007bd758cc046ccf","licence":"MIT","semif_id":"b8e2750307df9761e716","group_id":"a060fd4621be415e9f1a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","prohibited","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":88,"slot":87,"row_ids_sha256":"498de35dad9d2acded6f768118a578f04a4836928a5e02ba4db27fcbf684a64e","oracle":{"probs":[0.8004094759836032,0.06908418348097395,0.13050634053542284],"argmax":"permitted","top2_margin":0.6699031354481804,"near_tie":false,"group_logz":{"41":-0.23969650268554688,"334":-7.992399215698242,"42":-2.6894941329956055,"378":-9.986333847045898,"43":-2.0533981323242188,"340":-9.304887771606445}},"graph":{"mac_fp16":{"p":[0.8004600827967948,0.06912509117923807,0.13041482602396703],"max_abs_dp":9.151451145580913e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8039661331960638,0.06793108126312754,0.12810278554080853],"max_abs_dp":0.0035566572124605678,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8004094759836032,0.06908418348097395,0.13050634053542284]],"input_tokens":88,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8004094759836032,"probabilities":{"permitted":0.8004094759836032,"prohibited":0.06908418348097395,"insufficient":0.13050634053542284}}},"usage":{"input_tokens":88,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3101}},{"id":"semif_5d31b4bf2d0fc193f53b","source":"semif","publication":"publish_text","request":{"state":"The contractor reports working on the handover but supplies no delivery record for either item.","questions":{"answer":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 127 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":127,"line_sha256":"b8c0bfba132eefb32bc616bbf2cef79af3286d7ff4d49d6c40bd1dfc5e831a01","licence":"MIT","semif_id":"5d31b4bf2d0fc193f53b","group_id":"a060fd4621be415e9f1a","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["permitted","insufficient","prohibited"],"codes":["A","B","C"],"gold":"insufficient","row_len":93,"slot":92,"row_ids_sha256":"f9b60ad493ddc524463b58a3ed53b95086da3144b68f2f82cdd20e6a48879fe2","oracle":{"probs":[0.08468293341093695,0.3130934506618234,0.6022236159272396],"argmax":"prohibited","top2_margin":0.2891301652654162,"near_tie":false,"group_logz":{"41":-2.489386558532715,"334":-9.560014724731445,"42":-1.1817989349365234,"378":-7.925742149353027,"43":-0.5276718139648438,"340":-7.403440475463867}},"graph":{"mac_fp16":{"p":[0.08480483602286902,0.31292157260043224,0.6022735913766988],"max_abs_dp":0.0001718780613911841,"argmax_equal":true},"iphone_int8mlp":{"p":[0.08344964301644926,0.3107303505795765,0.6058200064039742],"max_abs_dp":0.0035963904767345456,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.08468293341093695,0.3130934506618234,0.6022236159272396]],"input_tokens":93,"response":{"answers":{"answer":{"type":"choice","choice":"prohibited","confidence":0.6022236159272396,"probabilities":{"permitted":0.08468293341093695,"insufficient":0.3130934506618234,"prohibited":0.6022236159272396}}},"usage":{"input_tokens":93,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3152}},{"id":"semif_a9230edfc6ec9e524af0","source":"semif","publication":"publish_text","request":{"state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"answer":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 128 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":128,"line_sha256":"cfb4f0a4ebc960e89a40dab1b5fe85ec1a5c7d1658eb6d6327a5adfc8a86d15c","licence":"MIT","semif_id":"a9230edfc6ec9e524af0","group_id":"2b0f8eda8903e38a47de","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","insufficient","A"],"codes":["A","B","C"],"gold":"A","row_len":82,"slot":81,"row_ids_sha256":"17c30d786c220d1e8e83d8b9e0d0a3d9fb05f2ed7e28040e7d00769da20acac3","oracle":{"probs":[0.017469222471774605,0.019974942003043887,0.9625558355251814],"argmax":"A","top2_margin":0.9425808935221376,"near_tie":false,"group_logz":{"41":-4.054391860961914,"334":-12.181358337402344,"42":-3.920353889465332,"378":-11.54636001586914,"43":-0.04524040222167969,"340":-8.262207984924316}},"graph":{"mac_fp16":{"p":[0.017466360785894936,0.01998672692918398,0.9625469122849212],"max_abs_dp":1.1784926140092494e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.017087689423445424,0.01959290442832467,0.9633194061482299],"max_abs_dp":0.0007635706230484374,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.017469222471774605,0.019974942003043887,0.9625558355251814]],"input_tokens":82,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9625558355251814,"probabilities":{"B":0.017469222471774605,"insufficient":0.019974942003043887,"A":0.9625558355251814}}},"usage":{"input_tokens":82,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3053}},{"id":"semif_c100e84585149a04be49","source":"semif","publication":"publish_text","request":{"state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"answer":{"type":"choice","instructions":"Select the package containing the timeline but missing the remedy report.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 129 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":129,"line_sha256":"bb70f424172e0290c15b8a9d69ac3e01045e3979dcb222c597afefd5791b79d5","licence":"MIT","semif_id":"c100e84585149a04be49","group_id":"2b0f8eda8903e38a47de","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"B","row_len":88,"slot":87,"row_ids_sha256":"e5ebfca266793b10bf5e9fed87adb38af95310b48dfca35fead742caa2fc27bf","oracle":{"probs":[0.9634784512303551,0.0177862139079499,0.018735334861694916],"argmax":"B","top2_margin":0.9447431163686603,"near_tie":false,"group_logz":{"41":-0.04388427734375,"334":-8.809532165527344,"42":-4.0360107421875,"378":-12.324621200561523,"43":-3.984023094177246,"340":-12.280854225158691}},"graph":{"mac_fp16":{"p":[0.9634640426399043,0.017785534145388977,0.018750423214706848],"max_abs_dp":1.5088353011932093e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9638292491154273,0.017665476888007403,0.018505273996565177],"max_abs_dp":0.000350797885072196,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9634784512303551,0.0177862139079499,0.018735334861694916]],"input_tokens":88,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.9634784512303551,"probabilities":{"B":0.9634784512303551,"A":0.0177862139079499,"insufficient":0.018735334861694916}}},"usage":{"input_tokens":88,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3063}},{"id":"semif_579beed184ffc1104aa9","source":"semif","publication":"publish_text","request":{"state":"A complete package requires both documents. Candidate A: The timeline is attached but the remedy report is absent. Candidate B: Both the timeline and remedy report are attached.","questions":{"answer":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 130 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":130,"line_sha256":"214117aff4ea34fdb07a68340eee580b04b114d74590c6ced3eb9a1065aec442","licence":"MIT","semif_id":"579beed184ffc1104aa9","group_id":"2b0f8eda8903e38a47de","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"B","row_len":79,"slot":78,"row_ids_sha256":"70f7bfcfb5cc7fc37794337e7bb0e3e4063b576854a4f98925985760d2625022","oracle":{"probs":[0.8865838122454425,0.059739742673379365,0.053676445081178165],"argmax":"B","top2_margin":0.8268440695720631,"near_tie":false,"group_logz":{"41":-0.12815284729003906,"334":-8.155218124389648,"42":-2.825531005859375,"378":-10.400457382202148,"43":-2.932554244995117,"340":-10.53062629699707}},"graph":{"mac_fp16":{"p":[0.8867333780371982,0.05973543132789252,0.053531190634909256],"max_abs_dp":0.0001495657917557347,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8848091531116117,0.060714015412125025,0.054476831476263206],"max_abs_dp":0.0017746591338307915,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.8865838122454425,0.059739742673379365,0.053676445081178165]],"input_tokens":79,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8865838122454425,"probabilities":{"B":0.8865838122454425,"A":0.059739742673379365,"insufficient":0.053676445081178165}}},"usage":{"input_tokens":79,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3104}},{"id":"semif_c4e9bd463b6fde5be743","source":"semif","publication":"publish_text","request":{"state":"A complete package requires both documents. Candidate A and Candidate B give only package titles and no attachment information.","questions":{"answer":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 131 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":131,"line_sha256":"6ca1b841866f92dc39e924111bf359b9d3c6974ddd92be25b2d0a37da7b857c2","licence":"MIT","semif_id":"c4e9bd463b6fde5be743","group_id":"2b0f8eda8903e38a47de","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","insufficient","A"],"codes":["A","B","C"],"gold":"insufficient","row_len":67,"slot":66,"row_ids_sha256":"37d08d3d0bd8302b056a68f24d045006ec87746c2339c4d44b87203ab418b2ea","oracle":{"probs":[0.02428345371456119,0.9530891950605707,0.022627351224868253],"argmax":"insufficient","top2_margin":0.9288057413460095,"near_tie":false,"group_logz":{"41":-3.725834846496582,"334":-11.802366256713867,"42":-0.05592155456542969,"378":-7.938413619995117,"43":-3.7964706420898438,"340":-11.491016387939453}},"graph":{"mac_fp16":{"p":[0.02428133552427072,0.9530888173491308,0.022629847126598393],"max_abs_dp":2.4959017301401376e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.024040923014318608,0.9535630734820874,0.022396003503594088],"max_abs_dp":0.00047387842151669357,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.02428345371456119,0.9530891950605707,0.022627351224868253]],"input_tokens":67,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9530891950605707,"probabilities":{"B":0.02428345371456119,"insufficient":0.9530891950605707,"A":0.022627351224868253}}},"usage":{"input_tokens":67,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3043}},{"id":"semif_b863998d4be2721a5dfd","source":"semif","publication":"publish_text","request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"contradicted"},"note":"SemIf authored144 line 132 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":132,"line_sha256":"bae89fc634702f230c942773ef3b0d1f8ed6ae8940147ad73d34cb55938e0079","licence":"MIT","semif_id":"b863998d4be2721a5dfd","group_id":"01020a1eaf46ec2fba5d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["supported","insufficient","contradicted"],"codes":["A","B","C"],"gold":"contradicted","row_len":96,"slot":95,"row_ids_sha256":"78a6ab3e51939bd9d01cebe8259fab4db0538ecd87bcfafc39d8383123f9f1c4","oracle":{"probs":[0.020354531916021736,0.25180977834385204,0.7278356897401262],"argmax":"contradicted","top2_margin":0.4760259113962742,"near_tie":false,"group_logz":{"41":-3.901238441467285,"334":-11.728378295898438,"42":-1.3858680725097656,"378":-8.740592002868652,"43":-0.3244667053222656,"340":-7.926717758178711}},"graph":{"mac_fp16":{"p":[0.0203151064845429,0.25122751738151133,0.7284573761339458],"max_abs_dp":0.0006216863938195427,"argmax_equal":true},"iphone_int8mlp":{"p":[0.020076861854213653,0.24329135850171751,0.7366317796440688],"max_abs_dp":0.008796089903942539,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.020354531916021736,0.25180977834385204,0.7278356897401262]],"input_tokens":96,"response":{"answers":{"answer":{"type":"choice","choice":"contradicted","confidence":0.7278356897401262,"probabilities":{"supported":0.020354531916021736,"insufficient":0.25180977834385204,"contradicted":0.7278356897401262}}},"usage":{"input_tokens":96,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3284}},{"id":"semif_37cebc3792e98049439f","source":"semif","publication":"publish_text","request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the outside workshop can perform this repair.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 133 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":133,"line_sha256":"7c2295b4f5f42981dc26faa73d1b81c448f5fd35781fc9653e6743f0fa486fc8","licence":"MIT","semif_id":"37cebc3792e98049439f","group_id":"01020a1eaf46ec2fba5d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","supported","contradicted"],"codes":["A","B","C"],"gold":"supported","row_len":93,"slot":92,"row_ids_sha256":"9db52800d8c4d2d1e6bf731893a7458e0fc9df64d63165986ff614a38dafdaae","oracle":{"probs":[0.3505616222740121,0.5845349438591283,0.06490343386685957],"argmax":"supported","top2_margin":0.2339733215851162,"near_tie":false,"group_logz":{"41":-1.064199447631836,"334":-8.667450904846191,"42":-0.5529193878173828,"378":-7.921627044677734,"43":-2.750835418701172,"340":-10.099922180175781}},"graph":{"mac_fp16":{"p":[0.3508921002576725,0.584095374019354,0.0650125257229736],"max_abs_dp":0.00043956983977433417,"argmax_equal":true},"iphone_int8mlp":{"p":[0.34686616075868393,0.5885714042796644,0.0645624349616517],"max_abs_dp":0.00403646042053607,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.3505616222740121,0.5845349438591283,0.06490343386685957]],"input_tokens":93,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.5845349438591283,"probabilities":{"insufficient":0.3505616222740121,"supported":0.5845349438591283,"contradicted":0.06490343386685957}}},"usage":{"input_tokens":93,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3159}},{"id":"semif_64d63f86e328aff1481c","source":"semif","publication":"publish_text","request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. The in-house workshop cannot perform this repair; the outside workshop can.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"supported"},"note":"SemIf authored144 line 134 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":134,"line_sha256":"d05f8d4c70bfccc253e82b18d18196370a1116b551a9494130a6c2c67fb20d91","licence":"MIT","semif_id":"64d63f86e328aff1481c","group_id":"01020a1eaf46ec2fba5d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["contradicted","insufficient","supported"],"codes":["A","B","C"],"gold":"supported","row_len":101,"slot":100,"row_ids_sha256":"617c32e29adeb25959d1ab721e31e73a617d15bc8853ca5d8935ae6c94b6e095","oracle":{"probs":[0.015444266578001823,0.013674395663911782,0.9708813377580864],"argmax":"supported","top2_margin":0.9554370711800846,"near_tie":false,"group_logz":{"41":-4.178182601928711,"334":-12.428504943847656,"42":-4.299895286560059,"378":-12.091631889343262,"43":-0.0372161865234375,"340":-8.19659423828125}},"graph":{"mac_fp16":{"p":[0.015487462362412985,0.013692638857247263,0.9708198987803398],"max_abs_dp":6.143897774657514e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.015155164494588726,0.013448991778649265,0.9713958437267619],"max_abs_dp":0.0005145059686755271,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.015444266578001823,0.013674395663911782,0.9708813377580864]],"input_tokens":101,"response":{"answers":{"answer":{"type":"choice","choice":"supported","confidence":0.9708813377580864,"probabilities":{"contradicted":0.015444266578001823,"insufficient":0.013674395663911782,"supported":0.9708813377580864}}},"usage":{"input_tokens":101,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3601}},{"id":"semif_6d64a8459f64eac7f4cc","source":"semif","publication":"publish_text","request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Neither workshop's ability to perform this repair is supplied.","questions":{"answer":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 135 (evidence_interpretation)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":135,"line_sha256":"39c08980690a5cd793c933f91c208b7151b2184a0ad54ceea9d92e41a3973188","licence":"MIT","semif_id":"6d64a8459f64eac7f4cc","group_id":"01020a1eaf46ec2fba5d","family":"evidence_interpretation","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","contradicted","supported"],"codes":["A","B","C"],"gold":"insufficient","row_len":98,"slot":97,"row_ids_sha256":"8b9de6cc566d9847d11662ef898cbdda0d48984195ca58774b8a14a8167403a3","oracle":{"probs":[0.9318488227307563,0.04418361032760344,0.023967566941640324],"argmax":"insufficient","top2_margin":0.8876652124031529,"near_tie":false,"group_logz":{"41":-0.07676124572753906,"334":-8.626999855041504,"42":-3.125577926635742,"378":-11.163851737976074,"43":-3.7372303009033203,"340":-11.79597282409668}},"graph":{"mac_fp16":{"p":[0.9315653844778352,0.04439939010374484,0.02403522541841992],"max_abs_dp":0.0002834382529210977,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9323234428752698,0.04389087072770938,0.02378568639702077],"max_abs_dp":0.00047462014451349344,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9318488227307563,0.04418361032760344,0.023967566941640324]],"input_tokens":98,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9318488227307563,"probabilities":{"insufficient":0.9318488227307563,"contradicted":0.04418361032760344,"supported":0.023967566941640324}}},"usage":{"input_tokens":98,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3613}},{"id":"semif_2708193212d8a4a523c7","source":"semif","publication":"publish_text","request":{"state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"answer":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"prohibited"},"note":"SemIf authored144 line 136 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":136,"line_sha256":"ad0ef7c8d3399bcd03f5dff1f65e2dbe800608405af114698003bb1779e3b594","licence":"MIT","semif_id":"2708193212d8a4a523c7","group_id":"6a06a2f278301c1412ee","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"prohibited","row_len":102,"slot":101,"row_ids_sha256":"068ce59d96e28008e33cbb27df417fdfae40d7bef400f42a1762635a8992704c","oracle":{"probs":[0.03325592366996387,0.8717640505152631,0.09498002581477309],"argmax":"permitted","top2_margin":0.77678402470049,"near_tie":false,"group_logz":{"41":-3.418667793273926,"334":-10.712220191955566,"42":-0.15238189697265625,"378":-7.216974258422852,"43":-2.369234085083008,"340":-9.19729995727539}},"graph":{"mac_fp16":{"p":[0.033300278837830835,0.8713370562239717,0.09536266493819742],"max_abs_dp":0.00042699429129133826,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03260310664027562,0.8744505731928499,0.09294632016687457],"max_abs_dp":0.0026865226775868445,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.03325592366996387,0.8717640505152631,0.09498002581477309]],"input_tokens":102,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8717640505152631,"probabilities":{"prohibited":0.03325592366996387,"permitted":0.8717640505152631,"insufficient":0.09498002581477309}}},"usage":{"input_tokens":102,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3635}},{"id":"semif_5a37d1dfcad20810a76a","source":"semif","publication":"publish_text","request":{"state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"answer":{"type":"choice","instructions":"Under the alternative policy, either suitable available venue may be booked without priority. May the school be booked?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 137 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":137,"line_sha256":"93feae55494ec8b724b53fea0f53ae766f729a277a2d981aa70b2d50aede3511","licence":"MIT","semif_id":"5a37d1dfcad20810a76a","group_id":"6a06a2f278301c1412ee","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["prohibited","permitted","insufficient"],"codes":["A","B","C"],"gold":"permitted","row_len":90,"slot":89,"row_ids_sha256":"ab73c0e7721484a40a90a31a33810191f0bdd3e16ea7dea2c4fb14d421692f30","oracle":{"probs":[0.019156349539665184,0.8882664251094395,0.09257722535089523],"argmax":"permitted","top2_margin":0.7956891997585444,"near_tie":false,"group_logz":{"41":-3.9690513610839844,"334":-11.368093490600586,"42":-0.1324138641357422,"378":-7.186260223388672,"43":-2.3936424255371094,"340":-9.204289436340332}},"graph":{"mac_fp16":{"p":[0.019158348341024634,0.8881880570395166,0.09265359461945875],"max_abs_dp":7.836806992289613e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.01932615158123637,0.8873855154567107,0.09328833296205295],"max_abs_dp":0.0008809096527288007,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.019156349539665184,0.8882664251094395,0.09257722535089523]],"input_tokens":90,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8882664251094395,"probabilities":{"prohibited":0.019156349539665184,"permitted":0.8882664251094395,"insufficient":0.09257722535089523}}},"usage":{"input_tokens":90,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3144}},{"id":"semif_acacd8b46acd25cb954b","source":"semif","publication":"publish_text","request":{"state":"The town hall is confirmed unavailable, while the school is available and suitable.","questions":{"answer":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"answer":"permitted"},"note":"SemIf authored144 line 138 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":138,"line_sha256":"2b02d4570f70e00cb67db4f8a6ae1fef2024fef2b22a46f0df44b9889685c033","licence":"MIT","semif_id":"acacd8b46acd25cb954b","group_id":"6a06a2f278301c1412ee","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","permitted","prohibited"],"codes":["A","B","C"],"gold":"permitted","row_len":101,"slot":100,"row_ids_sha256":"f757edeafc566154eab51672a0451587257b23dcf0d22cfcf5f395631853ce3b","oracle":{"probs":[0.09970086064694708,0.8738799830748507,0.02641915627820214],"argmax":"permitted","top2_margin":0.7741791224279037,"near_tie":false,"group_logz":{"41":-2.3177852630615234,"334":-9.687080383300781,"42":-0.1470165252685547,"378":-7.495460510253906,"43":-3.6458702087402344,"340":-10.8614501953125}},"graph":{"mac_fp16":{"p":[0.09966377749163043,0.8738756268545359,0.02646059565383375],"max_abs_dp":4.1439375631609904e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.10003979286459219,0.8737444623449979,0.026215744790409803],"max_abs_dp":0.0003389322176451104,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.09970086064694708,0.8738799830748507,0.02641915627820214]],"input_tokens":101,"response":{"answers":{"answer":{"type":"choice","choice":"permitted","confidence":0.8738799830748507,"probabilities":{"insufficient":0.09970086064694708,"permitted":0.8738799830748507,"prohibited":0.02641915627820214}}},"usage":{"input_tokens":101,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3597}},{"id":"semif_4089c199b8b796ec3ed8","source":"semif","publication":"publish_text","request":{"state":"The two venues are named, but neither availability nor suitability is provided.","questions":{"answer":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 139 (rule_application)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":139,"line_sha256":"24f452426c2249ee67e1791b97b0b4330d2f3b94cdfbc91447a9a04c9289e914","licence":"MIT","semif_id":"4089c199b8b796ec3ed8","group_id":"6a06a2f278301c1412ee","family":"rule_application","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","prohibited","permitted"],"codes":["A","B","C"],"gold":"insufficient","row_len":101,"slot":100,"row_ids_sha256":"1d2ac3595c80edebf3655a1f8224a36932404747190cfd38db15c216c85b3548","oracle":{"probs":[0.6073513369218767,0.10518572641206249,0.28746293666606076],"argmax":"insufficient","top2_margin":0.31988840025581594,"near_tie":false,"group_logz":{"41":-0.5182781219482422,"334":-7.969554901123047,"42":-2.271657943725586,"378":-9.30770492553711,"43":-1.266291618347168,"340":-8.296093940734863}},"graph":{"mac_fp16":{"p":[0.607418284005418,0.10512365282273405,0.28745806317184797],"max_abs_dp":6.694708354126888e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.6059493438194635,0.10496601349178632,0.28908464268875017],"max_abs_dp":0.0016217060226894064,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.6073513369218767,0.10518572641206249,0.28746293666606076]],"input_tokens":101,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.6073513369218767,"probabilities":{"insufficient":0.6073513369218767,"prohibited":0.10518572641206249,"permitted":0.28746293666606076}}},"usage":{"input_tokens":101,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3658}},{"id":"semif_02f2b95a5c5a4777a45c","source":"semif","publication":"publish_text","request":{"state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"answer":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"answer":"A"},"note":"SemIf authored144 line 140 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":140,"line_sha256":"c0d13120fc1e3b4366104c2b23058225cbc636563e4e65ed9b52a5c358379e96","licence":"MIT","semif_id":"02f2b95a5c5a4777a45c","group_id":"2797f8c23d05c0ee06b3","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["B","A","insufficient"],"codes":["A","B","C"],"gold":"A","row_len":89,"slot":88,"row_ids_sha256":"e3e402c6713f8cd69874cda3d5acbd8e2a8aeaa95622e67d7234117b5c29833c","oracle":{"probs":[0.016126005745112602,0.9645150903613053,0.019358903893582045],"argmax":"A","top2_margin":0.9451561864677233,"near_tie":false,"group_logz":{"41":-4.136438369750977,"334":-12.239225387573242,"42":-0.045246124267578125,"378":-8.086650848388672,"43":-3.953719139099121,"340":-11.804986953735352}},"graph":{"mac_fp16":{"p":[0.016149603071865117,0.9644240597943853,0.0194263371337496],"max_abs_dp":9.10305669200584e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.016150273284472244,0.9643934197322253,0.019456306983302446],"max_abs_dp":0.00012167062908008397,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.016126005745112602,0.9645150903613053,0.019358903893582045]],"input_tokens":89,"response":{"answers":{"answer":{"type":"choice","choice":"A","confidence":0.9645150903613053,"probabilities":{"B":0.016126005745112602,"A":0.9645150903613053,"insufficient":0.019358903893582045}}},"usage":{"input_tokens":89,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3147}},{"id":"semif_ebea4681e3dcc7d92ca2","source":"semif","publication":"publish_text","request":{"state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"answer":{"type":"choice","instructions":"Under the alternative priority, choose B whenever B is able; choose A only when B is unable and A is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 141 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":141,"line_sha256":"f72dabc70f174d755bfc975fe608bb7daa17d7ba8bd97177a3e6ee81c8d0dbe0","licence":"MIT","semif_id":"ebea4681e3dcc7d92ca2","group_id":"2797f8c23d05c0ee06b3","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"B","row_len":94,"slot":93,"row_ids_sha256":"0e78b11143887b37b7007a7db9ca8f51a6630faedcbc585fff5796f4818be204","oracle":{"probs":[0.06295990678913096,0.7790107886833995,0.15802930452746952],"argmax":"B","top2_margin":0.6209814841559299,"near_tie":false,"group_logz":{"41":-2.7782907485961914,"334":-10.276080131530762,"42":-0.26276397705078125,"378":-7.477542877197266,"43":-1.8580083847045898,"340":-8.980722427368164}},"graph":{"mac_fp16":{"p":[0.06293667561614946,0.7790264438372769,0.15803688054657364],"max_abs_dp":2.323117298150268e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.062314041399558415,0.7775632076205974,0.1601227509798442],"max_abs_dp":0.0020934464523746643,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.06295990678913096,0.7790107886833995,0.15802930452746952]],"input_tokens":94,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.7790107886833995,"probabilities":{"insufficient":0.06295990678913096,"B":0.7790107886833995,"A":0.15802930452746952}}},"usage":{"input_tokens":94,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.316}},{"id":"semif_b8c5b9b285cbdddc9f8d","source":"semif","publication":"publish_text","request":{"state":"Candidate A is the preferred translator but is confirmed unable to handle this language. Candidate B is the fallback and is confirmed able.","questions":{"answer":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"answer":"B"},"note":"SemIf authored144 line 142 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":142,"line_sha256":"2efe29ffb7db399f3bba4bbd7fd8ae18c674619aa9dbb29a38bc7cf349eb843e","licence":"MIT","semif_id":"b8c5b9b285cbdddc9f8d","group_id":"2797f8c23d05c0ee06b3","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["A","insufficient","B"],"codes":["A","B","C"],"gold":"B","row_len":85,"slot":84,"row_ids_sha256":"3403e48b7c93fe2fe86d274a67cebbc7500cf624fa79f9b3e0bf2e40bfb4fde7","oracle":{"probs":[0.01756060517360157,0.17623578612101382,0.8062036087053845],"argmax":"B","top2_margin":0.6299678225843708,"near_tie":false,"group_logz":{"41":-4.054194450378418,"334":-12.006391525268555,"42":-1.7480297088623047,"378":-9.12806224822998,"43":-0.22751617431640625,"340":-7.90085506439209}},"graph":{"mac_fp16":{"p":[0.017578913899399296,0.17645923289149984,0.8059618532091009],"max_abs_dp":0.00024175549628369097,"argmax_equal":true},"iphone_int8mlp":{"p":[0.017553634062808143,0.1788389317184579,0.8036074342187338],"max_abs_dp":0.0026031455974440953,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.01756060517360157,0.17623578612101382,0.8062036087053845]],"input_tokens":85,"response":{"answers":{"answer":{"type":"choice","choice":"B","confidence":0.8062036087053845,"probabilities":{"A":0.01756060517360157,"insufficient":0.17623578612101382,"B":0.8062036087053845}}},"usage":{"input_tokens":85,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3137}},{"id":"semif_7ed75d854d99b3a81916","source":"semif","publication":"publish_text","request":{"state":"Candidate A is preferred and Candidate B is the fallback, but neither candidate's ability to handle the language is supplied.","questions":{"answer":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"answer":"insufficient"},"note":"SemIf authored144 line 143 (candidate_selection)","provenance":{"file":"authored144.jsonl","repo":"TheoLeeCJ/SemIf","pin":"ca3ba65f","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","line":143,"line_sha256":"7e26c499b875039865b9a22e2c91b69239dee58b604b45d436ecda9f483b57d2","licence":"MIT","semif_id":"7ed75d854d99b3a81916","group_id":"2797f8c23d05c0ee06b3","family":"candidate_selection","split":"test"},"questions":[{"name":"answer","type":"choice","n_options":3,"keys":["insufficient","B","A"],"codes":["A","B","C"],"gold":"insufficient","row_len":83,"slot":82,"row_ids_sha256":"2de24249bd5c3652aea7203ebe7df1aa91a14040bdc86d8a2b11d4e09de1b082","oracle":{"probs":[0.9315813301991276,0.031438350990198836,0.036980318810673526],"argmax":"insufficient","top2_margin":0.894601011388454,"near_tie":false,"group_logz":{"41":-0.07876396179199219,"334":-8.767999649047852,"42":-3.467618942260742,"378":-11.522335052490234,"43":-3.3052616119384766,"340":-11.361117362976074}},"graph":{"mac_fp16":{"p":[0.9314843906849187,0.03145430190762186,0.037061307407459436],"max_abs_dp":9.693951420886027e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.931763140846545,0.03143821412702442,0.03679864502643061],"max_abs_dp":0.00018181064741740194,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.9315813301991276,0.031438350990198836,0.036980318810673526]],"input_tokens":83,"response":{"answers":{"answer":{"type":"choice","choice":"insufficient","confidence":0.9315813301991276,"probabilities":{"insufficient":0.9315813301991276,"B":0.031438350990198836,"A":0.036980318810673526}}},"usage":{"input_tokens":83,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.3121}},{"id":"own_ticket_01","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"f2733225cb91b2444528960167c98912735129842c717bd357649c522ebf5eb4","questions":[{"type":"choice","n_options":4,"row_len":161,"slot":160,"row_ids_sha256":"a8537f49d850e9cadc573f47f3e000559ad0c2460af3e8c7a7aa9d185f0f78af","oracle":{"probs":[0.9848151576759457,0.0043847392484382035,0.006090766637499786,0.004709336438116374],"argmax":"billing","top2_margin":0.978724391038446,"near_tie":false,"group_logz":{"41":-0.02172088623046875,"334":-8.897014617919922,"42":-5.436044692993164,"378":-14.054023742675781,"43":-5.107400894165039,"340":-13.612373352050781,"44":-5.364627838134766,"388":-13.850370407104492}},"graph":{"mac_fp16":{"p":[0.9848403647340807,0.004378834743562797,0.006078295929749095,0.004702504592607514],"max_abs_dp":2.5207058134912863e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9849211695161856,0.004359099019594032,0.006034782358075691,0.004684949106144705],"max_abs_dp":0.00010601184023983912,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":149,"slot":148,"row_ids_sha256":"a8744585b2b1d2d945493b6c14d2e2ecced844396417fb2cedaec797ab7d12ac","oracle":{"probs":[0.9927062284157491,0.007293771584250952],"argmax":"yes","top2_margin":0.9854124568314981,"near_tie":false,"group_logz":{"11683":-0.009202957153320312,"12447":-7.183465003967285,"2243":-4.922616958618164,"4547":-11.320653915405273,"19598":-13.001557350158691}},"graph":{"mac_fp16":{"p":[0.9927153236489847,0.007284676351015295],"max_abs_dp":9.095233235667521e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9927067367475413,0.007293263252458848],"max_abs_dp":5.083317922149888e-07,"argmax_equal":true}}},{"type":"score","n_options":3,"row_len":138,"slot":137,"row_ids_sha256":"25d7574baed0da28da49dd2bc05a56a99d3b97d50b5767e7a423bce2198facf6","oracle":{"probs":[0.989631638529771,0.007803443230845146,0.0025649182393838992],"argmax":"0","top2_margin":0.9818281952989258,"near_tie":false,"group_logz":{"24":-0.019777297973632812,"25":-4.862545013427734,"26":-5.975183486938477}},"graph":{"mac_fp16":{"p":[0.989618104027715,0.007811462673011589,0.0025704332992732963],"max_abs_dp":1.3534502055989606e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9898578597123825,0.007612835252342446,0.0025293050352750857],"max_abs_dp":0.00022622118261150792,"argmax_equal":true}}}]},{"id":"own_incident_02","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"d9ece826cfeb2a57f4cc0a0c234fb349c5c07dc103104a2f0c59010fcf7c2498","questions":[{"type":"score","n_options":5,"row_len":201,"slot":200,"row_ids_sha256":"98081af1203e00b1fb61a2f5da440fb50ac4269c5c9c256fd9d5d45c1d17837f","oracle":{"probs":[0.007149834071109108,0.013766008064640256,0.9535917687322017,0.020929089909319007,0.004563299222729962],"argmax":"2","top2_margin":0.9326626788228828,"near_tie":false,"group_logz":{"24":-4.95213508605957,"25":-4.297021865844727,"26":-0.05898857116699219,"27":-3.878084182739258,"28":-5.401178359985352}},"graph":{"mac_fp16":{"p":[0.007160239002149977,0.013794711150876996,0.953526760497784,0.020958128021037116,0.004560161328152097],"max_abs_dp":6.500823441779602e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.007129696334645113,0.01375040052851166,0.9540454411607502,0.020560200964927187,0.004514261011165883],"max_abs_dp":0.00045367242854843504,"argmax_equal":true}}},{"type":"choice","n_options":5,"row_len":193,"slot":192,"row_ids_sha256":"c5cb896bbfaa25f83e46304bcf80ff4355593a3c64bf2ed850b9feda6ac130a0","oracle":{"probs":[0.9515415343967192,0.009457432957153449,0.006313264441027559,0.004186556908397536,0.028501211296702345],"argmax":"power","top2_margin":0.9230403231000168,"near_tie":false,"group_logz":{"41":-0.05760765075683594,"334":-8.84830093383789,"42":-4.668889999389648,"378":-13.14930534362793,"43":-5.073038101196289,"340":-13.509416580200195,"44":-5.48381233215332,"388":-14.0985107421875,"45":-3.565744400024414,"407":-11.827253341674805}},"graph":{"mac_fp16":{"p":[0.95163864613064,0.009454782281800071,0.006307428121457532,0.004183343643976968,0.02841579982212535],"max_abs_dp":9.711173392079608e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.952602584376141,0.00933554919752003,0.006232605676461755,0.004131341959700536,0.02769791879017659],"max_abs_dp":0.001061049979421802,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":155,"slot":154,"row_ids_sha256":"eb6ba78fa2ecbd51d71957e293c61cf5e5b98e5f1bd6a845b6ca33e25a9ab273","oracle":{"probs":[0.9847892525595364,0.015210747440463638],"argmax":"yes","top2_margin":0.9695785051190727,"near_tie":false,"group_logz":{"11683":-0.020784378051757812,"12447":-5.857063293457031,"2243":-4.19120979309082,"4547":-9.54931640625,"19598":-11.575655937194824}},"graph":{"mac_fp16":{"p":[0.9847913920767701,0.015208607923229964],"max_abs_dp":2.139517233712418e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9848408403345688,0.015159159665431325],"max_abs_dp":5.158777503233569e-05,"argmax_equal":true}}}]},{"id":"own_email_03","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"9a9dfa77ba9fa299644e1bd148119a90d17ebf3ddafec5a99fd958cb784c2125","refused":"the request is refused (a question without instructions)","questions":[{"type":"choice","n_options":3,"row_len":221,"slot":220,"row_ids_sha256":"eb19236198694ba8b6cd6812a410e2de8c69461c68753f80f12e74a8f7433f4b","oracle":{"probs":[0.03908171190662607,0.9356159591382318,0.02530232895514213],"argmax":"fri","top2_margin":0.8965342472316058,"near_tie":false,"group_logz":{"41":-3.252960205078125,"334":-10.85319709777832,"42":-0.07740974426269531,"378":-7.529410362243652,"43":-3.687718391418457,"340":-11.008645057678223}},"graph":{"mac_fp16":{"p":[0.03893610993231835,0.9358228315305697,0.02524105853711188],"max_abs_dp":0.00020687239233785348,"argmax_equal":true},"iphone_int8mlp":{"p":[0.03946668626106954,0.9350030601368265,0.02553025360210379],"max_abs_dp":0.0006128990014052649,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":203,"slot":202,"row_ids_sha256":"97eea1bc5e0ee0150350e55cd28d10ebb58f453b008b0b18a65ed82282b101cb","oracle":{"probs":[0.9821750524921495,0.017824947507850467],"argmax":"yes","top2_margin":0.9643501049842991,"near_tie":false,"group_logz":{"11683":-0.0218353271484375,"12447":-8.202981948852539,"2243":-4.031005859375,"4547":-11.46577262878418,"19598":-12.668841361999512}},"graph":{"mac_fp16":{"p":[0.9821503164421935,0.017849683557806473],"max_abs_dp":2.4736049956008976e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9820450837987342,0.017954916201265922],"max_abs_dp":0.00012996869341545467,"argmax_equal":true}}}]},{"id":"own_meeting_04","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"b887a2559ad5d2cda2fa5d9dbe645b659b512e8404ab4385878cfd48d64af7e2","questions":[{"type":"choice","n_options":3,"row_len":169,"slot":168,"row_ids_sha256":"1bee759ca1b7c03ed872f64462064b8ea482843ceac34277a7b34369ba75ef52","oracle":{"probs":[0.0065935293528362315,0.987604375056557,0.00580209559060679],"argmax":"sam","top2_margin":0.9810108457037208,"near_tie":false,"group_logz":{"41":-5.027641296386719,"334":-13.737964630126953,"42":-0.0184478759765625,"378":-8.560791015625,"43":-5.155510902404785,"340":-13.505451202392578}},"graph":{"mac_fp16":{"p":[0.006587921056937504,0.9876106422209313,0.005801436722131299],"max_abs_dp":6.267164374351708e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.006619636287583563,0.9876001660931307,0.005780197619285723],"max_abs_dp":2.610693474733132e-05,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":148,"slot":147,"row_ids_sha256":"2f118c2df0497d660db13118e995c2ab16dd4e45c7794516795b6c38350d1def","oracle":{"probs":[0.9830288879577047,0.0169711120422952],"argmax":"yes","top2_margin":0.9660577759154095,"near_tie":false,"group_logz":{"11683":-0.036693572998046875,"12447":-4.1448211669921875,"2243":-4.095819473266602,"4547":-7.699419975280762,"19598":-10.700578689575195}},"graph":{"mac_fp16":{"p":[0.9830134962765178,0.01698650372348207],"max_abs_dp":1.5391681186915918e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9831674522889808,0.016832547711019157],"max_abs_dp":0.00013856433127612533,"argmax_equal":true}}},{"type":"choice","n_options":3,"row_len":173,"slot":172,"row_ids_sha256":"9bd418ffc31cd4f6d77864b5bb4511fb2091c985302c39171ab7057c4351d0d8","oracle":{"probs":[0.006937989846988747,0.9791247250965006,0.013937285056510674],"argmax":"postponed","top2_margin":0.9651874400399899,"near_tie":false,"group_logz":{"41":-4.97811222076416,"334":-12.886127471923828,"42":-0.02846527099609375,"378":-7.477147102355957,"43":-4.280556678771973,"340":-11.61629867553711}},"graph":{"mac_fp16":{"p":[0.006924753878009915,0.979170916680044,0.013904329441946117],"max_abs_dp":4.619158354335351e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.006890258047678497,0.9792523299875143,0.013857411964807189],"max_abs_dp":0.00012760489101371064,"argmax_equal":true}}}]},{"id":"own_review_05","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"4aba750c16d56fa58c898acd99fb7ce3b01a924eb7d663357d85ae4824840aea","questions":[{"type":"score","n_options":5,"row_len":157,"slot":156,"row_ids_sha256":"bbef95b0f1935443b5a78af28ba1a19af398356ff5729c6c29d4099833a9eaab","oracle":{"probs":[0.2777522032895745,0.6119952705392683,0.09705026505113425,0.01079862404417199,0.002403637075850887],"argmax":"1","top2_margin":0.33424306724969377,"near_tie":false,"group_logz":{"24":-1.290985107421875,"25":-0.5009899139404297,"26":-2.3424854278564453,"27":-4.538295745849609,"28":-6.040731430053711}},"graph":{"mac_fp16":{"p":[0.2783144218410621,0.6114659514782835,0.09701425203939568,0.010800758212308543,0.002404616428950138],"max_abs_dp":0.000562218551487581,"argmax_equal":true},"iphone_int8mlp":{"p":[0.282091558617667,0.6095323330884533,0.09530542120039787,0.010660579807629045,0.0024101072858527394],"max_abs_dp":0.004339355328092487,"argmax_equal":true}}},{"type":"choice","n_options":10,"row_len":187,"slot":186,"row_ids_sha256":"12e6a52fc8d58b1a88729631510a00ef3b485e814c68a92974b83c696a99e7a2","oracle":{"probs":[0.0043330248165516965,0.9731332421137914,0.003021666346946113,0.0018297399482078644,0.0028148915782752557,0.003187701238493076,0.0016636870533650815,0.001508793594702715,0.0069440255820294415,0.0015632277276373928],"argmax":"durability","top2_margin":0.9661892165317619,"near_tie":false,"group_logz":{"41":-5.450658798217773,"334":-13.87553596496582,"42":-0.036403656005859375,"378":-8.092262268066406,"43":-5.8111162185668945,"340":-14.012304306030273,"44":-6.312750816345215,"388":-14.751544952392578,"45":-5.882000923156738,"407":-13.660892486572266,"46":-5.757624626159668,"410":-13.814319610595703,"47":-6.407888412475586,"417":-14.598289489746094,"48":-6.505614280700684,"403":-14.498746871948242,"49":-4.979043006896973,"358":-11.933660507202148,"50":-6.470171928405762,"461":-13.763557434082031}},"graph":{"mac_fp16":{"p":[0.004332016681070756,0.9731312658332426,0.0030250154699195788,0.0018292328736896327,0.0028126284612041814,0.003185710610326654,0.0016634453104585475,0.0015088303204942638,0.0069487207762996585,0.00156313366329418],"max_abs_dp":4.6951942702169736e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.004344700374219557,0.9731202090171479,0.003001102177772546,0.0018195082326197029,0.002807287486482413,0.0031906593252626282,0.0016545359567184021,0.0015030060735731956,0.006997455784862635,0.0015615355713410495],"max_abs_dp":5.3430202833193094e-05,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":150,"slot":149,"row_ids_sha256":"694d5dc65b9fca69e1e5e1aaadb86695f521d74aaee4728f35694944f7809f56","oracle":{"probs":[0.011461355163087824,0.9885386448369122],"argmax":"no","top2_margin":0.9770772896738245,"near_tie":false,"group_logz":{"11683":-4.472970962524414,"12447":-10.436290740966797,"2243":-0.01572418212890625,"4547":-6.712427139282227,"19598":-8.846466064453125}},"graph":{"mac_fp16":{"p":[0.011463157894994351,0.9885368421050056],"max_abs_dp":1.802731906641597e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.011508962930914067,0.988491037069086],"max_abs_dp":4.7607767826243416e-05,"argmax_equal":true}}}]},{"id":"own_order_06","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"f1c3d2f07f25240a7986831d8afb9c4e7b7cbef01868c8595f4e6122b052ad9c","questions":[{"type":"noul","n_options":2,"row_len":248,"slot":247,"row_ids_sha256":"7d9ddb918ab190a70e74d1357f48c6c21d321474a5a943230d73f483b6662460","oracle":{"probs":[0.028909274708421393,0.9710907252915786],"argmax":"no","top2_margin":0.9421814505831572,"near_tie":false,"group_logz":{"11683":-3.552225112915039,"12447":-8.477952003479004,"2243":-0.037967681884765625,"4547":-5.377815246582031,"19598":-8.642589569091797}},"graph":{"mac_fp16":{"p":[0.028784783675007867,0.9712152163249921],"max_abs_dp":0.0001244910334135263,"argmax_equal":true},"iphone_int8mlp":{"p":[0.028664131239324594,0.9713358687606753],"max_abs_dp":0.00024514346909679935,"argmax_equal":true}}},{"type":"choice","n_options":3,"row_len":238,"slot":237,"row_ids_sha256":"4ab92ecc0a452b59a9ddefe91be60ac5bea103e95bacf366c60196ce357572a5","oracle":{"probs":[0.9919505169949644,0.004470469809779983,0.0035790131952555537],"argmax":"standard","top2_margin":0.9874800471851845,"near_tie":false,"group_logz":{"41":-0.014110565185546875,"334":-9.6846284866333,"42":-5.416290283203125,"378":-14.738540649414062,"43":-5.638696670532227,"340":-14.936220169067383}},"graph":{"mac_fp16":{"p":[0.9919564472797927,0.004468738809137947,0.0035748139110692947],"max_abs_dp":5.9302848283149245e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9920238521758329,0.004431125791015359,0.003545022033151843],"max_abs_dp":7.333518086849455e-05,"argmax_equal":true}}}]},{"id":"own_invoice_07","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"5fe483ebe95c823035e971c5a9addaec7f2358a4dfe03f569161504d848d2490","questions":[{"type":"choice","n_options":4,"row_len":200,"slot":199,"row_ids_sha256":"8795e77d023b3d038edd1f31e31235e73d280e75994bd0f52f5b30624380d46a","oracle":{"probs":[0.009615453171493153,0.9697700920601586,0.01628689005031153,0.004327564718036752],"argmax":"partial","top2_margin":0.9534832020098472,"near_tie":false,"group_logz":{"41":-4.650688171386719,"334":-13.17629623413086,"42":-0.03700065612792969,"378":-8.312989234924316,"43":-4.123699188232422,"340":-12.132232666015625,"44":-5.449054718017578,"388":-13.826916694641113}},"graph":{"mac_fp16":{"p":[0.009622362003750174,0.9698125231230219,0.016244664576391312,0.004320450296836586],"max_abs_dp":4.243106286327958e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.009467312318435438,0.9700739494200498,0.016197374328073742,0.004261363933441127],"max_abs_dp":0.0003038573598911354,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":179,"slot":178,"row_ids_sha256":"7293a9f6238ca7ef7dcd7559bfdb03f70cdaa1898a42b3d1c8ca4ea0ef9012fc","oracle":{"probs":[0.936992465636746,0.06300753436325397],"argmax":"yes","top2_margin":0.873984931273492,"near_tie":false,"group_logz":{"11683":-0.08135032653808594,"12447":-4.953094482421875,"2243":-2.780771255493164,"4547":-7.528008460998535,"19598":-10.33047866821289}},"graph":{"mac_fp16":{"p":[0.9371124580804909,0.06288754191950906],"max_abs_dp":0.00011999244374494289,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9362826203670791,0.06371737963292087],"max_abs_dp":0.0007098452696669005,"argmax_equal":true}}},{"type":"score","n_options":4,"row_len":212,"slot":211,"row_ids_sha256":"e851bbcd81b86f4b3f7ed6032b058c11f2610ffc199745d1d9a789ddfe706343","oracle":{"probs":[0.014263623295698838,0.021031653770302573,0.17734707767809194,0.7873576452559067],"argmax":"3","top2_margin":0.6100105675778147,"near_tie":false,"group_logz":{"24":-4.3279619216918945,"25":-3.939645767211914,"26":-1.807565689086914,"27":-0.31699180603027344}},"graph":{"mac_fp16":{"p":[0.014222776683660052,0.020949470924567646,0.17714562975430323,0.7876821226374691],"max_abs_dp":0.0003244773815623825,"argmax_equal":true},"iphone_int8mlp":{"p":[0.014591941034970126,0.021391172778186533,0.18030051575260242,0.7837163704342409],"max_abs_dp":0.003641274821665763,"argmax_equal":true}}}]},{"id":"own_sensor_08","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"e3a28a8f5ef4715804bfc3776226f6cb2d0d4b9b67b4d4f3ba191a3ffd6bbdc8","questions":[{"type":"choice","n_options":4,"row_len":210,"slot":209,"row_ids_sha256":"7eaa91552467fdd755bcf7af24768a03d8329d0b4bb40683769b4a09f70160d9","oracle":{"probs":[0.15266640810374033,0.629930682491781,0.11654535298331961,0.10085755642115911],"argmax":"humidity","top2_margin":0.47726427438804064,"near_tie":false,"group_logz":{"41":-1.9080677032470703,"334":-9.411642074584961,"42":-0.49071311950683594,"378":-7.731256484985352,"43":-2.178042411804199,"340":-9.278100967407227,"44":-2.3226137161254883,"388":-9.508708000183105}},"graph":{"mac_fp16":{"p":[0.1522756478378651,0.6299392972285162,0.11656758797912307,0.10121746695449567],"max_abs_dp":0.0003907602658752285,"argmax_equal":true},"iphone_int8mlp":{"p":[0.1495192067501536,0.6366955808640196,0.11523439242426277,0.09855081996156406],"max_abs_dp":0.0067648983722385925,"argmax_equal":true}}},{"type":"score","n_options":3,"row_len":227,"slot":226,"row_ids_sha256":"ff3ca73542dddcaef93bb9ad037ba3433a35a5a341a8b91d053e395395f3df3d","oracle":{"probs":[0.8881548780729266,0.05568507539551519,0.05616004653155808],"argmax":"0","top2_margin":0.8319948315413686,"near_tie":false,"group_logz":{"24":-0.14306259155273438,"25":-2.912496566772461,"26":-2.904003143310547}},"graph":{"mac_fp16":{"p":[0.8882622896172171,0.055633981251777716,0.05610372913100516],"max_abs_dp":0.00010741154429050326,"argmax_equal":true},"iphone_int8mlp":{"p":[0.8842365508708364,0.057161328071128885,0.05860212105803468],"max_abs_dp":0.003918327202090177,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":213,"slot":212,"row_ids_sha256":"0ed1cbf2affe41ca818da6bf7d517808144264ff147c1f36a224a0faa2eaf8c4","oracle":{"probs":[0.9387655881243979,0.061234411875602024],"argmax":"yes","top2_margin":0.8775311762487958,"near_tie":false,"group_logz":{"11683":-0.0738983154296875,"12447":-5.422117233276367,"2243":-2.8037548065185547,"4547":-7.782672882080078,"19598":-10.65095329284668}},"graph":{"mac_fp16":{"p":[0.938765263214075,0.061234736785924934],"max_abs_dp":3.2491032291026567e-07,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9385322375389696,0.06146776246103035],"max_abs_dp":0.0002333505854283241,"argmax_equal":true}}}]},{"id":"own_fiveq_09","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"070c8cb5c46703c5c56cd2f3ccdda749116aa4e3e995d90d5546e577893f35ba","questions":[{"type":"noul","n_options":2,"row_len":143,"slot":142,"row_ids_sha256":"fa3977e7ec1211d9924b3f349f993d352f0d2413a76022a4321b0d80b59cd66a","oracle":{"probs":[0.02065800534351217,0.9793419946564879],"argmax":"no","top2_margin":0.9586839893129757,"near_tie":false,"group_logz":{"11683":-3.883646011352539,"12447":-10.158315658569336,"2243":-0.024868011474609375,"4547":-7.097221374511719,"19598":-8.958032608032227}},"graph":{"mac_fp16":{"p":[0.02068072443961588,0.9793192755603841],"max_abs_dp":2.2719096103740455e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.02088572635598767,0.9791142736440124],"max_abs_dp":0.00022772101247549997,"argmax_equal":true}}},{"type":"choice","n_options":4,"row_len":148,"slot":147,"row_ids_sha256":"85dbf02217d39f4020a5d5eecc846c10354c33b5dfb387aea342baa6c25f1eff","oracle":{"probs":[0.005830144087192254,0.9848634511551384,0.0044199781280755785,0.004886426629593821],"argmax":"side","top2_margin":0.9790333070679461,"near_tie":false,"group_logz":{"41":-5.151847839355469,"334":-13.947728157043457,"42":-0.022386550903320312,"378":-8.638666152954102,"43":-5.428754806518555,"340":-13.892568588256836,"44":-5.328428268432617,"388":-13.904540061950684}},"graph":{"mac_fp16":{"p":[0.005826830207387911,0.9848490618012883,0.004429932668508631,0.004894175322815152],"max_abs_dp":1.4389353850119413e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.005847992447190703,0.9848764741805618,0.0044126635397083655,0.004862869832539103],"max_abs_dp":2.3556797054717464e-05,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":144,"slot":143,"row_ids_sha256":"04aa6dc88e2dd44ed794db2eb758be022d0ba1fabb053f9066a8d6963bb7d16a","oracle":{"probs":[0.9743214560874587,0.025678543912541367],"argmax":"yes","top2_margin":0.9486429121749174,"near_tie":false,"group_logz":{"11683":-0.030900955200195312,"12447":-6.164236068725586,"2243":-3.6669864654541016,"4547":-9.039173126220703,"19598":-11.51425552368164}},"graph":{"mac_fp16":{"p":[0.9743522957804249,0.025647704219574966],"max_abs_dp":3.0839692966400795e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.974640019067372,0.02535998093262798],"max_abs_dp":0.00031856297991338794,"argmax_equal":true}}},{"type":"score","n_options":3,"row_len":146,"slot":145,"row_ids_sha256":"a14934c5dc45743d4ac04746f52e242ad1e7793af1f091c6cd26a212b5f10869","oracle":{"probs":[0.26444239002619446,0.206522248908289,0.5290353610655165],"argmax":"2","top2_margin":0.26459297103932206,"near_tie":false,"group_logz":{"24":-1.343942642211914,"25":-1.5911579132080078,"26":-0.6505107879638672}},"graph":{"mac_fp16":{"p":[0.2637223291691452,0.2067413444173766,0.5295363264134783],"max_abs_dp":0.0007200608570492872,"argmax_equal":true},"iphone_int8mlp":{"p":[0.2643376031707962,0.20517919412751376,0.53048320270169],"max_abs_dp":0.001447841636173508,"argmax_equal":true}}},{"type":"choice","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"88369e6290a5a5ad8be7b22aa5cedc4952363c39a032f9a0955d17052c46eb22","oracle":{"probs":[0.9862593597302051,0.01374064026979496],"argmax":"yes","top2_margin":0.9725187194604101,"near_tie":false,"group_logz":{"41":-0.022668838500976562,"334":-8.883713722229004,"42":-4.296230316162109,"378":-12.720682144165039}},"graph":{"mac_fp16":{"p":[0.9862622305577061,0.013737769442293995],"max_abs_dp":2.870827501055828e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9863343426065786,0.013665657393421463],"max_abs_dp":7.498287637353052e-05,"argmax_equal":true}}}]},{"id":"own_long_log_10","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"b09133cffe337a074c3433fed3296fe92bac4f2c5b3e1d26823326ff7123f0fd","questions":[{"type":"choice","n_options":6,"row_len":1652,"slot":1651,"row_ids_sha256":"79515d86625ef49d3c3837fd69acb64419b9d845cc10ea4c0ef7ad97fae053cb","oracle":{"probs":[0.008947252398848532,0.01655991843500178,0.0539707775202545,0.026813064588185814,0.8776247765753978,0.016084210482311588],"argmax":"db_replica_2","top2_margin":0.8236539990551434,"near_tie":false,"group_logz":{"41":-4.74171257019043,"334":-12.273391723632812,"42":-4.126073837280273,"378":-11.287455558776855,"43":-2.9446163177490234,"340":-9.939370155334473,"44":-3.644169807434082,"388":-10.762718200683594,"45":-0.1558399200439453,"407":-6.906404495239258,"46":-4.155220985412598,"410":-11.237455368041992}},"graph":{"mac_fp16":{"p":[0.008931394252895148,0.016574220384516506,0.053919033488517974,0.026799167274918163,0.8776727901725683,0.01610339442658397],"max_abs_dp":5.174403173652253e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.008829215091975859,0.01644919997126062,0.05387205015412446,0.026570632630961304,0.8783892544544368,0.015889647697240987],"max_abs_dp":0.0007644778790389939,"argmax_equal":true}}},{"type":"score","n_options":4,"row_len":1643,"slot":1642,"row_ids_sha256":"a542a68e1a675c3bfe68e4c23ce9ef28c02f750f00bf23aff0163da2c0c2b354","oracle":{"probs":[0.09335149279919784,0.059435118592222676,0.8069077299823455,0.040305658626234066],"argmax":"2","top2_margin":0.7135562371831476,"near_tie":false,"group_logz":{"24":-2.417360305786133,"25":-2.868846893310547,"26":-0.26052284240722656,"27":-3.2572402954101562}},"graph":{"mac_fp16":{"p":[0.09323801420762551,0.0593858685263817,0.8070633570045054,0.04031276026148732],"max_abs_dp":0.00015562702215987656,"argmax_equal":true},"iphone_int8mlp":{"p":[0.09537877803767322,0.0604081748845074,0.8027293185621256,0.04148372851569375],"max_abs_dp":0.004178411420219907,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":1641,"slot":1640,"row_ids_sha256":"184789ba105bf46379285e9f297728f1f4dd30130443c274db562234f567032e","oracle":{"probs":[0.9015153328443315,0.09848466715566837],"argmax":"yes","top2_margin":0.8030306656886631,"near_tie":false,"group_logz":{"11683":-0.12225532531738281,"12447":-5.5121564865112305,"2243":-2.3364315032958984,"4547":-7.305834770202637,"19598":-10.051982879638672}},"graph":{"mac_fp16":{"p":[0.9013477940572803,0.09865220594271976],"max_abs_dp":0.00016753878705139114,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9028271120826127,0.0971728879173874],"max_abs_dp":0.0013117792382811855,"argmax_equal":true}}}]},{"id":"own_long_contract_11","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"b5f180cf87ad401094dcb9d659b6783672e07bed469d61d930fe9db68e0b4e56","questions":[{"type":"choice","n_options":4,"row_len":1455,"slot":1454,"row_ids_sha256":"f723fe05ce12ddc440bfd30a16cd28bb3e814f7a305befe699ae54e59a51e847","oracle":{"probs":[0.007670990893221738,0.007300232754081285,0.9773208406490133,0.007707935703683668],"argmax":"d90","top2_margin":0.9696129049453296,"near_tie":false,"group_logz":{"41":-4.87770938873291,"334":-13.666501998901367,"42":-4.927248954772949,"378":-13.266364097595215,"43":-0.030340194702148438,"340":-8.762927055358887,"44":-4.8729047775268555,"388":-13.326971054077148}},"graph":{"mac_fp16":{"p":[0.007679624264282164,0.007297667455873648,0.9773137273947201,0.00770898088512409],"max_abs_dp":8.633371060426387e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0075465196676341205,0.007167930583808889,0.9775956840843111,0.007689865664246013],"max_abs_dp":0.0002748434352978224,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":1444,"slot":1443,"row_ids_sha256":"a0e9c05929d89b44a98bb65317241228a04bf0d8033ea9df60faf08f57df9643","oracle":{"probs":[0.9843623268045123,0.01563767319548783],"argmax":"yes","top2_margin":0.9687246536090245,"near_tie":false,"group_logz":{"11683":-0.023303985595703125,"12447":-5.474374771118164,"2243":-4.165615081787109,"4547":-8.786043167114258,"19598":-11.398540496826172}},"graph":{"mac_fp16":{"p":[0.9843247831504027,0.01567521684959712],"max_abs_dp":3.7543654109573055e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9843043344219345,0.01569566557806552],"max_abs_dp":5.7992382577753965e-05,"argmax_equal":true}}},{"type":"score","n_options":4,"row_len":1451,"slot":1450,"row_ids_sha256":"04df2ae7df5f628b48c7333c289ff7f3f5633bb1fe43a61b51cb293c58d76156","oracle":{"probs":[0.012842338488969473,0.05740872689065848,0.8143042637399892,0.11544467088038282],"argmax":"2","top2_margin":0.6988595928596064,"near_tie":false,"group_logz":{"24":-4.379189491271973,"25":-2.8817405700683594,"26":-0.22960281372070312,"27":-2.183145523071289}},"graph":{"mac_fp16":{"p":[0.012858615581925855,0.05749057078244641,0.8144183828585003,0.11523243077712736],"max_abs_dp":0.00021224010325546216,"argmax_equal":true},"iphone_int8mlp":{"p":[0.013131875502292986,0.05677671488524657,0.8152726050616835,0.11481880455077689],"max_abs_dp":0.0009683413216943215,"argmax_equal":true}}}]},{"id":"own_long_minutes_12","source":"own","publication":"withhold_text","text":"withheld (invented names not screened)","request_sha256":"4704f75913e4403df3888cc3312fae03d5bec64a805c1176b474585fab9fd903","questions":[{"type":"choice","n_options":3,"row_len":1375,"slot":1374,"row_ids_sha256":"d03dcc6d2589f5171c630879b288a0109be13bb057083938447b58860766f489","oracle":{"probs":[0.007332336702911569,0.010544185258352682,0.9821234780387357],"argmax":"deferred","top2_margin":0.971579292780383,"near_tie":false,"group_logz":{"41":-4.922574996948242,"334":-14.477052688598633,"42":-4.559294700622559,"378":-13.451051712036133,"43":-0.025152206420898438,"340":-9.469415664672852}},"graph":{"mac_fp16":{"p":[0.0073513456231664815,0.010556949807473277,0.9820917045693602],"max_abs_dp":3.177346937544012e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.007248543759270469,0.010354411039583863,0.9823970452011456],"max_abs_dp":0.00027356716240989343,"argmax_equal":true}}},{"type":"choice","n_options":4,"row_len":1380,"slot":1379,"row_ids_sha256":"ac93947c7cb71c9d51fee53bdb87b976b7a92f2e01da5bd53d0b2cae91ef15c1","oracle":{"probs":[0.9861949631008925,0.0038485564765297943,0.004915707852024673,0.005040772570553143],"argmax":"thorley","top2_margin":0.9811541905303394,"near_tie":false,"group_logz":{"41":-0.019985198974609375,"334":-10.124239921569824,"42":-5.566141128540039,"378":-15.274873733520508,"43":-5.321403503417969,"340":-14.874229431152344,"44":-5.2962799072265625,"388":-14.986007690429688}},"graph":{"mac_fp16":{"p":[0.9861920774827602,0.0038479940277635926,0.004916665411689127,0.00504326307778721],"max_abs_dp":2.8856181323311247e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9862983468628512,0.003825816991118933,0.004886278509750151,0.004989557636279632],"max_abs_dp":0.00010338376195873078,"argmax_equal":true}}},{"type":"noul","n_options":2,"row_len":1356,"slot":1355,"row_ids_sha256":"6973aa1d16e50dc2246d6c737a79acee5985d644b3c68ef36f8529d9145db23a","oracle":{"probs":[0.0660199576052661,0.933980042394734],"argmax":"no","top2_margin":0.8679600847894678,"near_tie":false,"group_logz":{"11683":-2.75506591796875,"12447":-6.694241523742676,"2243":-0.10556793212890625,"4547":-3.5509395599365234,"19598":-8.353870391845703}},"graph":{"mac_fp16":{"p":[0.06640417298816316,0.9335958270118367],"max_abs_dp":0.000384215382897235,"argmax_equal":true},"iphone_int8mlp":{"p":[0.06715067413002193,0.9328493258699782],"max_abs_dp":0.001130716524755826,"argmax_equal":true}}}]},{"id":"card_refund","source":"card","publication":"publish_text","request":{"state":"I was charged twice this month, please refund one of them.","questions":{"refund":{"type":"noul","instructions":"Is the customer asking for a refund?"},"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}},"urgency":{"type":"score","instructions":"How urgent is this?","criteria":["Can wait","Today","Blocking the customer now"]}}},"gold":{"refund":null,"team":null,"urgency":null},"note":"model card example: model.system_one(state, questions)","provenance":{"file":"README.md","repo":"LiquidAI/d1-3B","revision":"da1fe36a861f24690f27f622dca1d8688503d113","section":"How to use","readme_sha256":"cf063f71ce852ef4c924d192c34f61727f11d121c60b01ad9092ba93ac03bda3"},"questions":[{"name":"refund","type":"noul","n_options":2,"keys":["yes","no"],"codes":null,"gold":null,"row_len":39,"slot":38,"row_ids_sha256":"4860c5242593b975e5af965553153241f106841857dd06a6c78edc8c02ec1359","oracle":{"probs":[0.9896227586306204,0.010377241369379547],"argmax":"yes","top2_margin":0.9792455172612409,"near_tie":false,"group_logz":{"11683":-0.017459869384765625,"12447":-5.638872146606445,"2243":-4.575168609619141,"4547":-9.732593536376953,"19598":-11.329362869262695}},"graph":{"mac_fp16":{"p":[0.9896317169498651,0.010368283050134937],"max_abs_dp":8.958319244634083e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9896946692114699,0.010305330788530197],"max_abs_dp":7.191058084943247e-05,"argmax_equal":true}}},{"name":"team","type":"choice","n_options":3,"keys":["billing","technical","fraud"],"codes":["A","B","C"],"gold":null,"row_len":61,"slot":60,"row_ids_sha256":"326fc6650161497336e2631bed2ab6bedf3dccb4fff181f25bff6e4317e44833","oracle":{"probs":[0.9835051691325284,0.01064694075297905,0.005847890114492638],"argmax":"billing","top2_margin":0.9728582283795494,"near_tie":false,"group_logz":{"41":-0.025846481323242188,"334":-8.818021774291992,"42":-4.55169677734375,"378":-12.863260269165039,"43":-5.150888442993164,"340":-13.634050369262695}},"graph":{"mac_fp16":{"p":[0.983499971781699,0.010642661793691754,0.005857366424609297],"max_abs_dp":9.476310116658497e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9835667192872184,0.010634730901953939,0.005798549810827583],"max_abs_dp":6.155015469000436e-05,"argmax_equal":true}}},{"name":"urgency","type":"score","n_options":3,"keys":["0","1","2"],"codes":null,"gold":null,"row_len":54,"slot":53,"row_ids_sha256":"b8c2ab3a4060bf947dc8369867b269eb77af5141aa3492b5ee8c0a6f2f6e6765","oracle":{"probs":[0.33857603078744675,0.567444155168246,0.09397981404430734],"argmax":"1","top2_margin":0.22886812438079923,"near_tie":false,"group_logz":{"24":-1.109567642211914,"25":-0.5931739807128906,"26":-2.3912363052368164}},"graph":{"mac_fp16":{"p":[0.33873647248991473,0.5672736783645534,0.09398984914553184],"max_abs_dp":0.00017047680369253193,"argmax_equal":true},"iphone_int8mlp":{"p":[0.337798065837446,0.5677139400714755,0.09448799409107857],"max_abs_dp":0.0007779649500007646,"argmax_equal":true}}}],"api":{"path":"tree","probs":[[0.9896224648122013,0.01037753518779873],[0.9835054699685586,0.010646822165773784,0.005847707865667626],[0.3385783508447072,0.5674415496303346,0.09398009952495819]],"input_tokens":116,"response":{"answers":{"refund":{"type":"noul","noul":0.9896224648122013},"team":{"type":"choice","choice":"billing","confidence":0.9835054699685586,"probabilities":{"billing":0.9835054699685586,"technical":0.010646822165773784,"fraud":0.005847707865667626}},"urgency":{"type":"score","score":0.7554017486802509,"confidence":0.5674415496303346,"probabilities":{"0":0.3385783508447072,"1":0.5674415496303346,"2":0.09398009952495819},"legend":{"0":"Can wait","1":"Today","2":"Blocking the customer now"}}},"usage":{"input_tokens":116,"output_tokens":0}},"max_abs_dp_vs_rows":2.605537911426481e-06,"seconds":0.446}},{"id":"card_ticket_00","source":"card","publication":"publish_text","request":{"state":"Where is my parcel? It was due Monday.","questions":{"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}}}},"gold":{"team":null},"note":"model card example: system_one_batch ticket 0","provenance":{"file":"README.md","repo":"LiquidAI/d1-3B","revision":"da1fe36a861f24690f27f622dca1d8688503d113","section":"How to use","readme_sha256":"cf063f71ce852ef4c924d192c34f61727f11d121c60b01ad9092ba93ac03bda3"},"questions":[{"name":"team","type":"choice","n_options":3,"keys":["billing","technical","fraud"],"codes":["A","B","C"],"gold":null,"row_len":58,"slot":57,"row_ids_sha256":"6e20c7f3ac246c7b32baaefdbb718aebd570e7542d4fc2f7167629cf232259cd","oracle":{"probs":[0.1558159573598305,0.8256628651221605,0.018521177518008938],"argmax":"technical","top2_margin":0.6698469077623299,"near_tie":false,"group_logz":{"41":-1.888824462890625,"334":-8.995217323303223,"42":-0.2213134765625,"378":-7.1851348876953125,"43":-4.018585205078125,"340":-11.124445915222168}},"graph":{"mac_fp16":{"p":[0.15584711791653344,0.8256193566444818,0.01853352543898477],"max_abs_dp":4.35084776786443e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.15118859280692865,0.8305955896550227,0.018215817538048636],"max_abs_dp":0.004932724532862198,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.1558159573598305,0.8256628651221605,0.018521177518008938]],"input_tokens":58,"response":{"answers":{"team":{"type":"choice","choice":"technical","confidence":0.8256628651221605,"probabilities":{"billing":0.1558159573598305,"technical":0.8256628651221605,"fraud":0.018521177518008938}}},"usage":{"input_tokens":58,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.2605}},{"id":"card_ticket_01","source":"card","publication":"publish_text","request":{"state":"The app crashes when I open settings.","questions":{"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}}}},"gold":{"team":null},"note":"model card example: system_one_batch ticket 1","provenance":{"file":"README.md","repo":"LiquidAI/d1-3B","revision":"da1fe36a861f24690f27f622dca1d8688503d113","section":"How to use","readme_sha256":"cf063f71ce852ef4c924d192c34f61727f11d121c60b01ad9092ba93ac03bda3"},"questions":[{"name":"team","type":"choice","n_options":3,"keys":["billing","technical","fraud"],"codes":["A","B","C"],"gold":null,"row_len":56,"slot":55,"row_ids_sha256":"1ca22783cf12913375b6d208863b70b917d4e4d46636e7ef879ddc771941f724","oracle":{"probs":[0.006503933493475309,0.987195489554394,0.006300576952130822],"argmax":"technical","top2_margin":0.9806915560609186,"near_tie":false,"group_logz":{"41":-5.04737663269043,"334":-13.141956329345703,"42":-0.024915695190429688,"378":-7.778657913208008,"43":-5.0791425704956055,"340":-12.943958282470703}},"graph":{"mac_fp16":{"p":[0.006504245217802115,0.9871932253587542,0.0063025294234436095],"max_abs_dp":2.264195639756217e-06,"argmax_equal":true},"iphone_int8mlp":{"p":[0.0065117982180930385,0.9872281567394655,0.0062600450424414675],"max_abs_dp":4.053190968935413e-05,"argmax_equal":true}}}],"api":{"path":"row","probs":[[0.006503933493475309,0.987195489554394,0.006300576952130822]],"input_tokens":56,"response":{"answers":{"team":{"type":"choice","choice":"technical","confidence":0.987195489554394,"probabilities":{"billing":0.006503933493475309,"technical":0.987195489554394,"fraud":0.006300576952130822}}},"usage":{"input_tokens":56,"output_tokens":0}},"max_abs_dp_vs_rows":0.0,"seconds":0.2474}},{"id":"long_15k","source":"own_long","publication":"publish_text","request":{"state":{"log":"cold-room monitoring, early shift","site_code":"CR-2","limits":{"temp_c_max":8.0,"humidity_pct_max":75},"units":["unit-03","unit-05","unit-07"],"readings":[{"time":"2026-03-02 06:00","unit":"unit-03","temp_c":3.6,"humidity_pct":58,"door":"closed"},{"time":"2026-03-02 06:00","unit":"unit-05","temp_c":4.7,"humidity_pct":69,"door":"closed"},{"time":"2026-03-02 06:00","unit":"unit-07","temp_c":5.2,"humidity_pct":68,"door":"closed"},{"time":"2026-03-02 06:15","unit":"unit-03","temp_c":3.9,"humidity_pct":65,"door":"closed"},{"time":"2026-03-02 06:15","unit":"unit-05","temp_c":4.3,"humidity_pct":64,"door":"closed"},{"time":"2026-03-02 06:15","unit":"unit-07","temp_c":5.5,"humidity_pct":63,"door":"closed"},{"time":"2026-03-02 06:30","unit":"unit-03","temp_c":4.2,"humidity_pct":60,"door":"closed"},{"time":"2026-03-02 06:30","unit":"unit-05","temp_c":4.6,"humidity_pct":59,"door":"closed"},{"time":"2026-03-02 06:30","unit":"unit-07","temp_c":7.6,"humidity_pct":58,"door":"closed"},{"time":"2026-03-02 06:45","unit":"unit-03","temp_c":3.8,"humidity_pct":67,"door":"closed"},{"time":"2026-03-02 06:45","unit":"unit-05","temp_c":4.2,"humidity_pct":66,"door":"closed"},{"time":"2026-03-02 06:45","unit":"unit-07","temp_c":9.3,"humidity_pct":65,"door":"open"},{"time":"2026-03-02 07:00","unit":"unit-03","temp_c":4.1,"humidity_pct":62,"door":"closed"},{"time":"2026-03-02 07:00","unit":"unit-05","temp_c":4.5,"humidity_pct":61,"door":"closed"},{"time":"2026-03-02 07:00","unit":"unit-07","temp_c":8.9,"humidity_pct":60,"door":"closed"},{"time":"2026-03-02 07:15","unit":"unit-03","temp_c":3.7,"humidity_pct":69,"door":"closed"},{"time":"2026-03-02 07:15","unit":"unit-05","temp_c":4.8,"humidity_pct":68,"door":"closed"},{"time":"2026-03-02 07:15","unit":"unit-07","temp_c":6.2,"humidity_pct":67,"door":"closed"},{"time":"2026-03-02 07:30","unit":"unit-03","temp_c":4.0,"humidity_pct":64,"door":"closed"},{"time":"2026-03-02 07:30","unit":"unit-05","temp_c":4.4,"humidity_pct":63,"door":"closed"},{"time":"2026-03-02 07:30","unit":"unit-07","temp_c":4.9,"humidity_pct":62,"door":"closed"},{"time":"2026-03-02 07:45","unit":"unit-03","temp_c":3.6,"humidity_pct":59,"door":"closed"},{"time":"2026-03-02 07:45","unit":"unit-05","temp_c":4.7,"humidity_pct":58,"door":"closed"},{"time":"2026-03-02 07:45","unit":"unit-07","temp_c":5.2,"humidity_pct":69,"door":"closed"}],"notes":["readings every 15 minutes per unit","the supervisor reviews any excursion at the end of the shift"]},"questions":{"temp_excursion":{"type":"noul","instructions":"Did any reading go above the temperature limit?"},"first_unit":{"type":"choice","instructions":"Which unit went over a limit first?","criteria":{"u03":"unit-03","u05":"unit-05","u07":"unit-07","none":"No unit went over a limit"}},"severity":{"type":"score","instructions":"How long did the worst temperature excursion last?","criteria":["No excursion","One reading only, back within limits by the next reading","Two or more readings in a row above the limit"]},"door_open":{"type":"noul","instructions":"Was a door open on a unit while its temperature was above the limit?","criteria":{"true":"A door reads open at a reading above the limit","false":"Every reading above the limit has its door closed"}}}},"gold":{"temp_excursion":"true","first_unit":"u07","severity":"2","door_open":"true"},"note":"cold-room log, 8 reading rounds of 3 units: unit-07 above 8.0 at 06:45 and 07:00, door open at 06:45","provenance":{"written_for":"d1-3B Core AI port (round 1)","generator":"make_fixtures.cold_room_log","n":8},"questions":[{"name":"temp_excursion","type":"noul","n_options":2,"keys":["yes","no"],"codes":null,"gold":"yes","row_len":1491,"slot":1490,"row_ids_sha256":"80b47e37b3c3e17fe8a0ac0f78afe856268bd158d71205814336585c1302e773","oracle":{"probs":[0.9389960907214145,0.06100390927858553],"argmax":"yes","top2_margin":0.877992181442829,"near_tie":false,"group_logz":{"11683":-0.08256721496582031,"12447":-4.845874786376953,"2243":-2.8164405822753906,"4547":-7.374444961547852,"19598":-9.952777862548828}},"graph":{"mac_fp16":{"p":[0.9389535544880309,0.06104644551196914],"max_abs_dp":4.2536233383683175e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9394388946549317,0.060561105345068346],"max_abs_dp":0.0004428039335171857,"argmax_equal":true}}},{"name":"first_unit","type":"choice","n_options":4,"keys":["u03","u05","u07","none"],"codes":["A","B","C","D"],"gold":"u07","row_len":1515,"slot":1514,"row_ids_sha256":"cf04815bac5e06b5f97173e47177c1649223c733649060bdb3c45c7052db76af","oracle":{"probs":[0.00497697565219558,0.004190697966413092,0.9686481805696011,0.02218414581179009],"argmax":"u07","top2_margin":0.946464034757811,"near_tie":false,"group_logz":{"41":-5.310738563537598,"334":-13.401742935180664,"42":-5.482693672180176,"378":-13.366426467895508,"43":-0.03965950012207031,"340":-7.904293060302734,"44":-3.816183090209961,"388":-11.46724796295166}},"graph":{"mac_fp16":{"p":[0.004980772208895604,0.004194790100449831,0.9687034722433655,0.02212096544728895],"max_abs_dp":6.318036450114056e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.004974077304042945,0.004180576999534178,0.968488693220807,0.02235665247561588],"max_abs_dp":0.0001725066638257923,"argmax_equal":true}}},{"name":"severity","type":"score","n_options":3,"keys":["0","1","2"],"codes":null,"gold":"2","row_len":1526,"slot":1525,"row_ids_sha256":"22bf7c841fa57dfe9e377f60b27310039d98dd4a2fca5ac2483b3b6a04c05b53","oracle":{"probs":[0.11093528857065148,0.3764973195804266,0.5125673918489219],"argmax":"2","top2_margin":0.13607007226849527,"near_tie":false,"group_logz":{"24":-2.249011993408203,"25":-1.027048110961914,"26":-0.7185268402099609}},"graph":{"mac_fp16":{"p":[0.11093340299042247,0.3763714642809789,0.5126951327285987],"max_abs_dp":0.00012774087967681869,"argmax_equal":true},"iphone_int8mlp":{"p":[0.11121860385735383,0.37917149604646866,0.5096099000961776],"max_abs_dp":0.00295749175274429,"argmax_equal":true}}},{"name":"door_open","type":"noul","n_options":2,"keys":["yes","no"],"codes":null,"gold":"yes","row_len":1522,"slot":1521,"row_ids_sha256":"dae7111cd741a9f386ef4cb54186324b3ae207dfc09a3f529ff9661e583fabbb","oracle":{"probs":[0.9227571353130154,0.07724286468698459],"argmax":"yes","top2_margin":0.8455142706260308,"near_tie":false,"group_logz":{"11683":-0.09287643432617188,"12447":-5.4762725830078125,"2243":-2.5732879638671875,"4547":-7.617818832397461,"19598":-10.642120361328125}},"graph":{"mac_fp16":{"p":[0.9229440733503252,0.07705592664967469],"max_abs_dp":0.00018693803730990277,"argmax_equal":true},"iphone_int8mlp":{"p":[0.9235821345034008,0.07641786549659926],"max_abs_dp":0.0008249991903853671,"argmax_equal":true}}}],"api":{"path":"tree","probs":[[0.9389965277502268,0.061003472249773295],[0.004976909914950764,0.00419070655915684,0.968648319165604,0.022184064360288316],[0.11093384120719489,0.3764995885425728,0.5125665702502323],[0.9227560477143776,0.07724395228562246]],"input_tokens":1644,"response":{"answers":{"temp_excursion":{"type":"noul","noul":0.9389965277502268},"first_unit":{"type":"choice","choice":"u07","confidence":0.968648319165604,"probabilities":{"u03":0.004976909914950764,"u05":0.00419070655915684,"u07":0.968648319165604,"none":0.022184064360288316}},"severity":{"type":"score","score":1.4016327290430375,"confidence":0.5125665702502323,"probabilities":{"0":0.11093384120719489,"1":0.3764995885425728,"2":0.5125665702502323},"legend":{"0":"No excursion","1":"One reading only, back within limits by the next reading","2":"Two or more readings in a row above the limit"}},"door_open":{"type":"noul","noul":0.9227560477143776}},"usage":{"input_tokens":1644,"output_tokens":0}},"max_abs_dp_vs_rows":2.2689621462235188e-06,"seconds":3.9924}},{"id":"long_34k","source":"own_long","publication":"publish_text","request":{"state":[{"seq":1,"time":"2026-04-14 09:03","actor":"the clerk","action":"create","order":"order 00400","amount":40.0},{"seq":2,"time":"2026-04-14 09:06","actor":"the cashier","action":"pay","order":"order 00400","amount":40.0},{"seq":3,"time":"2026-04-14 09:09","actor":"the dispatcher","action":"ship","order":"order 00400","amount":40.0},{"seq":4,"time":"2026-04-14 09:12","actor":"the clerk","action":"create","order":"order 00401","amount":77.25},{"seq":5,"time":"2026-04-14 09:15","actor":"the cashier","action":"pay","order":"order 00401","amount":77.25},{"seq":6,"time":"2026-04-14 09:18","actor":"the dispatcher","action":"ship","order":"order 00401","amount":77.25},{"seq":7,"time":"2026-04-14 09:21","actor":"the clerk","action":"create","order":"order 00402","amount":114.5},{"seq":8,"time":"2026-04-14 09:24","actor":"the cashier","action":"pay","order":"order 00402","amount":114.5},{"seq":9,"time":"2026-04-14 09:27","actor":"the dispatcher","action":"ship","order":"order 00402","amount":114.5},{"seq":10,"time":"2026-04-14 09:30","actor":"the clerk","action":"create","order":"order 00403","amount":151.75},{"seq":11,"time":"2026-04-14 09:33","actor":"the clerk","action":"cancel","order":"order 00403","amount":151.75,"reason":"customer request"},{"seq":12,"time":"2026-04-14 09:36","actor":"the clerk","action":"create","order":"order 00404","amount":188.0},{"seq":13,"time":"2026-04-14 09:39","actor":"the cashier","action":"pay","order":"order 00404","amount":188.0},{"seq":14,"time":"2026-04-14 09:42","actor":"the dispatcher","action":"ship","order":"order 00404","amount":188.0},{"seq":15,"time":"2026-04-14 09:45","actor":"the clerk","action":"create","order":"order 00405","amount":225.25},{"seq":16,"time":"2026-04-14 09:48","actor":"the cashier","action":"pay","order":"order 00405","amount":225.25},{"seq":17,"time":"2026-04-14 09:51","actor":"the dispatcher","action":"ship","order":"order 00405","amount":225.25},{"seq":18,"time":"2026-04-14 09:54","actor":"the clerk","action":"create","order":"order 00406","amount":262.5},{"seq":19,"time":"2026-04-14 09:57","actor":"the cashier","action":"pay","order":"order 00406","amount":262.5},{"seq":20,"time":"2026-04-14 10:00","actor":"the dispatcher","action":"ship","order":"order 00406","amount":262.5},{"seq":21,"time":"2026-04-14 10:03","actor":"the clerk","action":"refund","order":"order 00406","amount":640.0,"approved_by":"the clerk"},{"seq":22,"time":"2026-04-14 10:06","actor":"the clerk","action":"create","order":"order 00407","amount":299.75},{"seq":23,"time":"2026-04-14 10:09","actor":"the cashier","action":"pay","order":"order 00407","amount":299.75},{"seq":24,"time":"2026-04-14 10:12","actor":"the dispatcher","action":"ship","order":"order 00407","amount":299.75},{"seq":25,"time":"2026-04-14 10:15","actor":"the clerk","action":"create","order":"order 00408","amount":336.0},{"seq":26,"time":"2026-04-14 10:18","actor":"the cashier","action":"pay","order":"order 00408","amount":336.0},{"seq":27,"time":"2026-04-14 10:21","actor":"the dispatcher","action":"ship","order":"order 00408","amount":336.0},{"seq":28,"time":"2026-04-14 10:24","actor":"the clerk","action":"create","order":"order 00409","amount":373.25},{"seq":29,"time":"2026-04-14 10:27","actor":"the cashier","action":"pay","order":"order 00409","amount":373.25},{"seq":30,"time":"2026-04-14 10:30","actor":"the dispatcher","action":"ship","order":"order 00409","amount":373.25},{"seq":31,"time":"2026-04-14 10:33","actor":"the clerk","action":"refund","order":"order 00409","amount":30.0,"approved_by":"the supervisor"},{"seq":32,"time":"2026-04-14 10:36","actor":"the clerk","action":"refund","order":"order 00409","amount":45.0,"approved_by":"the supervisor"},{"seq":33,"time":"2026-04-14 10:39","actor":"the clerk","action":"create","order":"order 00410","amount":410.5},{"seq":34,"time":"2026-04-14 10:42","actor":"the cashier","action":"pay","order":"order 00410","amount":410.5},{"seq":35,"time":"2026-04-14 10:45","actor":"the dispatcher","action":"ship","order":"order 00410","amount":410.5},{"seq":36,"time":"2026-04-14 10:48","actor":"the clerk","action":"create","order":"order 00411","amount":47.75},{"seq":37,"time":"2026-04-14 10:51","actor":"the dispatcher","action":"ship","order":"order 00411","amount":47.75},{"seq":38,"time":"2026-04-14 10:54","actor":"the cashier","action":"pay","order":"order 00411","amount":47.75},{"seq":39,"time":"2026-04-14 10:57","actor":"the clerk","action":"create","order":"order 00412","amount":84.0},{"seq":40,"time":"2026-04-14 11:00","actor":"the cashier","action":"pay","order":"order 00412","amount":84.0},{"seq":41,"time":"2026-04-14 11:03","actor":"the dispatcher","action":"ship","order":"order 00412","amount":84.0},{"seq":42,"time":"2026-04-14 11:06","actor":"the clerk","action":"create","order":"order 00413","amount":121.25},{"seq":43,"time":"2026-04-14 11:09","actor":"the cashier","action":"pay","order":"order 00413","amount":121.25},{"seq":44,"time":"2026-04-14 11:12","actor":"the dispatcher","action":"ship","order":"order 00413","amount":121.25},{"seq":45,"time":"2026-04-14 11:15","actor":"the clerk","action":"create","order":"order 00414","amount":158.5},{"seq":46,"time":"2026-04-14 11:18","actor":"the clerk","action":"cancel","order":"order 00414","amount":158.5,"reason":"customer request"},{"seq":47,"time":"2026-04-14 11:21","actor":"the clerk","action":"create","order":"order 00415","amount":195.75},{"seq":48,"time":"2026-04-14 11:24","actor":"the cashier","action":"pay","order":"order 00415","amount":195.75},{"seq":49,"time":"2026-04-14 11:27","actor":"the dispatcher","action":"ship","order":"order 00415","amount":195.75},{"seq":50,"time":"2026-04-14 11:30","actor":"the clerk","action":"create","order":"order 00416","amount":232.0},{"seq":51,"time":"2026-04-14 11:33","actor":"the cashier","action":"pay","order":"order 00416","amount":232.0},{"seq":52,"time":"2026-04-14 11:36","actor":"the dispatcher","action":"ship","order":"order 00416","amount":232.0},{"seq":53,"time":"2026-04-14 11:39","actor":"the clerk","action":"create","order":"order 00417","amount":269.25},{"seq":54,"time":"2026-04-14 11:42","actor":"the cashier","action":"pay","order":"order 00417","amount":269.25},{"seq":55,"time":"2026-04-14 11:45","actor":"the dispatcher","action":"ship","order":"order 00417","amount":269.25}],"questions":{"refunded_twice":{"type":"choice","instructions":"Which order was refunded twice?","criteria":{"o00402":"order 00402","o00406":"order 00406","o00409":"order 00409","o00411":"order 00411"}},"big_refunds_approved":{"type":"noul","instructions":"Did the supervisor approve every refund of more than 500.00?"},"cancellations":{"type":"score","instructions":"How many orders were cancelled?","criteria":["None","One","Two","Three or more"]},"shipped_unpaid":{"type":"noul","instructions":"Was any order shipped before it was paid?"}}},"gold":{"refunded_twice":"o00409","big_refunds_approved":"false","cancellations":"2","shipped_unpaid":"true"},"note":"order ledger audit log, orders 00400-00417: 00409 refunded twice, 00406 refund 640.00 approved by the clerk, 00403 and 00414 cancelled, 00411 shipped before payment","provenance":{"written_for":"d1-3B Core AI port (round 1)","generator":"make_fixtures.order_ledger","n":18},"questions":[{"name":"refunded_twice","type":"choice","n_options":4,"keys":["o00402","o00406","o00409","o00411"],"codes":["A","B","C","D"],"gold":"o00409","row_len":3470,"slot":3469,"row_ids_sha256":"696d9d3c8e69903dc0cea4c0555e36973e229a2e03c9186742cf1abebb3352f1","oracle":{"probs":[0.006620247242538125,0.028214793258619086,0.9454612836421555,0.019703675856687267],"argmax":"o00409","top2_margin":0.9172464903835365,"near_tie":false,"group_logz":{"41":-5.025724411010742,"334":-13.427244186401367,"42":-3.5760107040405273,"378":-11.407052993774414,"43":-0.06418418884277344,"340":-8.154773712158203,"44":-3.935051918029785,"388":-11.985309600830078}},"graph":{"mac_fp16":{"p":[0.006621913887676964,0.028253888246782658,0.945413427633248,0.019710770232292448],"max_abs_dp":4.785600890755415e-05,"argmax_equal":true},"iphone_int8mlp":{"p":[0.006548491371802775,0.0282129720795555,0.9453875345450858,0.01985100200355589],"max_abs_dp":0.0001473261468686246,"argmax_equal":true}}},{"name":"big_refunds_approved","type":"noul","n_options":2,"keys":["yes","no"],"codes":null,"gold":"no","row_len":3451,"slot":3450,"row_ids_sha256":"627909af7afb945523533598255042b7616cdcc7c28708ba4a88f2722656b112","oracle":{"probs":[0.3646591211398083,0.6353408788601916],"argmax":"no","top2_margin":0.27068175772038333,"near_tie":false,"group_logz":{"11683":-1.0522632598876953,"12447":-4.949987411499023,"2243":-0.49706459045410156,"4547":-4.436689376831055,"19598":-8.377057075500488}},"graph":{"mac_fp16":{"p":[0.3649313417279422,0.6350686582720578],"max_abs_dp":0.00027222058813392014,"argmax_equal":true},"iphone_int8mlp":{"p":[0.36423932129217784,0.6357606787078222],"max_abs_dp":0.0004197998476305198,"argmax_equal":true}}},{"name":"cancellations","type":"score","n_options":4,"keys":["0","1","2","3"],"codes":null,"gold":"2","row_len":3461,"slot":3460,"row_ids_sha256":"fc9cc136da37695efee4e37ab2ae462c9fc0c4c5c705aaa746e8413e424ac125","oracle":{"probs":[0.020301401608427264,0.13139691903020606,0.827358478689852,0.020943200671514665],"argmax":"2","top2_margin":0.695961559659646,"near_tie":false,"group_logz":{"24":-3.9052562713623047,"25":-2.0377235412597656,"26":-0.1977081298828125,"27":-3.8741321563720703}},"graph":{"mac_fp16":{"p":[0.020315699478691867,0.13058775453680746,0.8281904557505189,0.02090609023398184],"max_abs_dp":0.0008319770606668442,"argmax_equal":true},"iphone_int8mlp":{"p":[0.020368790396786574,0.13963187989034426,0.8194771948768851,0.02052213483598403],"max_abs_dp":0.008234960860138196,"argmax_equal":true}}},{"name":"shipped_unpaid","type":"noul","n_options":2,"keys":["yes","no"],"codes":null,"gold":"yes","row_len":3446,"slot":3445,"row_ids_sha256":"0f48cd7c534e2fe175b1a154eb77240bf8896033ba91f33e4c8955a01cba7857","oracle":{"probs":[0.5187185502796423,0.48128144972035775],"argmax":"yes","top2_margin":0.037437100559284564,"near_tie":false,"group_logz":{"11683":-0.7175674438476562,"12447":-4.4075164794921875,"2243":-0.7924766540527344,"4547":-4.392104148864746,"19598":-8.033324241638184}},"graph":{"mac_fp16":{"p":[0.5185205755621437,0.4814794244378563],"max_abs_dp":0.00019797471749860573,"argmax_equal":true},"iphone_int8mlp":{"p":[0.5273011632909579,0.4726988367090421],"max_abs_dp":0.008582613011315632,"argmax_equal":true}}}],"api":{"path":"tree","probs":[[0.006620355432724561,0.028214447120391437,0.9454614062885373,0.01970379115834675],[0.3646569116428575,0.6353430883571425],[0.02030061961073947,0.13139386259644567,0.8273629041450911,0.02094261364772382],[0.5187261689769459,0.4812738310230541]],"input_tokens":3553,"response":{"answers":{"refunded_twice":{"type":"choice","choice":"o00409","confidence":0.9454614062885373,"probabilities":{"o00402":0.006620355432724561,"o00406":0.028214447120391437,"o00409":0.9454614062885373,"o00411":0.01970379115834675}},"big_refunds_approved":{"type":"noul","noul":0.3646569116428575},"cancellations":{"type":"score","score":1.8489475118297993,"confidence":0.8273629041450911,"probabilities":{"0":0.02030061961073947,"1":0.13139386259644567,"2":0.8273629041450911,"3":0.02094261364772382},"legend":{"0":"None","1":"One","2":"Two","3":"Three or more"}},"shipped_unpaid":{"type":"noul","noul":0.5187261689769459}},"usage":{"input_tokens":3553,"output_tokens":0}},"max_abs_dp_vs_rows":7.618697303646282e-06,"seconds":10.0035}}],"heldout":{"what":"120 transfer-v4 development records after the fixture's slices (the Core AI Kev lane's held-out file without its 10 tweet_offensive records): used to test a choice made on the fixture, never to make it; tv4h_28 renders to the fixture's tv4_002 row (a permuted row whose option order came out unchanged)","records":[{"id":"tv4h_00","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":60,"_meta":{"row":282,"text_sha256":"f764b3d9187ad0e0c6e98ca818eebbf3481f738e0f2e562377c125f27af714f2","row_sha256":"2048a5ee51c3949f2022371d25b2190325890748995426e82689d42e865c0268","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/282","group_id":"mmlu/test/282","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/282"}},"request_sha256":"0f7e8b937299bff682fc1a41ffed2de092331c2d14e11d2b47e7627646546395","questions":[{"type":"choice","n_options":4,"row_len":75,"slot":74,"row_ids_sha256":"dc4aac227c476cd8bfe917cace628b3ca8332c8879b6e088d3e6b2b0209ce63d","oracle":{"probs":[0.5941137438306521,0.06916600303340988,0.01511093945517948,0.32160931368075857],"argmax":"a","top2_margin":0.2725044301498935,"near_tie":false,"group_logz":{"73":-0.5812892913818359,"267":-7.740291595458984,"74":-2.7318506240844727,"293":-9.303801536560059,"75":-4.252941131591797,"280":-11.260841369628906,"76":-1.1950225830078125,"277":-7.27583122253418}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.588941742432498,0.06903605320906933,0.015215940208185081,0.3268062641502476],"max_abs_dp":0.005196950469489037,"argmax_equal":true}}}]},{"id":"tv4h_01","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":61,"_meta":{"row":8524,"text_sha256":"822364408ed0625c0b79c047029facb8b81fe5b5e08ad2add9a92236a5d5b269","row_sha256":"20e11bd9c0a720cb3a8f6acb69e9a3b4fefa0594c4c6d6317476c296ea756038","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8524","group_id":"mmlu/test/8524","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/8524"}},"request_sha256":"e266e6b939bed09914a4679df4840171fe4470e4668eaa926bf0777361075441","questions":[{"type":"choice","n_options":4,"row_len":149,"slot":148,"row_ids_sha256":"1a6974654e1ffcd80978a00e0ca670c52d683e16aa61f8ebb08b39c798f3d89d","oracle":{"probs":[0.0923852664948206,0.5679504927992093,0.28336695973523235,0.05629728097073769],"argmax":"b","top2_margin":0.28458353306397693,"near_tie":false,"group_logz":{"73":-2.4536304473876953,"267":-9.062616348266602,"74":-0.6375637054443359,"293":-6.909311294555664,"75":-1.332855224609375,"280":-7.418169975280762,"76":-2.9489517211914062,"277":-9.013233184814453}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.09164114590203562,0.566658910414059,0.2861863871100755,0.055513556573829774],"max_abs_dp":0.002819427374843142,"argmax_equal":true}}}]},{"id":"tv4h_02","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":62,"_meta":{"row":2715,"text_sha256":"58baa2441381d1ab03a94d50b013407342206bbe038af9f52b19e3755feeb54f","row_sha256":"2184b9fdf77f4a55d63eeb73ab996cf18d39f76749859d27707940163931cfb8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2715","group_id":"mmlu/test/2715","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/2715"}},"request_sha256":"60f6d3151ab4b4cf282b850e1be9a195c21455d5e7c4e1dee90ba79ce4e54eaa","questions":[{"type":"choice","n_options":4,"row_len":92,"slot":91,"row_ids_sha256":"34c99df49df04fb0169dfc32486838a3831c1f902d24e6c3f85b188274974ed1","oracle":{"probs":[0.8277842101909693,0.05620724720095822,0.008132665545318461,0.10787587706275406],"argmax":"a","top2_margin":0.7199083331282152,"near_tie":false,"group_logz":{"73":-0.25228118896484375,"267":-7.597699165344238,"74":-2.941987991333008,"293":-9.443411827087402,"75":-4.875144958496094,"280":-11.843482971191406,"76":-2.2900524139404297,"277":-8.23355484008789}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.826224630914642,0.0565707915996366,0.008131401084363818,0.10907317640135752],"max_abs_dp":0.0015595792763272165,"argmax_equal":true}}}]},{"id":"tv4h_03","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":63,"_meta":{"row":7350,"text_sha256":"4c6704fa4aecbfc95541b77dafcbfea7dac268e7aafcca3031e7ae007f5651d0","row_sha256":"2202beac91b1bdbc4e4a3048b09c6bc85f42db942a7b2afd39652b4f6055cc3d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/7350","group_id":"mmlu/test/7350","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/7350"}},"request_sha256":"0445128d68a467c3904f844a9b9f69b7ba39ceec2d4915832786a2f6d1badf33","questions":[{"type":"choice","n_options":4,"row_len":84,"slot":83,"row_ids_sha256":"114fb98ed9776e559007fc96b6766f63d772b0dae9d388d7efb66793516112d9","oracle":{"probs":[0.004756522388211852,0.9319962719364793,0.033939836496138714,0.029307369179170206],"argmax":"b","top2_margin":0.8980564354403405,"near_tie":false,"group_logz":{"73":-5.413098335266113,"267":-12.769617080688477,"74":-0.1352863311767578,"293":-7.036298751831055,"75":-3.448025703430176,"280":-10.382291793823242,"76":-3.594776153564453,"277":-10.55056095123291}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.00478788843585567,0.9314443434832007,0.03437122196299028,0.029396546117953275],"max_abs_dp":0.0005519284532785607,"argmax_equal":true}}}]},{"id":"tv4h_04","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":64,"_meta":{"row":4527,"text_sha256":"fa2936944c569913a16b3b985b605e5ad0bb3b42d27eea2a892d448bfbe0ed57","row_sha256":"224bf654f98530f4b7096d42d6917feb13dfbdf561cb6a377b129a4b414a210c","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4527","group_id":"mmlu/test/4527","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4527"}},"request_sha256":"8af5ead63bf6a9868a2e1d6183f5d8c4dffd154f16b5501f53bf498a4d3221bc","questions":[{"type":"choice","n_options":4,"row_len":154,"slot":153,"row_ids_sha256":"d698afbcfc2c681be14e9b9d3d7cab176bfc56225b7b34605ce0280574bdc4b9","oracle":{"probs":[0.00229417663976656,0.006504057972031639,0.98566780140054,0.0055339639876617365],"argmax":"c","top2_margin":0.9791637434285083,"near_tie":false,"group_logz":{"73":-6.112580299377441,"267":-13.029221534729004,"74":-5.070528030395508,"293":-12.008216857910156,"75":-0.04963493347167969,"280":-6.912878036499023,"76":-5.232049942016602,"277":-11.766239166259766}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0022755403302878925,0.006427698658536562,0.9858174560772702,0.005479304933905249],"max_abs_dp":0.00014965467673022648,"argmax_equal":true}}}]},{"id":"tv4h_05","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":65,"_meta":{"row":6364,"text_sha256":"7371f764a68ff91e345aba45d7bfb1528e39c959702df4406faea501c20fa2cf","row_sha256":"225b3e9b8e5698a9c1631bdfab0fb9b31a57be67b50b55cdd661b8c3e8cf279f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6364","group_id":"mmlu/test/6364","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6364"}},"request_sha256":"8817eb522ed5973bf24bb41ae15bf97507bee2d8adcce5bd1e44857138191b88","questions":[{"type":"choice","n_options":4,"row_len":72,"slot":71,"row_ids_sha256":"9925985f7c5698ebee8896b2eeedaf93efbafe41963428e9c7f3d0409635856f","oracle":{"probs":[0.17900074420258782,0.1586498571508276,0.08509247219408547,0.5772569264524992],"argmax":"d","top2_margin":0.39825618224991133,"near_tie":false,"group_logz":{"73":-1.801051139831543,"267":-8.963922500610352,"74":-1.9217414855957031,"293":-8.728069305419922,"75":-2.5447025299072266,"280":-9.444324493408203,"76":-0.6301536560058594,"277":-7.313106536865234}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.1756712386970967,0.15774862260493794,0.08276066104070721,0.5838194776572582],"max_abs_dp":0.006562551204759015,"argmax_equal":true}}}]},{"id":"tv4h_06","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":66,"_meta":{"row":4563,"text_sha256":"5331cc1333fb17a37a60d99714e5693ae650cae7f65955370422a480a217f7c8","row_sha256":"23f533cd5a96dd4943b45ee9c63c10f80bbcd70bef2db77925ca499bf52d1cf5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4563","group_id":"mmlu/test/4563","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4563"}},"request_sha256":"bc6dace359f820365df9fed905084d34d9cc216020f8757018fa2c90999dd166","questions":[{"type":"choice","n_options":4,"row_len":105,"slot":104,"row_ids_sha256":"dca6a174b2a5c2ece5db71ff3130a5284d138afe77276326708d934814165f74","oracle":{"probs":[0.45400966428333017,0.036319229660865336,0.04099224570197192,0.46867886035383266],"argmax":"d","top2_margin":0.014669196070502488,"near_tie":true,"group_logz":{"73":-0.8924026489257812,"267":-7.7000732421875,"74":-3.4181737899780273,"293":-9.900932312011719,"75":-3.297138214111328,"280":-9.911636352539062,"76":-0.8606033325195312,"277":-6.788022994995117}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.4573483614953067,0.037471284445067206,0.041874844721000906,0.46330550933862524],"max_abs_dp":0.005373351015207417,"argmax_equal":true}}}]},{"id":"tv4h_07","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":67,"_meta":{"row":4441,"text_sha256":"9f80c384e23e619a896a7a680e54dffae97e8e03de73c0ad2094a9032227371e","row_sha256":"241ad0a238016dd3f459f6e5468b9ba43b2d14d94e5945ff512733bea84c7283","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4441","group_id":"mmlu/test/4441","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4441"}},"request_sha256":"258fec54030942c62ee3e9debb5d6cb0459d31abc402d56435f8617680569bb1","questions":[{"type":"choice","n_options":4,"row_len":118,"slot":117,"row_ids_sha256":"93c0d5d6fcbb98a178e354f0d0cfa4830592bd37700ff5d0981b4a761fb2b565","oracle":{"probs":[0.09851357900553796,0.49853671902777613,0.3101788085721972,0.09277089339448864],"argmax":"b","top2_margin":0.18835791045557893,"near_tie":false,"group_logz":{"73":-2.41147518157959,"267":-8.894170761108398,"74":-0.7899923324584961,"293":-7.090959548950195,"75":-1.2645206451416016,"280":-7.387811660766602,"76":-2.471536636352539,"277":-8.376111030578613}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.09768403504170477,0.497915470802628,0.3120795877886884,0.09232090636697884],"max_abs_dp":0.0019007792164911774,"argmax_equal":true}}}]},{"id":"tv4h_08","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":68,"_meta":{"row":6910,"text_sha256":"08add8ef339b19f632d4cd3ed1c45f824889679fc9e7268120b865e29107b853","row_sha256":"243c6556acdb14a65b6e210646059f3904579d224d730ff3e1db1854006fc802","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6910","group_id":"mmlu/test/6910","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6910"}},"request_sha256":"f89bea14bb2b1bd30830f30ed2c0c66302ee0bc465727f91286f7781ba24a0b4","questions":[{"type":"choice","n_options":4,"row_len":104,"slot":103,"row_ids_sha256":"37311e92bc7f5babc15645f9b033da346b22e39640158e86908102c306a10a72","oracle":{"probs":[0.13254118395248538,0.2324881682850865,0.5138679853224705,0.12110266243995756],"argmax":"c","top2_margin":0.28137981703738396,"near_tie":false,"group_logz":{"73":-2.09234619140625,"267":-8.835479736328125,"74":-1.530400276184082,"293":-7.871246337890625,"75":-0.7372732162475586,"280":-7.146744728088379,"76":-2.182600975036621,"277":-8.04798698425293}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.1301439957530943,0.234513125573745,0.5149461362209469,0.12039674245221377],"max_abs_dp":0.0023971881993910815,"argmax_equal":true}}}]},{"id":"tv4h_09","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":69,"_meta":{"row":8800,"text_sha256":"811bf5bf013a07b39b29b5f0e44cf591f87cd435de584630cfdc16120e0258d6","row_sha256":"24905e1af1615c518414f5cd372639176ddf690d141547b1ac73c59abc4592a5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8800","group_id":"mmlu/test/8800","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/8800"}},"request_sha256":"42c6ad8e242f74554ed22e0ddd274805fd87a92da387f5ec28ac042e0fe60a17","questions":[{"type":"choice","n_options":4,"row_len":140,"slot":139,"row_ids_sha256":"a42ed4b38135a0cd6b1363f223389cd561ab25a22c58b3b5a439952ad774fb7f","oracle":{"probs":[0.36033198472865074,0.4490146957840896,0.17865647219850064,0.011996847288758996],"argmax":"b","top2_margin":0.08868271105543885,"near_tie":false,"group_logz":{"73":-1.1037139892578125,"267":-8.442181587219238,"74":-0.8836841583251953,"293":-7.432631492614746,"75":-1.8052749633789062,"280":-8.418230056762695,"76":-4.506095886230469,"277":-11.296646118164062}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.364912631121087,0.4427083692474893,0.18059441720153546,0.0117845824298882],"max_abs_dp":0.006306326536600293,"argmax_equal":true}}}]},{"id":"tv4h_10","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":70,"_meta":{"row":4472,"text_sha256":"e74dabb87afea0a70e21f777eeaf42a44d31d8d97f43f2d08389fe444f57a4de","row_sha256":"251010676810510fb50874da455582cb17fe90795b76223c2a0dd62d52d95724","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4472","group_id":"mmlu/test/4472","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4472"}},"request_sha256":"ead6616814d2a0e7c3288962d8bed09f624a2f0024c69e376b8d78e0b217b3a2","questions":[{"type":"choice","n_options":4,"row_len":177,"slot":176,"row_ids_sha256":"f56465a0ffaa88fb28428867b792d94807dd27664f1e55b24f323d02dfad1276","oracle":{"probs":[0.011062505710764562,0.3189151045762283,0.658398205879239,0.01162418383376811],"argmax":"c","top2_margin":0.33948310130301074,"near_tie":false,"group_logz":{"73":-4.584745407104492,"267":-11.761724472045898,"74":-1.2233819961547852,"293":-7.860626220703125,"75":-0.49849700927734375,"280":-7.218578338623047,"76":-4.535219192504883,"277":-11.191488265991211}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.01101419543135818,0.32292426220641374,0.6544395173532558,0.01162202500897234],"max_abs_dp":0.004009157630185445,"argmax_equal":true}}}]},{"id":"tv4h_11","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":71,"_meta":{"row":5801,"text_sha256":"3796fbb987fe807698dcb5b074df04ea08fb2286ddba3337c78225e1c48e4367","row_sha256":"258e79933164b8b9c20d756f1819cac2f8c79959d082348e5b605da73f607886","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5801","group_id":"mmlu/test/5801","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/5801"}},"request_sha256":"211ec7330d4a1752308af75f473989909622031b67722e8446880fb36f974e54","questions":[{"type":"choice","n_options":4,"row_len":319,"slot":318,"row_ids_sha256":"f0002b23da51e50ced4015056c44d2b5d6f70405ede63fb39ff6878a358be003","oracle":{"probs":[0.0023226232422833017,0.023134811155902987,0.009698524118101487,0.9648440414837123],"argmax":"d","top2_margin":0.9417092303278093,"near_tie":false,"group_logz":{"73":-6.1072282791137695,"267":-13.943389892578125,"74":-3.808587074279785,"293":-11.392040252685547,"75":-4.677951812744141,"280":-12.48196792602539,"76":-0.07795906066894531,"277":-7.615072250366211}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.002299719909247472,0.022905894049882945,0.009582822808473565,0.965211563232396],"max_abs_dp":0.0003675217486837079,"argmax_equal":true}}}]},{"id":"tv4h_12","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":72,"_meta":{"row":8718,"text_sha256":"a32bd31ba4a0d8b9564f68c3f60078adb94db71005b4df272123d4eee902801a","row_sha256":"25a1f2b97bec0c728d9151e77cd9765acbdbff78d4d76321dc548d73d67b397a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8718","group_id":"mmlu/test/8718","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/8718"}},"request_sha256":"ac03db9580db0f7bc5f56c99984352575cf421f5ef80e9784be61cb18dbf2095","questions":[{"type":"choice","n_options":4,"row_len":151,"slot":150,"row_ids_sha256":"72ba736806e29a19a5c2d4cedf696e0dc2a271de66a80c9dfa5dc1f11cae8ae4","oracle":{"probs":[0.18729757664045962,0.07786517394153215,0.7249106143123635,0.009926635105644746],"argmax":"c","top2_margin":0.5376130376719038,"near_tie":false,"group_logz":{"73":-1.719207763671875,"267":-8.774664878845215,"74":-2.5969276428222656,"293":-9.095833778381348,"75":-0.3658580780029297,"280":-7.109143257141113,"76":-4.656684875488281,"277":-11.231391906738281}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.19635222671625063,0.07941467731280254,0.7142389867539709,0.009994109216975975],"max_abs_dp":0.010671627558392571,"argmax_equal":true}}}]},{"id":"tv4h_13","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":73,"_meta":{"row":3557,"text_sha256":"198dd2b76fe680e1dacbd1335668b2cf4cb3d4f9fffc588672e7d77a07901af8","row_sha256":"26845c1670c6e75a2d4a6f5cedb8efe497d67a7b7731c69be410252e96d3decc","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3557","group_id":"mmlu/test/3557","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/3557"}},"request_sha256":"5318e1eaad4789c3284d58edd66610cf7750e411f5ffe57d7e162bbbf338ea3e","questions":[{"type":"choice","n_options":4,"row_len":90,"slot":89,"row_ids_sha256":"e95fc5879e0155b4d146885a6b0770149f51e1c8692bfe726122e3d8fa145134","oracle":{"probs":[0.003067750340081722,0.011435038896916432,0.9807041751564923,0.004793035606509532],"argmax":"c","top2_margin":0.9692691362595759,"near_tie":false,"group_logz":{"73":-5.824148178100586,"267":-13.350852966308594,"74":-4.508410453796387,"293":-11.825220108032227,"75":-0.05682182312011719,"280":-7.56239128112793,"76":-5.377928733825684,"277":-12.526159286499023}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.003043134440298008,0.0114103298146882,0.9807667455263542,0.004779790218659511],"max_abs_dp":6.25703698619029e-05,"argmax_equal":true}}}]},{"id":"tv4h_14","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":74,"_meta":{"row":13887,"text_sha256":"ad12a30dc2689720958e98441b0bec2fd12b917195620d41c3d708b9952ad1de","row_sha256":"280a540cd9fecd12ca41f8a28ecbd2a22f9f99dee5bb95f89913b8b5faf5c874","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13887","group_id":"mmlu/test/13887","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13887"}},"request_sha256":"b3cce23ddc98e8b34d655af823000f767cf5a9454a377cb252035893e7f40f56","questions":[{"type":"choice","n_options":4,"row_len":76,"slot":75,"row_ids_sha256":"36f4f375f0136afe4aa11b835ecd657b04a112e5dd3ea3a96be76013ae348b3b","oracle":{"probs":[0.0358112964915383,0.02421374074314439,0.922692810117096,0.01728215264822127],"argmax":"c","top2_margin":0.8868815136255577,"near_tie":false,"group_logz":{"73":-3.386707305908203,"267":-10.098043441772461,"74":-3.778050422668457,"293":-10.36939811706543,"75":-0.13767433166503906,"280":-6.808435440063477,"76":-4.115296363830566,"277":-10.421195983886719}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.03571887050505514,0.024002448683204225,0.9232190234505188,0.017059657361221805],"max_abs_dp":0.0005262133334227315,"argmax_equal":true}}}]},{"id":"tv4h_15","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":75,"_meta":{"row":8047,"text_sha256":"e52382280110bfcd67d3a91beccfadf2876e36a576891d8a75b3d9961b4d0449","row_sha256":"28254f59ed7f5a97aa710645d9ab88f8094ef152d148725988eee0ef5c1ac4f3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8047","group_id":"mmlu/test/8047","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/8047"}},"request_sha256":"09a9da413d025c073369300d72bd6aceecb66672e6b0313e117e58600c162379","questions":[{"type":"choice","n_options":4,"row_len":72,"slot":71,"row_ids_sha256":"9eb9050c4c4ff0f09ff4308a3ff247d6380857fbf5c43a48541f2c95a595296c","oracle":{"probs":[0.06179311853745182,0.0048803634587110305,0.9229165358580108,0.010409982145826465],"argmax":"c","top2_margin":0.861123417320559,"near_tie":false,"group_logz":{"73":-2.8295440673828125,"267":-9.79419994354248,"74":-5.36811637878418,"293":-12.271316528320312,"75":-0.12579727172851562,"280":-6.963082313537598,"76":-4.610570907592773,"277":-10.936197280883789}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0629027321385085,0.004875581269954386,0.9218682508302396,0.010353435761297514],"max_abs_dp":0.001109613601056679,"argmax_equal":true}}}]},{"id":"tv4h_16","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":76,"_meta":{"row":9900,"text_sha256":"8375b63d6a821982197076188109d3204c7e6732d1d3ff9b7bb610145fef13c0","row_sha256":"293e97cddb795f8c71f60dfa1b09e9c57613ddef1b10b9e9f273f84c6cbad0e1","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9900","group_id":"mmlu/test/9900","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/9900"}},"request_sha256":"525640c6879e20525cd9baf5e12a6b694d57fb4044ae3bcc78f4fa5bfbe27a00","questions":[{"type":"choice","n_options":4,"row_len":90,"slot":89,"row_ids_sha256":"3ead04206cda7948203c9a78f0ed349f1d3a060e7fdd7d4d01eec87abe63b9b3","oracle":{"probs":[0.025164891901758624,0.8769947310871965,0.026279184874497683,0.07156119213654723],"argmax":"b","top2_margin":0.8054335389506493,"near_tie":false,"group_logz":{"73":-3.7343645095825195,"267":-10.52603816986084,"74":-0.18331336975097656,"293":-6.90655517578125,"75":-3.691037178039551,"280":-10.320390701293945,"76":-2.6892614364624023,"277":-8.959248542785645}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.02478189456589462,0.8771615699171982,0.026193990562148786,0.07186254495475834],"max_abs_dp":0.0003829973358640028,"argmax_equal":true}}}]},{"id":"tv4h_17","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":77,"_meta":{"row":12517,"text_sha256":"f2f9b485c469f4be1682eb28ca7e240e67c830cd1c0cff7138ee98b090fccf3f","row_sha256":"29bbf79ec6a3ae443184c74b2792f0167e7745df3c7c59ef3ef2388cd14ba690","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12517","group_id":"mmlu/test/12517","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/12517"}},"request_sha256":"1d41a5bf56a4882d482cc17d1f73402aa6ee95dcf8372129a2a8743c2a2e53c7","questions":[{"type":"choice","n_options":4,"row_len":79,"slot":78,"row_ids_sha256":"cfb1908fc2e01e8c4fb9e4911af88b3d18b6b1f4cb72dc40505819d8202cb277","oracle":{"probs":[0.010706747922029547,0.640864425876073,0.3368382184080602,0.011590607793837325],"argmax":"b","top2_margin":0.30402620746801273,"near_tie":false,"group_logz":{"73":-4.619752883911133,"267":-11.343385696411133,"74":-0.5278091430664062,"293":-6.94038200378418,"75":-1.1710243225097656,"280":-7.377918243408203,"76":-4.540431976318359,"277":-10.892035484313965}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.010699654297704529,0.6393958306016488,0.33832734658776953,0.011577168512877094],"max_abs_dp":0.0014891281797093092,"argmax_equal":true}}}]},{"id":"tv4h_18","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":78,"_meta":{"row":6949,"text_sha256":"257ad45a0015775345cabbacaa6a55bfcbff4f409fb831ba6f533bed3ccbbb38","row_sha256":"2c346d2340c7eebe04b1ac82f63e627335efcdaa19acdece5f30ee104c67cb6e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6949","group_id":"mmlu/test/6949","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6949"}},"request_sha256":"5942a5cf8610f27bf8a3f6972e2f6c713bb17adcbb3c882739cfede2a051f721","questions":[{"type":"choice","n_options":4,"row_len":76,"slot":75,"row_ids_sha256":"6798b3d2b73f59e14f008691df9ed26f6dc7e88c8273eeda9b9e35152dc7a983","oracle":{"probs":[0.0038706316467671436,0.02616193333608894,0.9642490849516011,0.005718350065542731],"argmax":"c","top2_margin":0.9380871516155121,"near_tie":false,"group_logz":{"73":-5.604439735412598,"267":-12.4238920211792,"74":-3.693552017211914,"293":-10.30404281616211,"75":-0.08650779724121094,"280":-7.016750335693359,"76":-5.214177131652832,"277":-11.95371150970459}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.003808058715433485,0.02635264284490475,0.9641467374790947,0.005692560960566904],"max_abs_dp":0.00019070950881580964,"argmax_equal":true}}}]},{"id":"tv4h_19","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":79,"_meta":{"row":4674,"text_sha256":"f72f2f388107a78803b8f5852f79102fe5d0884cd4d62e91641e86443db15aa0","row_sha256":"2c549e4d1c079994107ca839d795cff8fa4ec4915fe76db3404085878da3b0fe","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4674","group_id":"mmlu/test/4674","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4674"}},"request_sha256":"89b686c2df17cbd02a048466b8d659a82b59446ed992dafc6f7098a02801593a","questions":[{"type":"choice","n_options":4,"row_len":116,"slot":115,"row_ids_sha256":"ba0c1575a580848bff054aa30db39cfdcd4cd2877e5b713f011b00262850e5c3","oracle":{"probs":[0.03626846922066802,0.11838089506711265,0.04411429417207352,0.8012363415401459],"argmax":"d","top2_margin":0.6828554464730332,"near_tie":false,"group_logz":{"73":-3.3842391967773438,"267":-10.929543495178223,"74":-2.2012805938720703,"293":-9.240055084228516,"75":-3.188404083251953,"280":-10.365738868713379,"76":-0.289031982421875,"277":-7.150590896606445}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.03668089677126874,0.11826636048892891,0.04410393496540225,0.8009488077744001],"max_abs_dp":0.00041242755060071773,"argmax_equal":true}}}]},{"id":"tv4h_20","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":480,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/none_present","group_id":"mmlu/test/13292","variant":"none_present","parent_id":"mmlu/test/13292","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13292/none_present"}},"request_sha256":"1931f099f369d2a517d4925794a7b7b5f06bb09ecfbec3e756fc09a743462669","questions":[{"type":"choice","n_options":5,"row_len":115,"slot":114,"row_ids_sha256":"b44c010c1b637a51d7b68ed6abff0c383cfff7e03399bb99d3ef123f49b1a574","oracle":{"probs":[0.5576379895704132,0.060215080335222125,0.3149936664727181,0.01567229489131268,0.0514809687303339],"argmax":"c","top2_margin":0.24264432309769512,"near_tie":false,"group_logz":{"41":-0.6015892028808594,"334":-8.296133995056152,"42":-2.827376365661621,"378":-10.194496154785156,"43":-1.1727466583251953,"340":-8.523393630981445,"44":-4.173404693603516,"388":-11.759851455688477,"45":-2.9840869903564453,"407":-10.256380081176758}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.5561649618535801,0.06030553604656586,0.31659730300855804,0.015678922761658512,0.051253276329637415],"max_abs_dp":0.0016036365358399474,"argmax_equal":true}}}]},{"id":"tv4h_21","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":481,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/none_absent","group_id":"mmlu/test/13292","variant":"none_absent","parent_id":"mmlu/test/13292","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13292/none_absent"}},"request_sha256":"b0f9fe864bdcd2cc5b7324810f7e7e7c3a08c3098c4a77b5b8a769f8dcb3dad7","questions":[{"type":"choice","n_options":4,"row_len":107,"slot":106,"row_ids_sha256":"9d75488d8928dcc433d321d8c8fc8e323fee05d4321056eae40c0e53d21fe863","oracle":{"probs":[0.29264973927238563,0.02188721376361698,0.6103207902314806,0.07514225673251693],"argmax":"none_of_these","top2_margin":0.3176710509590949,"near_tie":false,"group_logz":{"41":-1.2455024719238281,"334":-8.685674667358398,"42":-3.838576316833496,"378":-11.252714157104492,"43":-0.5104942321777344,"340":-7.818120002746582,"44":-2.605095863342285,"388":-9.760634422302246}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.29304940301451404,0.02183279123329441,0.6092867755852083,0.07583103016698331],"max_abs_dp":0.0010340146462722855,"argmax_equal":true}}}]},{"id":"tv4h_22","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":482,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/permuted","group_id":"mmlu/test/13292","variant":"permuted","parent_id":"mmlu/test/13292"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13292/permuted"}},"request_sha256":"a48ef8781d96d0b1c37fc8d2a3cb35120a6e9a5edcd9b27b0e49c90fc2ff0a70","questions":[{"type":"choice","n_options":4,"row_len":106,"slot":105,"row_ids_sha256":"0c42fabb152b02c0a7dd38a4be77446665ac10a1ce2f91c4a67252905f7e1eef","oracle":{"probs":[0.016504695289753447,0.9402370721791271,0.008830645789323055,0.034427586741796294],"argmax":"c","top2_margin":0.9058094854373309,"near_tie":false,"group_logz":{"73":-4.146039009094238,"267":-11.283683776855469,"75":-0.10355186462402344,"280":-7.171601295471191,"76":-4.771455764770508,"277":-11.397255897521973,"74":-3.410825729370117,"293":-10.39753246307373}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.016709213493661418,0.9398581549009105,0.008889287768296537,0.034543343837131625],"max_abs_dp":0.00037891727821659504,"argmax_equal":true}}}]},{"id":"tv4h_23","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":483,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/none_present","group_id":"mmlu/test/588","variant":"none_present","parent_id":"mmlu/test/588","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/588/none_present"}},"request_sha256":"4c7176396801a8171309e56e80f2eb12a9a7083b7c35e0d34e03dbc914310a4e","questions":[{"type":"choice","n_options":5,"row_len":86,"slot":85,"row_ids_sha256":"d723d03bdb485850664d656b81b479704b24e37c2e24c1e4a1cc27ef4ed5c804","oracle":{"probs":[0.006198270628026666,0.9521410863806014,0.004909173191645951,0.03285580514163556,0.0038956646580904676],"argmax":"c","top2_margin":0.9192852812389658,"near_tie":false,"group_logz":{"41":-5.096433639526367,"334":-13.167882919311523,"42":-0.06199073791503906,"378":-7.798198699951172,"43":-5.329598426818848,"340":-13.099838256835938,"44":-3.4285755157470703,"388":-10.926979064941406,"45":-5.560839653015137,"407":-12.914138793945312}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.006171258926519459,0.9527190304267883,0.004888651781759761,0.03232776873972545,0.003893290125207084],"max_abs_dp":0.0005779440461869401,"argmax_equal":true}}}]},{"id":"tv4h_24","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":484,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/none_absent","group_id":"mmlu/test/588","variant":"none_absent","parent_id":"mmlu/test/588","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/588/none_absent"}},"request_sha256":"54803f868c5b9e4280f001f283418d874be887004a9a01ea3aecdbceb06e14c6","questions":[{"type":"choice","n_options":4,"row_len":78,"slot":77,"row_ids_sha256":"2a9b45be7d6679b16a629ff7fa98f0bc1e69e341a371719ef944cb0ebfc1a60a","oracle":{"probs":[0.008008062297765529,0.9668903948204612,0.009969540700750942,0.015132002181022336],"argmax":"none_of_these","top2_margin":0.951758392639439,"near_tie":false,"group_logz":{"41":-4.835271835327148,"334":-13.139309883117676,"42":-0.04163551330566406,"378":-8.071279525756836,"43":-4.616186141967773,"340":-12.364875793457031,"44":-4.198908805847168,"388":-12.043257713317871}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.008076604501292984,0.9669171794089516,0.009895077273492996,0.015111138816262375],"max_abs_dp":7.446342725794634e-05,"argmax_equal":true}}}]},{"id":"tv4h_25","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":485,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/permuted","group_id":"mmlu/test/588","variant":"permuted","parent_id":"mmlu/test/588"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/588/permuted"}},"request_sha256":"3993d58af1adad42c0a57c7463164412b440f8f8fdcde5d8f6d92a22ad3d8123","questions":[{"type":"choice","n_options":4,"row_len":77,"slot":76,"row_ids_sha256":"aafe909aa9dc468f0f58d1411008c8bcb7f7007f7c5b0dbf62a232559c3532d1","oracle":{"probs":[0.9913806461824427,0.003466733746627034,0.0016089983686143857,0.0035436217023159956],"argmax":"c","top2_margin":0.9878370244801267,"near_tie":false,"group_logz":{"75":-0.0415496826171875,"280":-8.122518539428711,"76":-5.69743537902832,"277":-13.044000625610352,"73":-6.465036392211914,"267":-14.1759614944458,"74":-5.675498962402344,"293":-13.592350006103516}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9914117799171832,0.0034571876279489906,0.0015962257557997723,0.0035348066990680536],"max_abs_dp":3.113373474050718e-05,"argmax_equal":true}}}]},{"id":"tv4h_26","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":486,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/none_present","group_id":"mmlu/test/4589","variant":"none_present","parent_id":"mmlu/test/4589","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4589/none_present"}},"request_sha256":"0c8917f790e3bad0dded73be7ac2db693a412dfdde3eb7474355c37d30ad2dd8","questions":[{"type":"choice","n_options":5,"row_len":119,"slot":118,"row_ids_sha256":"ffe22472ce80fe401e5898731110f75b04c2da5668436e580ed5e4271851c980","oracle":{"probs":[0.6623268064595123,0.32166267159567985,0.0032731878964788776,0.009825170236552806,0.0029121638117761556],"argmax":"a","top2_margin":0.3406641348638324,"near_tie":false,"group_logz":{"41":-0.4257030487060547,"334":-8.200569152832031,"42":-1.147958755493164,"378":-8.54784107208252,"43":-5.7356977462768555,"340":-13.62796688079834,"44":-4.636514663696289,"388":-12.25483226776123,"45":-5.852565765380859,"407":-13.525028228759766}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.6574275117255801,0.3265139589102911,0.0032354845031444535,0.009923857239939887,0.002899187621044515],"max_abs_dp":0.004899294733932136,"argmax_equal":true}}}]},{"id":"tv4h_27","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":487,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/none_absent","group_id":"mmlu/test/4589","variant":"none_absent","parent_id":"mmlu/test/4589","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4589/none_absent"}},"request_sha256":"d4f07540a6564d850c025fe96da079ad3505fa8f0da91848f1c43f3474e65d30","questions":[{"type":"choice","n_options":4,"row_len":108,"slot":107,"row_ids_sha256":"4b04cc675010c3cf43c4df0a66dc5114b71ff16101a85c0d8187bfe311146d89","oracle":{"probs":[0.045650124951670534,0.028871735609068087,0.023157159300727752,0.9023209801385336],"argmax":"none_of_these","top2_margin":0.856670855186863,"near_tie":false,"group_logz":{"41":-3.1003503799438477,"334":-10.718819618225098,"42":-3.5584936141967773,"378":-10.926464080810547,"43":-3.779052734375,"340":-11.284795761108398,"44":-0.11638641357421875,"388":-7.672341346740723}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.04392282226044497,0.029088437674688678,0.02322825873555679,0.9037604813293095],"max_abs_dp":0.001727302691225563,"argmax_equal":true}}}]},{"id":"tv4h_28","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":488,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/permuted","group_id":"mmlu/test/4589","variant":"permuted","parent_id":"mmlu/test/4589"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/4589/permuted"}},"request_sha256":"a395a1856eb321f102af71d4807ff0f0f682a264315d40972f4df31a35ed7525","questions":[{"type":"choice","n_options":4,"row_len":110,"slot":109,"row_ids_sha256":"ce0f3d6ca3790d5929bc2fdf6c2eb005d77d876ee28e783b775009086a17a724","oracle":{"probs":[0.9731359394576886,0.0052661614276986535,0.01420519113840435,0.007392707976208357],"argmax":"a","top2_margin":0.9589307483192843,"near_tie":false,"group_logz":{"73":-0.06461334228515625,"267":-8.253935813903809,"74":-5.283835411071777,"293":-12.700796127319336,"75":-4.291529655456543,"280":-11.565847396850586,"76":-4.944643020629883,"277":-11.656245231628418}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9727798744662309,0.00526216123810256,0.014503901808074881,0.007454062487591588],"max_abs_dp":0.000356064991457683,"argmax_equal":true}}}]},{"id":"tv4h_29","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":489,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/none_present","group_id":"mmlu/test/6099","variant":"none_present","parent_id":"mmlu/test/6099","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6099/none_present"}},"request_sha256":"0acec0a04adb05df81104e9b32d3bcdc8112bfd52c0d41955c8f94420f4ad2c5","questions":[{"type":"choice","n_options":5,"row_len":103,"slot":102,"row_ids_sha256":"88a60aa8e82036bb9ec902e8683c796e8eee6a4ffc82e69bbebe38ebc55c4f8f","oracle":{"probs":[0.6711749729901756,0.02177105184698713,0.011503663837159568,0.021564660596729304,0.2739856507289484],"argmax":"a","top2_margin":0.39718932226122716,"near_tie":false,"group_logz":{"41":-0.4156818389892578,"334":-8.245668411254883,"42":-3.844130516052246,"378":-11.291463851928711,"43":-4.482046127319336,"340":-11.946892738342285,"44":-3.8536558151245117,"388":-11.28559684753418,"45":-1.311635971069336,"407":-8.481050491333008}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.6727095336449923,0.021616422003809914,0.011557126822594953,0.021339477467224956,0.27277744006137794],"max_abs_dp":0.001534560654816719,"argmax_equal":true}}}]},{"id":"tv4h_30","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":490,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/none_absent","group_id":"mmlu/test/6099","variant":"none_absent","parent_id":"mmlu/test/6099","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6099/none_absent"}},"request_sha256":"1d56f6f1605a70a81326543b5400655e6869183f29fa813fba586d994d72d837","questions":[{"type":"choice","n_options":4,"row_len":92,"slot":91,"row_ids_sha256":"70740d898ab31897abee4f50a915dc4fe3a32cbf0bdca65a8dce882b5ce0771d","oracle":{"probs":[0.07258256451584034,0.042656719775553145,0.8673870565314524,0.01737365917715408],"argmax":"none_of_these","top2_margin":0.7948044920156121,"near_tie":false,"group_logz":{"41":-2.6412487030029297,"334":-9.858709335327148,"42":-3.172788619995117,"378":-10.021203994750977,"43":-0.16048812866210938,"340":-7.303336143493652,"44":-4.071018218994141,"388":-11.18979263305664}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.07248099084276514,0.04322782991443433,0.8667796671711294,0.017511512071671106],"max_abs_dp":0.0006073893603230074,"argmax_equal":true}}}]},{"id":"tv4h_31","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":491,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/permuted","group_id":"mmlu/test/6099","variant":"permuted","parent_id":"mmlu/test/6099"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6099/permuted"}},"request_sha256":"3909102a08f46ebb402bfaaeddc5a60471ca2bd166df3b46c482b82e2952d2e5","questions":[{"type":"choice","n_options":4,"row_len":94,"slot":93,"row_ids_sha256":"c84cc1000ac37be88f7160c318283be1235cab90b81aa7ee5ba13bbfb36aa9b1","oracle":{"probs":[0.9171975033187058,0.0409948336924119,0.01015829992832257,0.031649363060559675],"argmax":"a","top2_margin":0.8762026696262939,"near_tie":false,"group_logz":{"73":-0.13077926635742188,"267":-7.6707868576049805,"74":-3.2386560440063477,"293":-9.789966583251953,"76":-4.633810997009277,"277":-10.504043579101562,"75":-3.4973840713500977,"280":-10.141027450561523}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9172141341614424,0.04084492468747771,0.010335681247696087,0.03160525990338375],"max_abs_dp":0.00017738131937351624,"argmax_equal":true}}}]},{"id":"tv4h_32","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":492,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/none_present","group_id":"mmlu/test/6139","variant":"none_present","parent_id":"mmlu/test/6139","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6139/none_present"}},"request_sha256":"664206280bb95e8d2dba19b48ba28383f9063d56c9e4ad6309282deec7328662","questions":[{"type":"choice","n_options":5,"row_len":91,"slot":90,"row_ids_sha256":"1e9adc833054a6f6fae215f2b072128440a7451ce7f95e1e640fb6cb9917746e","oracle":{"probs":[0.9631536159390043,0.006400737703066278,0.005810223982985594,0.005895308898424056,0.018740113476519764],"argmax":"c","top2_margin":0.9444135024624846,"near_tie":false,"group_logz":{"41":-0.048023223876953125,"334":-8.842279434204102,"42":-5.061822891235352,"378":-13.452681541442871,"43":-5.15861701965332,"340":-13.50239372253418,"44":-5.144079208374023,"388":-13.563743591308594,"45":-3.987569808959961,"407":-12.326891899108887}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9631890633576478,0.006394185428172868,0.0057947552852126594,0.005884590447792885,0.01873740548117378],"max_abs_dp":3.5447418643475004e-05,"argmax_equal":true}}}]},{"id":"tv4h_33","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":493,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/none_absent","group_id":"mmlu/test/6139","variant":"none_absent","parent_id":"mmlu/test/6139","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6139/none_absent"}},"request_sha256":"c38311a3d9eda22b3f24795fd68bcdb37cd206f1db558b0e4fb5e7f79528c4fa","questions":[{"type":"choice","n_options":4,"row_len":82,"slot":81,"row_ids_sha256":"e77a937db5f7b30b855e0e3be9fe2d3faf908f8a9b91446457c7df3410ca83e0","oracle":{"probs":[0.03881464515252975,0.025102022938223823,0.010933722153049455,0.925149609756197],"argmax":"none_of_these","top2_margin":0.8863349646036672,"near_tie":false,"group_logz":{"41":-3.2588062286376953,"334":-10.878774642944336,"42":-3.694655418395996,"378":-11.050634384155273,"43":-4.525752067565918,"340":-12.270994186401367,"44":-0.08764839172363281,"388":-7.755989074707031}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.03936336047090027,0.02551478853075141,0.010931619422997094,0.9241902315753513],"max_abs_dp":0.0009593781808456825,"argmax_equal":true}}}]},{"id":"tv4h_34","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":494,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/permuted","group_id":"mmlu/test/6139","variant":"permuted","parent_id":"mmlu/test/6139"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/6139/permuted"}},"request_sha256":"febb34864cb3f1df3cb819981502d94057902c90499cbdb5c679acad738aab31","questions":[{"type":"choice","n_options":4,"row_len":82,"slot":81,"row_ids_sha256":"72d59d97d0a4b276acbe6896db987cfac07ab850d09118f03d58faaf817b7dd9","oracle":{"probs":[0.9835934036541537,0.0034477852700851674,0.006340177738997052,0.006618633336764072],"argmax":"c","top2_margin":0.9769747703173897,"near_tie":false,"group_logz":{"75":-0.05755424499511719,"280":-7.811332702636719,"73":-5.711034774780273,"267":-13.255916595458984,"74":-5.101860046386719,"293":-12.59786605834961,"76":-5.058877944946289,"277":-12.121493339538574}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9836082326566605,0.0034287744112247182,0.006351202091672233,0.006611790840442483],"max_abs_dp":1.90108588604492e-05,"argmax_equal":true}}}]},{"id":"tv4h_35","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":495,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/none_present","group_id":"mmlu/test/13810","variant":"none_present","parent_id":"mmlu/test/13810","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13810/none_present"}},"request_sha256":"5bd7b1e6314e60d35dfe6cea2647c8c33fc624fe919f4d4543381bdbaed11b8b","questions":[{"type":"choice","n_options":5,"row_len":102,"slot":101,"row_ids_sha256":"f8e7019adfed48a043841e927dd59436e04f05493f61304ef44342bccbfd3021","oracle":{"probs":[0.0064160923046458415,0.0038222970722867995,0.01757222345089514,0.3530111655133685,0.6191782216588038],"argmax":"a","top2_margin":0.2661670561454353,"near_tie":false,"group_logz":{"41":-5.066585540771484,"334":-13.311151504516602,"42":-5.584543228149414,"378":-13.69389820098877,"43":-4.059075355529785,"340":-11.930669784545898,"44":-1.0588951110839844,"388":-8.59520435333252,"45":-0.49700164794921875,"407":-7.747959136962891}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.006371172981743064,0.0037998559563712318,0.01757690165657155,0.35818088332293235,0.6140711860823819],"max_abs_dp":0.0051697178095638785,"argmax_equal":true}}}]},{"id":"tv4h_36","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":496,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/none_absent","group_id":"mmlu/test/13810","variant":"none_absent","parent_id":"mmlu/test/13810","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13810/none_absent"}},"request_sha256":"39e99033e943286a9c9242c1f1b6f075452e49cd6f729de44c895d3335822576","questions":[{"type":"choice","n_options":4,"row_len":96,"slot":95,"row_ids_sha256":"eda2796f0bc3e0d858a1236543994a259e748943641ad51478cef350cd3513ae","oracle":{"probs":[0.006531383598173894,0.005894291700874524,0.46473862123659576,0.5228357034643558],"argmax":"a","top2_margin":0.058097082227760055,"near_tie":false,"group_logz":{"41":-5.044191360473633,"334":-12.959128379821777,"42":-5.146825790405273,"378":-12.800931930541992,"43":-0.7793350219726562,"340":-8.185416221618652,"44":-0.6615428924560547,"388":-7.980999946594238}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.006507447454916382,0.0058165995382727246,0.47207232272275734,0.5156036302840535],"max_abs_dp":0.007333701486161581,"argmax_equal":true}}}]},{"id":"tv4h_37","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":497,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/permuted","group_id":"mmlu/test/13810","variant":"permuted","parent_id":"mmlu/test/13810"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/13810/permuted"}},"request_sha256":"9711cb3745a9a8b53ad5eab459464ac9ef15f001bec130779adbff86bfd94c4f","questions":[{"type":"choice","n_options":4,"row_len":93,"slot":92,"row_ids_sha256":"5148c891f3dac80cecea9b03bf8bcd886b8e55a038d6d3fdae5ccc375f66630c","oracle":{"probs":[0.007301854099436925,0.04043239544117968,0.005626326185026654,0.9466394242743567],"argmax":"a","top2_margin":0.9062070288331769,"near_tie":false,"group_logz":{"75":-5.005531311035156,"280":-12.465570449829102,"74":-3.2940282821655273,"293":-10.369074821472168,"76":-5.266202926635742,"277":-12.28481674194336,"73":-0.14074134826660156,"267":-8.233781814575195}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.007363474055830468,0.0410735812606552,0.0055971039778169125,0.9459658407056976],"max_abs_dp":0.0006735835686590974,"argmax_equal":true}}}]},{"id":"tv4h_38","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":498,"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434/none_present","group_id":"mmlu/test/12434","variant":"none_present","parent_id":"mmlu/test/12434","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/12434/none_present"}},"request_sha256":"49209bee78510d439d8a2f75a561a395bd68e840bd2822f7657e8474ed29c254","questions":[{"type":"choice","n_options":5,"row_len":258,"slot":257,"row_ids_sha256":"2511ff948e86c19dd21a79010b966c995d34e3fd46446e88b51f5ae51692410f","oracle":{"probs":[0.043475631186508594,0.13303885209334165,0.3815633163497479,0.43765978998809324,0.004262410382308729],"argmax":"b","top2_margin":0.05609647363834536,"near_tie":false,"group_logz":{"41":-3.1639585494995117,"334":-10.186477661132812,"42":-2.045517921447754,"378":-8.697983741760254,"43":-0.99188232421875,"340":-7.659238815307617,"44":-0.8547172546386719,"388":-7.430620193481445,"45":-5.486324310302734,"407":-12.438167572021484}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.04321291910839852,0.12885799911037935,0.38089463229799214,0.4428098685612846,0.004224580921945371],"max_abs_dp":0.005150078573191386,"argmax_equal":true}}}]},{"id":"tv4h_39","source":"tv4h","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":499,"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434/none_absent","group_id":"mmlu/test/12434","variant":"none_absent","parent_id":"mmlu/test/12434","none_key":"none_of_these"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_head_heldout","meta_id":"mmlu/test/12434/none_absent"}},"request_sha256":"d32172be11aa4af8dce576bd12b64c812bab25260f3d7849b724dd008d57f7cb","questions":[{"type":"choice","n_options":4,"row_len":251,"slot":250,"row_ids_sha256":"4e2be2e9bda2b09861f01f7a183d872b201925421165ef7bf3c6a1ae24e72c94","oracle":{"probs":[0.4156625121317902,0.5221935658448291,0.04830304615885017,0.013840875864530484],"argmax":"b","top2_margin":0.10653105371303895,"near_tie":false,"group_logz":{"41":-0.8957366943359375,"334":-8.269474029541016,"42":-0.667572021484375,"378":-7.836376190185547,"43":-3.0481157302856445,"340":-10.127880096435547,"44":-4.2979841232299805,"388":-11.652544021606445}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.41310183857228566,0.525428149618547,0.04781837367877707,0.013651638130390291],"max_abs_dp":0.0032345837737178806,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":100,"_meta":{"row":688,"text_sha256":"4066949ec1b6e92f58345cb5fd30ddf7893f1aba241fdb6a6e1cbec273a084af","row_sha256":"0e67443af7734973a4fa055e05dfcc7d5df45ea2e63aebc622f0471ed7ee2e56","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/688","group_id":"emotion/test/688","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/688"}},"request_sha256":"bdc08a741c84caf70468252ff28b0bc569c912a5ff231cd95c8bcd01ac0532f0","questions":[{"type":"choice","n_options":6,"row_len":53,"slot":52,"row_ids_sha256":"532da72679d2d3cd99c7acd7870e41c2ebe77b02d2b7656f0f7082ee9eccfd01","oracle":{"probs":[0.1426856717492121,0.005798433389378423,0.005306185291211183,0.8335622218872105,0.009829455759960158,0.002818031923027701],"argmax":"anger","top2_margin":0.6908765501379984,"near_tie":false,"group_logz":{"41":-1.962717056274414,"334":-9.054409980773926,"42":-5.165773391723633,"378":-12.568587303161621,"43":-5.254487991333008,"340":-12.77044677734375,"44":-0.19765281677246094,"388":-7.199977874755859,"45":-4.637977600097656,"407":-11.822340965270996,"46":-5.887322425842285,"410":-13.352447509765625}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.144905130896377,0.005769667843999874,0.005262862030586253,0.8314632720023972,0.009788563724270746,0.0028105035023688863],"max_abs_dp":0.0022194591471648983,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":101,"_meta":{"row":1156,"text_sha256":"5c94328e0abcde5f76e9ececdcf831bdfcca8340bde56410a2ee6ea63ede8efa","row_sha256":"0eb690bd9b84ae63ee53771271da390f4f3568f09398ef81aaf16e75aa955ec4","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1156","group_id":"emotion/test/1156","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/1156"}},"request_sha256":"25b9fa1d71c33ed688b3ea7a2219c63250ef0fb66fe3aacf4cf8f002f6630684","questions":[{"type":"choice","n_options":6,"row_len":84,"slot":83,"row_ids_sha256":"5adf0d0afc42c39f0804305e7dc893343d2436b8771a9999716c72878ebbb10a","oracle":{"probs":[0.02114822412101698,0.7925360218463317,0.018717285900393798,0.009871239916816394,0.009839894274195704,0.14788733394124542],"argmax":"joy","top2_margin":0.6446486879050862,"near_tie":false,"group_logz":{"41":-3.8859634399414062,"334":-10.555339813232422,"42":-0.2622814178466797,"378":-6.707228660583496,"43":-4.0080718994140625,"340":-10.480123519897461,"44":-4.647893905639648,"388":-11.34351634979248,"45":-4.651074409484863,"407":-10.942939758300781,"46":-1.9410686492919922,"410":-7.970493316650391}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.021131085810157234,0.7927221677995808,0.01862423384166543,0.009722291952809771,0.009766529668386014,0.14803369092740065],"max_abs_dp":0.0001861459532491727,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":102,"_meta":{"row":1762,"text_sha256":"ad39fc80381cfdb8445d4823ad85ad62a450b9aa9e3e6c2fac218c5af8bbcdff","row_sha256":"0ed670b67b6b443b13254f6f3ceba1594a40b3d5efa018cb847a1bf4cc09de92","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1762","group_id":"emotion/test/1762","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/1762"}},"request_sha256":"1838d9f9fadb15471382f8d6d7c5f8d20b96b9601d0cb7bf70343acfc9239126","questions":[{"type":"choice","n_options":6,"row_len":68,"slot":67,"row_ids_sha256":"50010b62ad49a612f973d9b28e7d607f386b26200474bc35a25f087b26123325","oracle":{"probs":[0.03787208404140185,0.061183068414204886,0.1462382282386836,0.004908980246284792,0.7434966314608296,0.006301007598595327],"argmax":"fear","top2_margin":0.597258403222146,"near_tie":false,"group_logz":{"41":-3.295114517211914,"334":-10.769372940063477,"42":-2.815458297729492,"378":-9.78449821472168,"43":-1.944091796875,"340":-8.86174488067627,"44":-5.338262557983398,"388":-12.741289138793945,"45":-0.3179645538330078,"407":-7.052722930908203,"46":-5.088619232177734,"410":-12.511308670043945}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.03758139679801782,0.06097445800048121,0.14504260670173247,0.004816153760190298,0.7453539331175041,0.006231451622074198],"max_abs_dp":0.001857301656674526,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":103,"_meta":{"row":485,"text_sha256":"fa4cd7567c7ddff76ea7698ba5b6216ea8436250b6bbd3dc121b9ac95e96339a","row_sha256":"0f5c1699eaf05166edeb880862d73f5e69b9b19355324a0191a5aee53c028243","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/485","group_id":"emotion/test/485","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/485"}},"request_sha256":"1571d14eae04a5a6a45f52ad439265d0dfba6cb04a0c5760dd2e9538cf416197","questions":[{"type":"choice","n_options":6,"row_len":64,"slot":63,"row_ids_sha256":"8dd785d31757273d0cc309256068e00d400059aae905540d75c3a32b69d904da","oracle":{"probs":[0.516480346188073,0.3082028023082441,0.1590973826082506,0.0062365223009567036,0.006573631893602308,0.003409314700873208],"argmax":"sadness","top2_margin":0.20827754387982894,"near_tie":false,"group_logz":{"41":-0.6816043853759766,"334":-7.938057899475098,"42":-1.1978836059570312,"378":-8.176422119140625,"43":-1.8591251373291016,"340":-8.773331642150879,"44":-5.09821891784668,"388":-12.482645034790039,"45":-5.045575141906738,"407":-12.384364128112793,"46":-5.702130317687988,"410":-13.289154052734375}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.5146641505169411,0.30901310280632155,0.16012525457495808,0.006190988520562128,0.006610323102382421,0.0033961804788347288],"max_abs_dp":0.0018161956711318927,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":104,"_meta":{"row":482,"text_sha256":"72cedf6ff6119d3cdfd596952a47db17640ed14c4e7bc994ef8e6f0524f3e54f","row_sha256":"0f8ea79039cfe96a6aafe20683ec52f9058868b3fbc01fef4fc0f1993302d2b6","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/482","group_id":"emotion/test/482","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/482"}},"request_sha256":"e0bd1435fc1925fa8a85c48596a7f325a729710c54437b3e740b643cf4c4bc69","questions":[{"type":"choice","n_options":6,"row_len":57,"slot":56,"row_ids_sha256":"50c0411e3d6a748c88cc415159e78a98909d20e50f8c26b49957ba916f763c89","oracle":{"probs":[0.4435469530156021,0.2704873424124123,0.0196214881088675,0.24094609096854105,0.011832396824139776,0.013565728670437403],"argmax":"sadness","top2_margin":0.1730596106031898,"near_tie":false,"group_logz":{"41":-0.8500041961669922,"334":-7.635824203491211,"42":-1.3445825576782227,"378":-7.841268539428711,"43":-3.9681825637817383,"340":-10.737004280090332,"44":-1.4602346420288086,"388":-7.896422386169434,"45":-4.473966598510742,"407":-11.140798568725586,"46":-4.337261199951172,"410":-11.212833404541016}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.4449440684412329,0.26913778741502004,0.01945373793559698,0.24112575583547016,0.011893211858028415,0.013445438514651423],"max_abs_dp":0.0013971154256308127,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":105,"_meta":{"row":959,"text_sha256":"24c2a3b49262128f8e7f4493cf92e2df90ac37301bef8868e23a0655b278617e","row_sha256":"0f9c21d82eb205905d920e2ebee7fd94a0a69419bf1fbfcc21411261819828c2","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/959","group_id":"emotion/test/959","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/959"}},"request_sha256":"315cf6e29f7662c25154e833b4f7bc98195867934f36824eea6e9a4efff0f1f6","questions":[{"type":"choice","n_options":6,"row_len":52,"slot":51,"row_ids_sha256":"2a29cc74e063624c8d5cbb563136eea786ff8868d6d2f49f327a1720093a88fe","oracle":{"probs":[0.016741401995701725,0.41621212846560696,0.5546159095276596,0.005082180586855093,0.003824421407429794,0.003523958016746848],"argmax":"love","top2_margin":0.13840378106205264,"near_tie":false,"group_logz":{"41":-4.113811492919922,"334":-11.031540870666504,"42":-0.9005012512207031,"378":-7.324014663696289,"43":-0.6134204864501953,"340":-7.012596130371094,"44":-5.30595588684082,"388":-12.241935729980469,"45":-5.590289115905762,"407":-12.32145881652832,"46":-5.672111511230469,"410":-12.731325149536133}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.016491375002562366,0.42120835755126973,0.5499405940350525,0.005026720678032853,0.003811981942534276,0.0035209707905483395],"max_abs_dp":0.004996229085662773,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":106,"_meta":{"row":573,"text_sha256":"40b80bafb098041e5616de300eb5307603e3c1d76ffcb2a93a8d1f335e72b0b3","row_sha256":"0fd082dba48e728637716ff7ccbde925e36a81b67e911fe071cbcd9c163c9601","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/573","group_id":"emotion/test/573","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/573"}},"request_sha256":"7be5107e3146551662f75fdf144cca4fa8d840a209b28d08e054c175d0b21516","questions":[{"type":"choice","n_options":6,"row_len":59,"slot":58,"row_ids_sha256":"3963d65d71715d3eefc02663fb80294ad26d0ccb2f4496c6112363a66ac16b8c","oracle":{"probs":[0.9554883539080314,0.0068179927095704555,0.005485594382454173,0.009292750676590451,0.00591003223704722,0.017005276086306143],"argmax":"sadness","top2_margin":0.9384830778217252,"near_tie":false,"group_logz":{"41":-0.05506324768066406,"334":-8.257546424865723,"42":-4.997720718383789,"378":-13.047220230102539,"43":-5.215160369873047,"340":-13.321247100830078,"44":-4.688051223754883,"388":-12.55177116394043,"45":-5.140634536743164,"407":-13.288379669189453,"46":-4.083762168884277,"410":-11.769733428955078}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9559771357772728,0.006790762801800533,0.00541383921026436,0.009127623278209793,0.005867455649052488,0.016823183283399983],"max_abs_dp":0.0004887818692413948,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":107,"_meta":{"row":588,"text_sha256":"567395b877b01a51dda43199e8133c4ca29b4c2aec64fe320ea22003806b2e07","row_sha256":"0fe855203820afc20abc794e92cf670acd215f362b31c7cd545a6f6f318bb48e","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/588","group_id":"emotion/test/588","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/588"}},"request_sha256":"0be081bb2771d9a07ef87d0824a26ef329c20d687b3a18a9888507780293c4c8","questions":[{"type":"choice","n_options":6,"row_len":79,"slot":78,"row_ids_sha256":"3d2df2edad42b069ba6002a34e829079f2fcfdd05072e545dc32e4e58df61f94","oracle":{"probs":[0.5337322702996335,0.3857418031982786,0.053373301674397015,0.011636546100639035,0.01075807461463205,0.0047580041124196965],"argmax":"sadness","top2_margin":0.14799046710135494,"near_tie":false,"group_logz":{"41":-0.6551647186279297,"334":-7.436640739440918,"42":-0.9798908233642578,"378":-7.532129287719727,"43":-2.9577484130859375,"340":-9.509655952453613,"44":-4.480908393859863,"388":-11.293004035949707,"45":-4.5594024658203125,"407":-11.293107032775879,"46":-5.37523078918457,"410":-12.43703842163086}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.5367093298006347,0.3823254868712568,0.053834851577907086,0.01157069678707454,0.01083192810016137,0.004727706862965528],"max_abs_dp":0.003416316327021829,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":108,"_meta":{"row":1176,"text_sha256":"6ae2fe2353747ab43c1fb4639d95c826246c76ce3f9cb7d99c0305f94c6ed03c","row_sha256":"109d5f2888e3918274d98ae9f43e42980b477fcd20df8778fec8b0c450f1426d","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1176","group_id":"emotion/test/1176","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/1176"}},"request_sha256":"3207d569e44aba394c6ffce47d84564d3ad69e118fab1eee7afd0cee5dc77f49","questions":[{"type":"choice","n_options":6,"row_len":64,"slot":63,"row_ids_sha256":"46e5c0160808208f063c94f569410e6fe1b355701de88975d090cfd15b8b5f6e","oracle":{"probs":[0.02092185804500085,0.9383857511309437,0.010827340201888299,0.007623370602054575,0.005693953132114082,0.01654772688799848],"argmax":"joy","top2_margin":0.9174638930859429,"near_tie":false,"group_logz":{"41":-3.8901453018188477,"334":-11.248716354370117,"42":-0.08677864074707031,"378":-7.118223190307617,"43":-4.54886531829834,"340":-11.714078903198242,"44":-4.899721145629883,"388":-12.2106294631958,"45":-5.191534996032715,"407":-12.134333610534668,"46":-4.124691009521484,"410":-11.099356651306152}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.020819195663344007,0.9387797939898025,0.010685107869578095,0.007531435984998552,0.005700282738332249,0.016484183753944615],"max_abs_dp":0.00039404285885880075,"argmax_equal":true}}}]},{"id":"tv4xh_emotion_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":109,"_meta":{"row":851,"text_sha256":"65d13295c3844d56ae0ad485e08f040ef338d406397893d51e5028943de7b6d7","row_sha256":"128280c0bbce14f63d49c543a4bd6585e6ac4f51b940f1ed10bd09ebc8c23b78","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/851","group_id":"emotion/test/851","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"emotion/test/851"}},"request_sha256":"4a46645345ad3b1e54d231a14f0b6b2b2417f1023ee9cba5d10f9c1dafd85a95","questions":[{"type":"choice","n_options":6,"row_len":75,"slot":74,"row_ids_sha256":"537a40d4aeed6e8d4f8d61b291038e8f4ed0e58cb6fdbc85a070b747ea9d5288","oracle":{"probs":[0.9343525264332874,0.007745898885304525,0.00537159671816194,0.020624629340387692,0.022034511374720636,0.009870837248137813],"argmax":"sadness","top2_margin":0.9123180150585667,"near_tie":false,"group_logz":{"41":-0.07936668395996094,"334":-7.887641906738281,"42":-4.87205696105957,"378":-12.550159454345703,"43":-5.238095283508301,"340":-12.973503112792969,"44":-3.8927345275878906,"388":-11.219563484191895,"45":-3.826610565185547,"407":-11.280878067016602,"46":-4.629635810852051,"410":-12.100341796875}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9348071388200667,0.00771215020160542,0.005318720858023732,0.020379170959688057,0.021948889132616493,0.009833930027999555],"max_abs_dp":0.0004546123867793517,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":260,"_meta":{"row":2378,"text_sha256":"c6fe821d0c7c11b37b5f21a0c4954effcab910883eb1dbaa16c1ca8313ff491c","row_sha256":"06855f8415c6e0bfc6c726dde4f27b4bf77d91f43ced854a9aa1bbfd9bc64722","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2378","group_id":"qnli/validation/2378","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/2378"}},"request_sha256":"30cfbb2053fbea16c7182db2e43deee45d4a9353b33bf7a5a104fcd0661ccbfc","questions":[{"type":"noul","n_options":2,"row_len":76,"slot":75,"row_ids_sha256":"a3e368378451b5d9ebdb65157431e761c25ad44e03d5206ec1016ec005e86aed","oracle":{"probs":[0.013954476846481477,0.9860455231535185],"argmax":"no","top2_margin":0.972091046307037,"near_tie":false,"group_logz":{"11683":-4.290918350219727,"12447":-8.396615982055664,"2243":-0.033016204833984375,"4547":-4.161869049072266,"19598":-8.356451034545898}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.014050884052297687,0.9859491159477023],"max_abs_dp":9.64072058162093e-05,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":261,"_meta":{"row":3361,"text_sha256":"512d1eded0f043e83e2e66c0ed6900d213324728e1babc58b65849537f9718cd","row_sha256":"06c348434d5500b8032a2275f91cb9f1556d654a743f2446e97068a47f1238e5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3361","group_id":"qnli/validation/3361","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/3361"}},"request_sha256":"2c09f00d38d3641dc1d004c8e215d20aa3cd7644602dd63b893e19aec9038538","questions":[{"type":"noul","n_options":2,"row_len":70,"slot":69,"row_ids_sha256":"3143f86e6586e85ba0210b0731388942d4d021a7420e903e989a46dd5f0328c0","oracle":{"probs":[0.0720551780337937,0.9279448219662062],"argmax":"no","top2_margin":0.8558896439324125,"near_tie":false,"group_logz":{"11683":-2.6606788635253906,"12447":-6.33460807800293,"2243":-0.10513877868652344,"4547":-3.964362144470215,"19598":-7.974508285522461}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.07173624732127248,0.9282637526787276],"max_abs_dp":0.0003189307125214258,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":262,"_meta":{"row":407,"text_sha256":"fb2b941cbe4a24159c78a58b85908a39cb2999038a1dfd40d1e94f0af7cd07ad","row_sha256":"0759615d0bc41bd3973690f7d0dd7be2404699c5e316d67dc6e945a499b0a49f","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/407","group_id":"qnli/validation/407","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/407"}},"request_sha256":"75f1399443006423884fb287764d6937aacd82d45293c54f03458275696953c3","questions":[{"type":"noul","n_options":2,"row_len":69,"slot":68,"row_ids_sha256":"4c25de03b9c70e568c03ef5023711342af4c3203b2e1f620a1dc06119967078b","oracle":{"probs":[0.9391863579022436,0.06081364209775646],"argmax":"yes","top2_margin":0.8783727158044872,"near_tie":false,"group_logz":{"11683":-0.08971405029296875,"12447":-4.363222122192383,"2243":-2.826913833618164,"4547":-6.610759735107422,"19598":-9.422861099243164}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9392363155847132,0.06076368441528678],"max_abs_dp":4.99576824696743e-05,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":263,"_meta":{"row":4731,"text_sha256":"1a6c7b51a51440bfb18df4d6f1776a43fc8f6e9dda43b9a58621967409c573bd","row_sha256":"08047603cb2b88d97bed089e0d47c928ac3e6d4d33a4d901dec8888a04dea1f9","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4731","group_id":"qnli/validation/4731","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/4731"}},"request_sha256":"a93980d77f2d3cc6396e972d3e908dc218895bd5acfc18e96241864ebfe952db","questions":[{"type":"noul","n_options":2,"row_len":75,"slot":74,"row_ids_sha256":"ac089bace4e4e7de989dbf475b389480338ce41a4d7e69beca7c902f058e58cb","oracle":{"probs":[0.6976012724927243,0.30239872750727564],"argmax":"yes","top2_margin":0.39520254498544866,"near_tie":false,"group_logz":{"11683":-0.39819908142089844,"12447":-4.544258117675781,"2243":-1.234100341796875,"4547":-5.3377685546875,"19598":-8.713569641113281}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.6937422139756492,0.3062577860243509],"max_abs_dp":0.0038590585170752423,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":264,"_meta":{"row":946,"text_sha256":"faabefe33b508352d8d589e298d323b2eb91c9ec9a4a45b18da5f2cdec71a911","row_sha256":"087b55e02c0d683adc4bc13b8a78cc17c1d364e14fbc406ce98772efa6b7db17","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/946","group_id":"qnli/validation/946","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/946"}},"request_sha256":"7e6f2fc1afab334cb6fdd1745bb744721a38f3db6eed32064f8dc0ed200fc040","questions":[{"type":"noul","n_options":2,"row_len":68,"slot":67,"row_ids_sha256":"62e56e5485d804028c8518a77bb22edb4131e1802b7c222d13ac6d8cd8ac3196","oracle":{"probs":[0.9730165762305412,0.026983423769458906],"argmax":"yes","top2_margin":0.9460331524610822,"near_tie":false,"group_logz":{"11683":-0.04424858093261719,"12447":-4.621114730834961,"2243":-3.629426956176758,"4547":-7.871869087219238,"19598":-10.452217102050781}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9730659511293307,0.026934048870669275],"max_abs_dp":4.937489878963086e-05,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":265,"_meta":{"row":886,"text_sha256":"a2323b14a06d6655fa963b755d46c58d9dd666e2ff51cceea05537b44d09c4e7","row_sha256":"08d7c1a79b327fc0434770ddc94d5522c8f3a8f2b738f0092974a40ffc5c6dc5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/886","group_id":"qnli/validation/886","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/886"}},"request_sha256":"2b6f4242a6beb2c74ee7b9861f2862732f9ad29fa495e0bbda16e04ad19bf5c8","questions":[{"type":"noul","n_options":2,"row_len":78,"slot":77,"row_ids_sha256":"d9aa1379060accafcc363ec6a0a5ec51f2a76a6be49b1b60167348e62f09f959","oracle":{"probs":[0.023378232578035684,0.9766217674219644],"argmax":"no","top2_margin":0.9532435348439287,"near_tie":false,"group_logz":{"11683":-3.7759151458740234,"12447":-7.87492561340332,"2243":-0.043621063232421875,"4547":-4.202417373657227,"19598":-8.376745223999023}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.023615868066463663,0.9763841319335362],"max_abs_dp":0.00023763548842814508,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":266,"_meta":{"row":4136,"text_sha256":"e17111ca25b490ecb466646b5efc3cb6912875ca2143eba611fd180e3d1cf134","row_sha256":"08ee5449e51919b980722057d809f2a333e076f51977a606f9a8f48d65a5f36c","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4136","group_id":"qnli/validation/4136","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/4136"}},"request_sha256":"29e74c70f1bcf12c8cc8525308b74b0e0fdfe8cc003d9eea3bc03e7759891a6d","questions":[{"type":"noul","n_options":2,"row_len":58,"slot":57,"row_ids_sha256":"77d4b83fb0505eaaaea3960cc945c90033f55e55d0d49612506b347200f420be","oracle":{"probs":[0.5547390204871414,0.44526097951285865],"argmax":"yes","top2_margin":0.10947804097428276,"near_tie":false,"group_logz":{"11683":-0.6151580810546875,"12447":-4.924320220947266,"2243":-0.8349952697753906,"4547":-4.970415115356445,"19598":-8.552334785461426}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.5535521873378246,0.44644781266217537],"max_abs_dp":0.0011868331493167794,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":267,"_meta":{"row":3906,"text_sha256":"e5672b77c362c7b972fb1cfffcecb6024a6b23bb903a2e8b89bb46d8282a9218","row_sha256":"0901c7bb1e03299bb60f1e70a3de28c973907aae639550a523378e73ca1a3ab5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3906","group_id":"qnli/validation/3906","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/3906"}},"request_sha256":"b029bc475fd5a057ef7a5124cba12fc30493f1730573c035c456ac16fe14acd9","questions":[{"type":"noul","n_options":2,"row_len":105,"slot":104,"row_ids_sha256":"3cf2813efc2e1e5cd17d69c0529ce0f9e60a28619d4e4d875313c44b92a05f15","oracle":{"probs":[0.01894745062396102,0.9810525493760389],"argmax":"no","top2_margin":0.962105098752078,"near_tie":false,"group_logz":{"11683":-3.974872589111328,"12447":-8.986846923828125,"2243":-0.02791595458984375,"4547":-5.3196821212768555,"19598":-8.88302993774414}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.01900445227277491,0.9809955477272252],"max_abs_dp":5.7001648813889e-05,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":268,"_meta":{"row":798,"text_sha256":"33c94e5e38ca4e83a4a28cf20e08766628445271022a436d777e09586bea4c54","row_sha256":"090efeb48f5520325b2bffb843f851a59b96d5029285aa75059a92c3a94a1674","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/798","group_id":"qnli/validation/798","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/798"}},"request_sha256":"3d70ec1293eaa90cd3f686a97ca0673190866cbafce4319b266730a26baaee6d","questions":[{"type":"noul","n_options":2,"row_len":55,"slot":54,"row_ids_sha256":"3b38f79daccb6a8e34f8ff4c593cbf43397ea50eb8268a665fe58912e9396aa7","oracle":{"probs":[0.5046018254713115,0.4953981745286885],"argmax":"yes","top2_margin":0.009203650942622987,"near_tie":true,"group_logz":{"11683":-0.7774600982666016,"12447":-3.848215103149414,"2243":-0.795867919921875,"4547":-3.402667999267578,"19598":-7.534327507019043}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.5071489938308184,0.49285100616918165],"max_abs_dp":0.0025471683595068573,"argmax_equal":true}}}]},{"id":"tv4xh_qnli_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":269,"_meta":{"row":4307,"text_sha256":"8a67f1b71d6a48a812bf0d42fc6741acba8265638e28e476a3cf5e6c0fcd1ab0","row_sha256":"09abcf8960243b05d2b20ec2cdd1204138159cf01a9019d7827eb4398b9acce4","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4307","group_id":"qnli/validation/4307","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"qnli/validation/4307"}},"request_sha256":"0e14a212bb478d1996c569b2cd89d5a5d4b6356f98de924c5e0b9f4d0e30f331","questions":[{"type":"noul","n_options":2,"row_len":74,"slot":73,"row_ids_sha256":"5f0a2d8f5e14c240dd68444fb3b269cbef41bf8e875c8728e43a2047897b1157","oracle":{"probs":[0.03795742621191662,0.9620425737880834],"argmax":"no","top2_margin":0.9240851475761668,"near_tie":false,"group_logz":{"11683":-3.293964385986328,"12447":-7.286855697631836,"2243":-0.061370849609375,"4547":-4.178619384765625,"19598":-8.26129150390625}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.037973275807367333,0.9620267241926327],"max_abs_dp":1.5849595450756482e-05,"argmax_equal":true}}}]},{"id":"tv4xh_paws_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":340,"_meta":{"row":6859,"text_sha256":"d133a2515f30159c5dae04ad8d8ba56b6b9d753268e97060eb8ce7113ef08801","row_sha256":"08cea23d05d2955c437449fb24606ba4401ee55476205aa55a1633a3238df235","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6859","group_id":"paws/test/6859","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/6859"}},"request_sha256":"1a581ff68ef2cbcaf0e4ae56c008f20994f271d6dc311b8a4ea60ccc1fae887b","questions":[{"type":"noul","n_options":2,"row_len":113,"slot":112,"row_ids_sha256":"bd0077339ebfd9c00985ae2f7a72c2f64c7a2464b75f0306e1bf70f8276ca0f1","oracle":{"probs":[0.9341746544790175,0.06582534552098245],"argmax":"yes","top2_margin":0.8683493089580351,"near_tie":false,"group_logz":{"11683":-0.08820152282714844,"12447":-5.908794403076172,"2243":-2.7408599853515625,"4547":-8.305644035339355,"19598":-10.15453815460205}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9348821257788649,0.06511787422113514],"max_abs_dp":0.0007074712998473842,"argmax_equal":true}}}]},{"id":"tv4xh_paws_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":341,"_meta":{"row":1560,"text_sha256":"3f57aafe302c45afd11ba6814fe7261a1109baebb6c0a8671dd8e4af6e07bdb7","row_sha256":"08fec8b3386611646a7473ab9a745837c008760a4612800a8d1b2d0362fc9a1c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1560","group_id":"paws/test/1560","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/1560"}},"request_sha256":"ef717671487e344bf843c26ef42ef61bd1631303c7a7b8c93bf92bab84c52ad0","questions":[{"type":"noul","n_options":2,"row_len":81,"slot":80,"row_ids_sha256":"67452005ea4eb1eb5d0e5731df69b1080384a70d0f57c53a488b1e540efe565c","oracle":{"probs":[0.9213001074743846,0.07869989252561542],"argmax":"yes","top2_margin":0.8426002149487692,"near_tie":false,"group_logz":{"11683":-0.10285758972167969,"12447":-5.829950332641602,"2243":-2.5630016326904297,"4547":-7.912413597106934,"19598":-10.148658752441406}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9208259426581376,0.07917405734186245],"max_abs_dp":0.0004741648162470691,"argmax_equal":true}}}]},{"id":"tv4xh_paws_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":342,"_meta":{"row":7712,"text_sha256":"933523538c3eb6e0fd147ff96af499139a9c056a9e3345901f9e3ee457b1a48c","row_sha256":"0901ebdc21f24c42b0651b75962b879dd88d2f015bce4121c9c110027fd959a9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/7712","group_id":"paws/test/7712","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/7712"}},"request_sha256":"a924a185336b145f98246d7979bc15e9702b60f012401ba0c820fd6d8e52bcdc","questions":[{"type":"noul","n_options":2,"row_len":136,"slot":135,"row_ids_sha256":"60a2554df9a21d80e2438735578501a19ee6534f8c36ed36cb7a43e868d3604a","oracle":{"probs":[0.7398139884997434,0.2601860115002565],"argmax":"yes","top2_margin":0.47962797699948695,"near_tie":false,"group_logz":{"11683":-0.34032630920410156,"12447":-5.777768135070801,"2243":-1.3853282928466797,"4547":-6.656132698059082,"19598":-8.805971145629883}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.7420018366866783,0.25799816331332165],"max_abs_dp":0.0021878481869348487,"argmax_equal":true}}}]},{"id":"tv4xh_paws_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":343,"_meta":{"row":3530,"text_sha256":"4da4508cc8c69c075e3f4f1aaa931b5b394a47ab9ea23efa69272f55dc44237a","row_sha256":"0954a85c777d1b7a2b28638281d2e238c4238ed6f881be0d21eda0b9e2209d63","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3530","group_id":"paws/test/3530","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/3530"}},"request_sha256":"cb86f7460611faaa86ea2dd9933c31e6f038f8e288b10faa0f9116696bf4c7df","questions":[{"type":"noul","n_options":2,"row_len":115,"slot":114,"row_ids_sha256":"27d1df90d35541298f8b12ce2c46eca98352a3ec6e493293da3b83d3295813c8","oracle":{"probs":[0.028843272150192993,0.971156727849807],"argmax":"no","top2_margin":0.942313455699614,"near_tie":false,"group_logz":{"11683":-3.5542736053466797,"12447":-8.693364143371582,"2243":-0.037662506103515625,"4547":-5.619161605834961,"19598":-8.29442024230957}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.028993352099719597,0.9710066479002804],"max_abs_dp":0.00015007994952664916,"argmax_equal":true}}}]},{"id":"tv4xh_paws_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":344,"_meta":{"row":5349,"text_sha256":"a12f11b13b1aa82b9763c901029e81375771ec4343a069c040c560d8eb39dc1b","row_sha256":"09c772b82040c213ae17b52d3e35354212d0a2dc85d4c3c6e503a372db3b9091","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5349","group_id":"paws/test/5349","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/5349"}},"request_sha256":"3b874ffd342236b3605819c098f69c01e4bee0d385e4cd106ba62a854ff5de13","questions":[{"type":"noul","n_options":2,"row_len":89,"slot":88,"row_ids_sha256":"8a8fe089d19fc74c548b41793da51e5f17545f5652da65aa385326db26abf7a9","oracle":{"probs":[0.8338212581445146,0.1661787418554854],"argmax":"yes","top2_margin":0.6676425162890292,"near_tie":false,"group_logz":{"11683":-0.20432090759277344,"12447":-5.632833480834961,"2243":-1.8172760009765625,"4547":-6.929681777954102,"19598":-9.24057388305664}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.837340864040998,0.16265913595900214],"max_abs_dp":0.0035196058964833687,"argmax_equal":true}}}]},{"id":"tv4xh_paws_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":345,"_meta":{"row":5078,"text_sha256":"84a9caa371f97bff330cfd3abcfa5c5c761640e50d7df225849cf33513014959","row_sha256":"0a78ee6268fc5390fd0f2de018ae91de423f3ec86b913a532d2d0d3ee0481a91","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5078","group_id":"paws/test/5078","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/5078"}},"request_sha256":"412533898802480d15a1f9c7f4ba89a9379268f8ba912584b35d9b06cf7ec45b","questions":[{"type":"noul","n_options":2,"row_len":129,"slot":128,"row_ids_sha256":"33c1613d0456cd3a76a5735018cf485a026c369ae7b0744453d9e42782778e3d","oracle":{"probs":[0.9743464496497696,0.025653550350230557],"argmax":"yes","top2_margin":0.948692899299539,"near_tie":false,"group_logz":{"11683":-0.034427642822265625,"12447":-6.3236236572265625,"2243":-3.6715126037597656,"4547":-9.53665828704834,"19598":-11.146551132202148}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9745425754626803,0.025457424537319764],"max_abs_dp":0.00019612581291079262,"argmax_equal":true}}}]},{"id":"tv4xh_paws_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":346,"_meta":{"row":3539,"text_sha256":"44505af2003c8d22ac185056a2799a0936c7b7617c63754193dcbde1805528ed","row_sha256":"0aac5ddeb4adeec6d64faf8248c48f7f682c63b8864e2242df28c6051c66e5eb","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3539","group_id":"paws/test/3539","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/3539"}},"request_sha256":"81af6201d237d13709443ba739ddb904ac924f8dceecd3b740c1790e9b370c03","questions":[{"type":"noul","n_options":2,"row_len":108,"slot":107,"row_ids_sha256":"6a26ebf25354614a45461aff86487c604bd7d4c7c4bd8b96022023eecea4d599","oracle":{"probs":[0.019990681557075108,0.9800093184429249],"argmax":"no","top2_margin":0.9600186368858499,"near_tie":false,"group_logz":{"11683":-3.9203872680664062,"12447":-8.992083549499512,"2243":-0.0280914306640625,"4547":-5.431678771972656,"19598":-8.545135498046875}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.02005850412528873,0.9799414958747112],"max_abs_dp":6.782256821369348e-05,"argmax_equal":true}}}]},{"id":"tv4xh_paws_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":347,"_meta":{"row":4865,"text_sha256":"0136388d20728451308640a9979e44675ccfbb86ecdca02bb0de1d097ff658c6","row_sha256":"0ae573d07f9666cb5a633e726f3cc5ffa13c09f558f67b229f54de306f4a0d20","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4865","group_id":"paws/test/4865","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/4865"}},"request_sha256":"39b86994114a2abcae15f07f0cda58753f3dd4ade44132f76d48ce2005b4d373","questions":[{"type":"noul","n_options":2,"row_len":98,"slot":97,"row_ids_sha256":"fc1aaed3a0448e8d22d17d7037c49195efd0975b3d58ed7e0c5645ae2d2ab579","oracle":{"probs":[0.04851528171868182,0.9514847182813182],"argmax":"no","top2_margin":0.9029694365626364,"near_tie":false,"group_logz":{"11683":-3.0347537994384766,"12447":-8.070068359375,"2243":-0.0586090087890625,"4547":-5.422983169555664,"19598":-8.308813095092773}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.04828827592062026,0.9517117240793798],"max_abs_dp":0.00022700579806156362,"argmax_equal":true}}}]},{"id":"tv4xh_paws_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":348,"_meta":{"row":1721,"text_sha256":"6df8a93c3d0dc1aab6e8823532dbad98697089398e8c3cf4eefa19162d57a3b7","row_sha256":"0afc01bb583b9ffd617e8c2c6d87d7fcc42ccc569b4a27fe5968dcb27f16e04e","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1721","group_id":"paws/test/1721","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/1721"}},"request_sha256":"e6f59b313197b41f3206d47ef689db48f6df8c25839287853979a5daac6dc2ae","questions":[{"type":"noul","n_options":2,"row_len":111,"slot":110,"row_ids_sha256":"d5cca00361529120f1380c91898bb086d15db2eb1b709b54a016a6dffd704203","oracle":{"probs":[0.934973414018664,0.06502658598133615],"argmax":"yes","top2_margin":0.8699468280373278,"near_tie":false,"group_logz":{"11683":-0.08477401733398438,"12447":-5.582645416259766,"2243":-2.7504959106445312,"4547":-7.872991561889648,"19598":-10.078882217407227}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9355752243763912,0.06442477562360877],"max_abs_dp":0.0006018103577273837,"argmax_equal":true}}}]},{"id":"tv4xh_paws_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":349,"_meta":{"row":2075,"text_sha256":"cd001b26bb35a17a3661e47c3e9c41997bfe37e8f0e0f9a34139c31a5c8f1256","row_sha256":"0b1e3adfbd74bad139926051a69f053eb8783b7cd3ba969f6c0bc64da3aff570","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2075","group_id":"paws/test/2075","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"paws/test/2075"}},"request_sha256":"db09351e38ab181faf6004adc9ef020eedc2ec6131229bd232f97dd8c2d2cefd","questions":[{"type":"noul","n_options":2,"row_len":119,"slot":118,"row_ids_sha256":"9f2fa20daaeeaa18d6d38b39efc6fe478396e00f7dce55cfea332bab76368119","oracle":{"probs":[0.04444495778680473,0.9555550422131953],"argmax":"no","top2_margin":0.9111100844263906,"near_tie":false,"group_logz":{"11683":-3.123952865600586,"12447":-8.413593292236328,"2243":-0.055912017822265625,"4547":-5.740155220031738,"19598":-8.344938278198242}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.04504251099749234,0.9549574890025077],"max_abs_dp":0.0005975532106876624,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":420,"_meta":{"row":788,"text_sha256":"170c9dc2f4e8bdae8fb55bc0daa860622ec13f12d01815fc5715ea18c6001d1b","row_sha256":"089f0fbe86676aa88f493fd5bbab51c01699d8a8cb18a43ab98acb9b950c1b65","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/788","group_id":"sciq/test/788","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/788"}},"request_sha256":"4c81341793d5a06e814fde2f20deeb741941fb2b79f4601ea6cb64cbf001932f","questions":[{"type":"choice","n_options":4,"row_len":132,"slot":131,"row_ids_sha256":"b5b2dd15249496ab9a6f539904424d66a4e3c118c2313a277c4d5ee9f526a933","oracle":{"probs":[0.0009908364553643534,0.004337348651946119,0.024285592228987473,0.970386222663702],"argmax":"d","top2_margin":0.9461006304347145,"near_tie":false,"group_logz":{"73":-6.942197799682617,"267":-15.180662155151367,"74":-5.465728759765625,"293":-13.84210205078125,"75":-3.7431087493896484,"280":-11.92379093170166,"76":-0.0552978515625,"277":-8.513501167297363}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.00098542552442288,0.004355986554943953,0.02464168608080004,0.970016901839833],"max_abs_dp":0.00036932082386897935,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":421,"_meta":{"row":410,"text_sha256":"c59626ff7454f49e722fb8d510f3fc827cf740161419fbcdf03c89c27b9f2ac1","row_sha256":"08b076723422504a5a47b8ee01021ce401b7d69a47950f6a991f9e0fe637f4c1","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/410","group_id":"sciq/test/410","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/410"}},"request_sha256":"aec2b345dd69b16249880ec15cebfefe9f7427ffd657ec105e3994d7c043dda5","questions":[{"type":"choice","n_options":4,"row_len":107,"slot":106,"row_ids_sha256":"188790ff5d5a51030f99bcbca1c20b5e4262fa02ad24b7f846d78085a4d2e6e4","oracle":{"probs":[0.0021527404438056253,0.9915761122667329,0.0029289555707584184,0.003342191718703003],"argmax":"b","top2_margin":0.9882339205480299,"near_tie":false,"group_logz":{"73":-6.165053367614746,"267":-14.235064506530762,"74":-0.03249931335449219,"293":-8.626348495483398,"75":-5.857149124145508,"280":-13.9647216796875,"76":-5.725168228149414,"277":-13.693625450134277}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0021445602628500735,0.9916157609348378,0.002925615481750177,0.0033140633205620156],"max_abs_dp":3.964866810490797e-05,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":422,"_meta":{"row":686,"text_sha256":"f95a6c8b8bac788a6a6d238326fda619d29715cda257c89621074d940c0cc7be","row_sha256":"08e084e01d87d9f2cbde9b49eabd9e58f7acf4450ead304dbb20da2dafaecdcb","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/686","group_id":"sciq/test/686","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/686"}},"request_sha256":"5ac1b87d3e4c3dab07923b9e8490aac1013d5ac4c9f3ab8371c651420122b8b5","questions":[{"type":"choice","n_options":4,"row_len":58,"slot":57,"row_ids_sha256":"bcc423bbb88bbd1af302abf5ee802317b2c623a8fca2c03e8bde362ec4c240c5","oracle":{"probs":[0.1854910072620376,0.06362185965828869,0.740412491842428,0.010474641237245797],"argmax":"c","top2_margin":0.5549214845803904,"near_tie":false,"group_logz":{"73":-1.7309207916259766,"267":-8.964365005493164,"74":-2.8009700775146484,"293":-9.698019027709961,"75":-0.34671974182128906,"280":-7.475262641906738,"76":-4.6049699783325195,"277":-11.444707870483398}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.184122045804774,0.06308427601984429,0.7423918053186437,0.010401872856738123],"max_abs_dp":0.001979313476215716,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":423,"_meta":{"row":976,"text_sha256":"fded9b7a4f8b099c6859a4b62ebcd0138ad19b093deab16390cee8e81bdce783","row_sha256":"09199701412203509a8df1c8625f4915047a241dd6f600b4794dfd858a9fcd87","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/976","group_id":"sciq/test/976","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/976"}},"request_sha256":"0e57c4b3a27933cda90e0ec7055df4d13466e25d86e7e722baf516208c5fe11d","questions":[{"type":"choice","n_options":4,"row_len":147,"slot":146,"row_ids_sha256":"13c7ea894c3951c66b01cf5d5c203876623972c3e27133dd4116f3264b22440a","oracle":{"probs":[0.9843759912481487,0.0037127804033644245,0.009242251285813038,0.0026689770626737532],"argmax":"a","top2_margin":0.9751337399623357,"near_tie":false,"group_logz":{"73":-0.04323005676269531,"267":-9.085628509521484,"74":-5.623456954956055,"293":-13.656765937805176,"75":-4.711452484130859,"280":-12.342862129211426,"76":-5.953542709350586,"277":-13.295976638793945}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9843168048230581,0.0037117711229323475,0.009296136828585369,0.002675287225424161],"max_abs_dp":5.918642509061822e-05,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":424,"_meta":{"row":137,"text_sha256":"ba59711c47485dd22635a822488147114de4269bcd004b5f239a21eabba1b52f","row_sha256":"092761ded6b92eda54ab2be77c2961e46f118981511d4bbef3c6a6b4b0b2e173","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/137","group_id":"sciq/test/137","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/137"}},"request_sha256":"47fa29ca3a0672ba9a3e6398b561670fcd00c31eebd4e98c84ec1b66d6f1593d","questions":[{"type":"choice","n_options":4,"row_len":190,"slot":189,"row_ids_sha256":"cbc675b403f76f1f80c4589f913d27eb4b40e4a98580701264e8499f22610b30","oracle":{"probs":[0.0013134575049628472,0.0022357777009903764,0.9932399009651417,0.003210863828905097],"argmax":"c","top2_margin":0.9900290371362366,"near_tie":false,"group_logz":{"73":-6.669424057006836,"267":-14.85601806640625,"74":-6.137497901916504,"293":-14.65717887878418,"75":-0.04111480712890625,"280":-8.859441757202148,"76":-5.775547027587891,"277":-13.756559371948242}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0013073154960612427,0.0022435179097063168,0.9932397603375457,0.0032094062566866955],"max_abs_dp":7.740208715940368e-06,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":425,"_meta":{"row":35,"text_sha256":"47508d01df5c535f3993458b1da869515387561265ba901c455c9f88935cd443","row_sha256":"092c2204f4631862369ba3f128354dd476f8039320d2b30c5f0f4d747668d6c6","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/35","group_id":"sciq/test/35","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/35"}},"request_sha256":"399d11cac3d31315724527758b180c314de6fb4320685418eb67820d346a8c02","questions":[{"type":"choice","n_options":4,"row_len":66,"slot":65,"row_ids_sha256":"c565c7fbbc877eb27e4367a944e87058c48dfe83172b3047fbbcfe6ae3932be9","oracle":{"probs":[0.6599468224422581,0.30006551764710954,0.028916674461819537,0.01107098544881272],"argmax":"a","top2_margin":0.35988130479514857,"near_tie":false,"group_logz":{"73":-0.4688835144042969,"267":-8.259621620178223,"74":-1.2570419311523438,"293":-8.182825088500977,"75":-3.5966243743896484,"280":-10.663248062133789,"76":-4.55671501159668,"277":-11.584479331970215}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.6578636922577202,0.3027488279816101,0.02841877151114941,0.010968708249520155],"max_abs_dp":0.002683310334500588,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":426,"_meta":{"row":147,"text_sha256":"1f357b2078baccd4b71f306ba81f4168d1efa1af175dc640dd49aaafe4ed5807","row_sha256":"092f40e1f611dfa93a75e886c3fc54805a4c9388ad4987b3f75104cd6dcb844f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/147","group_id":"sciq/test/147","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/147"}},"request_sha256":"450f626d71de69a13189e1c98b6b2589be02d6436cdd1ec04149b746bc58402d","questions":[{"type":"choice","n_options":4,"row_len":92,"slot":91,"row_ids_sha256":"6884ec24f301f58502f40ad1d6ffdaef917ca95a7c1077ee1408f562d2626bd6","oracle":{"probs":[0.001409594771762426,0.010377016697967263,0.02760704257077325,0.960606345959497],"argmax":"d","top2_margin":0.9329993033887237,"near_tie":false,"group_logz":{"73":-6.6089067459106445,"267":-14.774791717529297,"74":-4.612615585327148,"293":-12.736239433288574,"75":-3.6341381072998047,"280":-11.631192207336426,"76":-0.08464431762695312,"277":-8.458427429199219}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.001393710344474708,0.010243820355620731,0.027467983751953258,0.9608944855479513],"max_abs_dp":0.00028813958845430587,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":427,"_meta":{"row":587,"text_sha256":"9dac3ba48027e8b7920c28a5d9235c43e6acc3f513b2c3c2ca1a8519e73533a9","row_sha256":"096911b23029621e070636b393a01e0af96ecf891faf1425a9696c0694fff9b4","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/587","group_id":"sciq/test/587","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/587"}},"request_sha256":"f7508e19f367b0dd4ed0fba334299d0714a960174cbdd2d079b0791df90faf7a","questions":[{"type":"choice","n_options":4,"row_len":88,"slot":87,"row_ids_sha256":"3d23f1c28006264d657d476bc7db4ce2c8bf85ec46cf3fd7e926d6acc1a3196c","oracle":{"probs":[0.0018978532377281886,0.003807567373189933,0.9902873357660877,0.004007243622994201],"argmax":"c","top2_margin":0.9862800921430935,"near_tie":false,"group_logz":{"73":-6.305109977722168,"267":-14.198843002319336,"74":-5.608842849731445,"293":-13.9721097946167,"75":-0.04783821105957031,"280":-8.638107299804688,"76":-5.557729721069336,"277":-13.395042419433594}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.001878670650751151,0.0037603843592467316,0.9903697849147682,0.003991160075233945],"max_abs_dp":8.24491486804968e-05,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":428,"_meta":{"row":391,"text_sha256":"bb74034114f171d22ac664d0c93b509caf320f74c1165c459bf85f5ceee71fd9","row_sha256":"0980dc8fcead1e881d97e910a81689b6d90c7efc15f8794274102390abcbd81c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/391","group_id":"sciq/test/391","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/391"}},"request_sha256":"76dd6a05c7c0e426c094adc23f615aaebd067232ef95546221cff8d4c5e867a4","questions":[{"type":"choice","n_options":4,"row_len":127,"slot":126,"row_ids_sha256":"62e11625beb9417597021f17c472c2de0bcca7ab8777a6e48f02f07f27369362","oracle":{"probs":[0.0016254963891352953,0.9912680752236352,0.004030400230595598,0.00307602815663391],"argmax":"b","top2_margin":0.9872376749930396,"near_tie":false,"group_logz":{"73":-6.453708648681641,"267":-14.254356384277344,"74":-0.04053688049316406,"293":-8.368439674377441,"75":-5.545656204223633,"280":-13.537247657775879,"76":-5.815882682800293,"277":-13.848453521728516}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.001611921337791112,0.9913092721842348,0.0040206743267511905,0.0030581321512229745],"max_abs_dp":4.119696059956546e-05,"argmax_equal":true}}}]},{"id":"tv4xh_sciq_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":429,"_meta":{"row":152,"text_sha256":"d802e064623d270f2c3f8943f3a50f5f20dd68352561f76f818dd74b7ed93274","row_sha256":"098e4320fc7563547edc0b02ef6fd8785d06e77c5f13e132ee92b91d4a7bffc9","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/152","group_id":"sciq/test/152","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"sciq/test/152"}},"request_sha256":"5b1b75484a012d2c6f79486a02b54231a5dc52210dd2632a7cc4da98589114ae","questions":[{"type":"choice","n_options":4,"row_len":198,"slot":197,"row_ids_sha256":"bcfb8abf5e4259157c22ba301ad6cb6b25102651aa2d13ded6c105061015cf1d","oracle":{"probs":[0.0016086117965650008,0.9878820753868764,0.006934568819236269,0.0035747439973222448],"argmax":"b","top2_margin":0.9809475065676402,"near_tie":false,"group_logz":{"73":-6.453124046325684,"267":-14.598360061645508,"74":-0.032932281494140625,"293":-8.651225090026855,"75":-4.991976737976074,"280":-12.892019271850586,"76":-5.65460205078125,"277":-13.663146018981934}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0016137455410885103,0.9878957640349805,0.006915849277153592,0.0035746411467773216],"max_abs_dp":1.871954208267773e-05,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":608,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0010","pair_id":"v3-legacy-transfer-authorization-0010","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0010","id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/a","text_sha256":"2ac33202a06425450b3a236002e2add85eca8f5bc6094f6260822e29b29b2074","row_sha256":"8f9e8f9d91b8b3c5ab87f3b8ed6882f368a521c502000b7c37b886935178a782","group_id":"v3-legacy-transfer-authorization-0010","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/a"}},"request_sha256":"6b95b595ff16fb739f3fb7e1dddc06ac5615b6eed786070b125ad7016f666967","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"d28838cd2276a013466d4803e992f9c26d7c0fd2c10aa0af501320bbe171ec67","oracle":{"probs":[0.9773798679790063,0.022620132020993597],"argmax":"yes","top2_margin":0.9547597359580128,"near_tie":false,"group_logz":{"11683":-0.03191184997558594,"12447":-5.382835388183594,"2243":-3.7979469299316406,"4547":-8.615954399108887,"19598":-10.196043014526367}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9768315990197294,0.02316840098027062],"max_abs_dp":0.0005482689592770235,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":609,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0010","pair_id":"v3-legacy-transfer-authorization-0010","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0010","id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/b","text_sha256":"207a5a797dd41df412b0a35a3643f88083c8ba89c2eb3cf1f78538f13eef40d5","row_sha256":"8696645f99fda114766ee32c3b31ebf785c1aa93b1f8d63e2202149d54cc086e","group_id":"v3-legacy-transfer-authorization-0010","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/b"}},"request_sha256":"295a32e069dfdb1e12f05582b125c28ed6cc9a038f3ee6d2cc9e62ccd39edaa0","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"c253794289686eaa38615a4896b98b30a2eae369cbbebc3ff5d099aaad77c712","oracle":{"probs":[0.010068781173480883,0.9899312188265191],"argmax":"no","top2_margin":0.9798624376530383,"near_tie":false,"group_logz":{"11683":-4.603387832641602,"12447":-9.917906761169434,"2243":-0.015192031860351562,"4547":-5.6828813552856445,"19598":-8.800966262817383}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.010071568646609104,0.9899284313533908],"max_abs_dp":2.7874731283183962e-06,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":610,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0011","pair_id":"v3-legacy-transfer-authorization-0011","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0011","id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/a","text_sha256":"4c4b0487b6b6c9b022f3870d584814d80e4f40ac777172038bda483c7f8ab741","row_sha256":"d0a716aa1cb80425b7682ad7671a84ef7f8bf99aa7ad7a74b4e03349f414436f","group_id":"v3-legacy-transfer-authorization-0011","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/a"}},"request_sha256":"a59287c12636b4be3ddff735a7c9c19fa767e034b357d8060f1037903315a1ec","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"1998da4df8fc112db538759fcabcbf7133e9db750851eee8e447851cf74085f6","oracle":{"probs":[0.976293583131362,0.023706416868637885],"argmax":"yes","top2_margin":0.9525871662627242,"near_tie":false,"group_logz":{"11683":-0.032989501953125,"12447":-5.40519905090332,"2243":-3.751007080078125,"4547":-8.54007339477539,"19598":-10.250924110412598}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9752177893904349,0.024782210609565227],"max_abs_dp":0.0010757937409273423,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":611,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0011","pair_id":"v3-legacy-transfer-authorization-0011","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0011","id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/b","text_sha256":"a3fbd4cfe8588ca42161d56c340dafd53e834a8d6b47507611f2f35553c60d2e","row_sha256":"8a00bdd178fc7ab32b2392906cfdd8bb74c04d23842506222d46e8c526de3274","group_id":"v3-legacy-transfer-authorization-0011","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/b"}},"request_sha256":"9f7e1f95eab7538da861f0cda85a4b6edbbc790453e64ff9828cafacb66f3097","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"6f832fa247c089812a0da74f8592c1d934ece30141a6faca569e6b8d3bc46217","oracle":{"probs":[0.010871523965548722,0.9891284760344513],"argmax":"no","top2_margin":0.9782569520689026,"near_tie":false,"group_logz":{"11683":-4.526668548583984,"12447":-9.986695289611816,"2243":-0.0159912109375,"4547":-5.814756393432617,"19598":-8.650226593017578}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.0108607416196718,0.9891392583803281],"max_abs_dp":1.07823458769217e-05,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":612,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0012","pair_id":"v3-legacy-transfer-authorization-0012","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0012","id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/a","text_sha256":"70358674ea1eb7a2b4304de22754ec09d492d0f012fdeb25034914b0487b6de2","row_sha256":"5829235d159024fe1ee0c45ca901977dc2d6c392e53a9bdffe691c975ec41491","group_id":"v3-legacy-transfer-authorization-0012","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/a"}},"request_sha256":"864a18478ab2fb4169fdca760d541287bb96a52b7417bd9e4ee749ac5cacb5cc","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"bd4d410bcaa4b8a405f31d5ee03b0955b3ccf0d17b6853874b5e9193c74680e0","oracle":{"probs":[0.9759407600287442,0.024059239971255766],"argmax":"yes","top2_margin":0.9518815200574884,"near_tie":false,"group_logz":{"11683":-0.03268241882324219,"12447":-5.5542497634887695,"2243":-3.735565185546875,"4547":-8.71066665649414,"19598":-10.271684646606445}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9746730822964095,0.02532691770359043],"max_abs_dp":0.0012676777323347155,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":613,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0012","pair_id":"v3-legacy-transfer-authorization-0012","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0012","id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/b","text_sha256":"e66cbeedd72ceb3350f3dbc36b5eb1ce9df69e2a65fa177459b9c4d916eccb6d","row_sha256":"b1bd359c5413cad9411f83d4280580493a147db3903f6a4399aa9ab6fa35ad7c","group_id":"v3-legacy-transfer-authorization-0012","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/b"}},"request_sha256":"94767e1a0362c1a3a932816616efe9c0b79333f8efb585e76989f5360b57d5e1","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"1f5fe0d99f8b93d36ab34b594c84fab39d4255eeeeb3f2be3baf8ffd8bd94da3","oracle":{"probs":[0.014234832415303426,0.9857651675846966],"argmax":"no","top2_margin":0.9715303351693931,"near_tie":false,"group_logz":{"11683":-4.257596969604492,"12447":-9.637067794799805,"2243":-0.019870758056640625,"4547":-5.715741157531738,"19598":-8.751866340637207}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.01431020797708692,0.9856897920229131],"max_abs_dp":7.537556178349725e-05,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":614,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0013","pair_id":"v3-legacy-transfer-authorization-0013","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0013","id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/a","text_sha256":"9ea1528671ea68f95b395ebe9ac59201e76192ed3ae4a84f5f3731bfbc2d0bc5","row_sha256":"dab2995b774ef92b9a4625ab67555d2eecb87ac4a6165e59ae39b8bc91f13fc0","group_id":"v3-legacy-transfer-authorization-0013","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/a"}},"request_sha256":"cabaf6563fce52e8435dfe9d5075552bc6e98cd3c27dbc660997020b24997309","questions":[{"type":"noul","n_options":2,"row_len":91,"slot":90,"row_ids_sha256":"fe6056a8d84886c39e7eda50720f2a666228d46c7db78a2ba44291e8c4580e82","oracle":{"probs":[0.9784192088731293,0.021580791126870717],"argmax":"yes","top2_margin":0.9568384177462586,"near_tie":false,"group_logz":{"11683":-0.030511856079101562,"12447":-5.427127838134766,"2243":-3.844646453857422,"4547":-8.69394302368164,"19598":-10.193659782409668}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9775313060735302,0.022468693926469687],"max_abs_dp":0.0008879027995990496,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":615,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0013","pair_id":"v3-legacy-transfer-authorization-0013","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0013","id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/b","text_sha256":"d88b238a44fb7d026de87ab14cef084e9c629c16e73655a8e9c5ac2effe3d5cc","row_sha256":"a1cc230ce00109c7e4180f7929b9eb1eb919c39bda07e751562bbefacead253e","group_id":"v3-legacy-transfer-authorization-0013","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/b"}},"request_sha256":"12939d7529cff086f305d1771ce1823f18f92f55ff491e3a1eb8a49cc35e3d1e","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"a00a42bc021b243f28b075dd3366735ab7884d3ecd2e93b119713adc92609f72","oracle":{"probs":[0.010131731476117152,0.9898682685238828],"argmax":"no","top2_margin":0.9797365370477656,"near_tie":false,"group_logz":{"11683":-4.596744537353516,"12447":-10.011009216308594,"2243":-0.014844894409179688,"4547":-5.825291633605957,"19598":-8.81732177734375}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.010116213730275134,0.989883786269725],"max_abs_dp":1.5517745842186237e-05,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":616,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0014","pair_id":"v3-legacy-transfer-authorization-0014","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0014","id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/a","text_sha256":"5e5d94d8e1caaac4ccc98f427dcbf7b88d1ee6640c146ff35f3f4977b6e6735c","row_sha256":"21a16cb236c149612f5fd4a80e84b426b14f02f17665088c0fc8f181ec849e72","group_id":"v3-legacy-transfer-authorization-0014","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/a"}},"request_sha256":"740bf3b08fce2376bbfb53db2babde3581b44504634599b418613be1b2fce9f3","questions":[{"type":"noul","n_options":2,"row_len":93,"slot":92,"row_ids_sha256":"7eb150259a1a21ca0721a29e3e3109ddf1d68bfb2e4e348a358acca9853c5763","oracle":{"probs":[0.9803671456091495,0.019632854390850524],"argmax":"yes","top2_margin":0.960734291218299,"near_tie":false,"group_logz":{"11683":-0.028564453125,"12447":-5.377523422241211,"2243":-3.9392871856689453,"4547":-8.733531951904297,"19598":-10.355803489685059}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9806187598482994,0.01938124015170067],"max_abs_dp":0.0002516142391498777,"argmax_equal":true}}}]},{"id":"tv4xh_legacy_holdout_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":617,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0014","pair_id":"v3-legacy-transfer-authorization-0014","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0014","id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/b","text_sha256":"804b414a261a737f6bab9cd62d420ad7f6aca02cab94b0f86cc27d1af97bf38c","row_sha256":"e51f048b2a4e12e4bceaa23b4edbb570287ebda28dcb7d662ea632e07f677208","group_id":"v3-legacy-transfer-authorization-0014","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/b"}},"request_sha256":"e49c2cf6fcd0ed35acb45d9201cc65b1385c00e6bb80f1965f995ee2088baecb","questions":[{"type":"noul","n_options":2,"row_len":92,"slot":91,"row_ids_sha256":"d603c6997cdd8c36e133b446750abbec6975553c686b898d8c11196974cd48b1","oracle":{"probs":[0.0251725691688095,0.9748274308311905],"argmax":"no","top2_margin":0.949654861662381,"near_tie":false,"group_logz":{"11683":-3.6895885467529297,"12447":-8.751436233520508,"2243":-0.03308296203613281,"4547":-5.368356704711914,"19598":-8.600130081176758}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.02507191083624618,0.9749280891637537],"max_abs_dp":0.00010065833256332063,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_00","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":688,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"251beac991d1b72ad9ca630d5713afaaca28c5546f2dc9d7174ce38417ccaf7b","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/5/relevant/a"}},"request_sha256":"214ebaf5901333cd4d15892b74ed0ca067b5189b7915719d407ce89628f7680a","questions":[{"type":"choice","n_options":2,"row_len":159,"slot":158,"row_ids_sha256":"6371e8f25c465d9f97ba7f63e2db3eb7031c3a54bfdec91f19db3a8ae91c7c13","oracle":{"probs":[0.9205945572331155,0.07940544276688458],"argmax":"reject","top2_margin":0.8411891144662309,"near_tie":false,"group_logz":{"41":-0.08934211730957031,"334":-8.53860092163086,"42":-2.539794921875,"378":-10.446249008178711}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9208980741975344,0.0791019258024656],"max_abs_dp":0.0003035169644189928,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_01","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":689,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"5742c6c5c95abb772f41cf33b0ee4fcd42c492eafdac4ba407bc86336056f6db","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":27,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":true}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/5/relevant/b"}},"request_sha256":"d952088c6373b715238975ce0d1b94f450f1e67ae5778c14ee90976359999f32","questions":[{"type":"choice","n_options":2,"row_len":159,"slot":158,"row_ids_sha256":"bc3266a88c9e7adc12db96e0dfd58b8518a0127c971ecd74d08a661ff407e211","oracle":{"probs":[0.3301387284435438,0.6698612715564563],"argmax":"accept","top2_margin":0.33972254311291245,"near_tie":false,"group_logz":{"41":-1.1212348937988281,"334":-9.174623489379883,"42":-0.4136772155761719,"378":-8.345138549804688}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.3303403810486968,0.6696596189513031],"max_abs_dp":0.00020165260515314642,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_02","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":690,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"251beac991d1b72ad9ca630d5713afaaca28c5546f2dc9d7174ce38417ccaf7b","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/a"}},"request_sha256":"214ebaf5901333cd4d15892b74ed0ca067b5189b7915719d407ce89628f7680a","questions":[{"type":"choice","n_options":2,"row_len":159,"slot":158,"row_ids_sha256":"6371e8f25c465d9f97ba7f63e2db3eb7031c3a54bfdec91f19db3a8ae91c7c13","oracle":{"probs":[0.9205945572331155,0.07940544276688458],"argmax":"reject","top2_margin":0.8411891144662309,"near_tie":false,"group_logz":{"41":-0.08934211730957031,"334":-8.53860092163086,"42":-2.539794921875,"378":-10.446249008178711}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9208980741975344,0.0791019258024656],"max_abs_dp":0.0003035169644189928,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_03","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":691,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e171f541fec96b43ea97de6218051909a0b44ac40e9031152a1bc858f901f36a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":293},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/b"}},"request_sha256":"ce62d2625c457ba244abd0e20ead32da7c78b8814aa92abff0f212ddc6082283","questions":[{"type":"choice","n_options":2,"row_len":159,"slot":158,"row_ids_sha256":"ac6ed1ee71d9af8371f2f1a0637caf6da704e1a27c0a5f4b2cdb01f724a94690","oracle":{"probs":[0.9217993487808199,0.07820065121918007],"argmax":"reject","top2_margin":0.8435986975616399,"near_tie":false,"group_logz":{"41":-0.08800888061523438,"334":-8.543445587158203,"42":-2.555058479309082,"378":-10.46705436706543}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9220933130812854,0.07790668691871454],"max_abs_dp":0.0002939643004655351,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_04","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":692,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"7353c7726b7121059309fd377f668e83744801d7610694938d98685c75d17ac6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/6/relevant/a"}},"request_sha256":"c91f654bbd6a83cfd69a5d1cb4ef18fc7dd681a59eafcb69d13c1eabe76eba56","questions":[{"type":"choice","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"132e5888d360f0e081cdeb9d5ed4b0f7da9c6d6b42929ff0846c32760a25f324","oracle":{"probs":[0.9816213141667125,0.01837868583328736],"argmax":"reject","top2_margin":0.9632426283334251,"near_tie":false,"group_logz":{"41":-0.024499893188476562,"334":-8.902998924255371,"42":-4.002513885498047,"378":-12.421074867248535}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9816680621920827,0.018331937807917375],"max_abs_dp":4.674802537019129e-05,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_05","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":693,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"bd8f6e378d492fa24aa8c05b8eb162a455fccaa1c2d63c0050d5b6948ce7f8a0","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":true,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":true}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/6/relevant/b"}},"request_sha256":"07b768195bb468eca945dbda330b5cc3e1f7db82ab44b2712036bcd46f610b3e","questions":[{"type":"choice","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"35b5182f32370a6de3e63791fceb8b5a85c9ef092c21389651ef1a56c0491e5e","oracle":{"probs":[0.3304669751865198,0.6695330248134802],"argmax":"accept","top2_margin":0.3390660496269604,"near_tie":false,"group_logz":{"41":-1.1201324462890625,"334":-9.148496627807617,"42":-0.4140586853027344,"378":-8.35519027709961}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.33559758280127017,0.6644024171987299],"max_abs_dp":0.005130607614750371,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_06","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":694,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"7353c7726b7121059309fd377f668e83744801d7610694938d98685c75d17ac6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/a"}},"request_sha256":"c91f654bbd6a83cfd69a5d1cb4ef18fc7dd681a59eafcb69d13c1eabe76eba56","questions":[{"type":"choice","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"132e5888d360f0e081cdeb9d5ed4b0f7da9c6d6b42929ff0846c32760a25f324","oracle":{"probs":[0.9816213141667125,0.01837868583328736],"argmax":"reject","top2_margin":0.9632426283334251,"near_tie":false,"group_logz":{"41":-0.024499893188476562,"334":-8.902998924255371,"42":-4.002513885498047,"378":-12.421074867248535}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9816680621920827,0.018331937807917375],"max_abs_dp":4.674802537019129e-05,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_07","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":695,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"95ded910e2fd724e735da663b6f6c9e7e25ca07902beda1a9bf409f494b31402","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":692},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/b"}},"request_sha256":"b6a98245f19d30f75be09619ff99fc89b27f49d5e82916eb23670c91846a0cc2","questions":[{"type":"choice","n_options":2,"row_len":139,"slot":138,"row_ids_sha256":"76986fd1a8ac431f2749563a00415dece8d89c56409a4278eb83ab438209e3d4","oracle":{"probs":[0.981487498508709,0.0185125014912911],"argmax":"reject","top2_margin":0.9629749970174178,"near_tie":false,"group_logz":{"41":-0.02466583251953125,"334":-8.908286094665527,"42":-3.995288848876953,"378":-12.419984817504883}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9815684318895184,0.01843156811048161],"max_abs_dp":8.093338080948989e-05,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_08","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":696,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/7/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/7","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/7/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"11ebc2a7a60f00b085f98a334782aea0934153e1f3797e2ba71fd49ebb4e6e24","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"lt","fields":["request value"],"threshold":14},{"kind":"flag","fields":["package verified"],"threshold":5},{"kind":"ge","fields":["member value"],"threshold":39}],"facts":{"request value":15,"package verified":true,"member value":38,"routing reference":827},"order":["request value","routing reference","package verified","member value"],"deciding_field":"member value","label":false}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/7/relevant/a"}},"request_sha256":"3bf7676458159337a6578a3e426474e655e29ddd691e2f7bee1d835c512e89b5","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"51b7235a4a81d7af119b1582436c020e0cf65b7fa2c0dbee359ad59d4953229a","oracle":{"probs":[0.04247281208687999,0.9575271879131201],"argmax":"reject","top2_margin":0.91505437582624,"near_tie":false,"group_logz":{"41":-3.165891647338867,"334":-11.242661476135254,"42":-0.05040168762207031,"378":-8.0956392288208}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.04283862666675459,0.9571613733332455],"max_abs_dp":0.00036581457987461263,"argmax_equal":true}}}]},{"id":"tv4xh_composition_holdout_09","source":"tv4xh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":697,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/7/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/7","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/7/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"47249d3f980ee4f26c4fdaf8f7f60d8479c35fc4f45d4ea9f0278492a6b4f528","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"lt","fields":["request value"],"threshold":14},{"kind":"flag","fields":["package verified"],"threshold":5},{"kind":"ge","fields":["member value"],"threshold":39}],"facts":{"request value":15,"package verified":true,"member value":39,"routing reference":827},"order":["request value","routing reference","package verified","member value"],"deciding_field":"member value","label":true}},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_sources_heldout","meta_id":"composition/v3-composition-transfer/held_and_or/7/relevant/b"}},"request_sha256":"8c38f481edc0d9b74acfc455fb5806572ba8fda39a44b715c18e87b6a143d461","questions":[{"type":"choice","n_options":2,"row_len":137,"slot":136,"row_ids_sha256":"fd790f48929cf48d451e3ff645a56b0641024d3549140c6935ce1d929d6efd7a","oracle":{"probs":[0.12241272726435322,0.8775872727356467],"argmax":"reject","top2_margin":0.7551745454712935,"near_tie":false,"group_logz":{"41":-2.1075515747070312,"334":-10.314126968383789,"42":-0.1377735137939453,"378":-8.336631774902344}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.12495775307260769,0.8750422469273923],"max_abs_dp":0.0025450258082544697,"argmax_equal":true}}}]},{"id":"tv4sh_00","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":648,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0010","pair_id":"v3-legacy-transfer-deadline-0010","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0010","id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/a","text_sha256":"7b32f3b140896e6014ca68fe6005032694325a0fe586deedec53c90f3edfdf57","row_sha256":"abb6f317836f5b1c653110cd9a1a2bb6caa6ffde537b07e10eae0b242268a870","group_id":"v3-legacy-transfer-deadline-0010","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/a"}},"request_sha256":"81d6766001d1987e0d1f732b862be05f171e5c34d40ef16b19eafaafd7772e7f","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"d29afdc797c5f476ab0487fc12c9f840d280268a5f48114ede93887829c2623a","oracle":{"probs":[0.9606380993387349,0.036785672371009805,0.0025762282902552554],"argmax":"0","top2_margin":0.9238524269677251,"near_tie":false,"group_logz":{"24":-0.054027557373046875,"25":-3.316516876220703,"26":-5.975298881530762}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.960578682962643,0.03683542449074672,0.0025858925466102542],"max_abs_dp":5.941637609185868e-05,"argmax_equal":true}}}]},{"id":"tv4sh_01","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":649,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0010","pair_id":"v3-legacy-transfer-deadline-0010","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0010","id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/b","text_sha256":"3209194f4ecf26e0b1ee079354d40c7ca89a478a7a4ee6a7dba34ab145dcba2c","row_sha256":"12054c79dfc1493a2fdb659a1f1fa4a96959f13c98c6a42e94e5494c18b2aba6","group_id":"v3-legacy-transfer-deadline-0010","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/b"}},"request_sha256":"af9b6251f8357a68e1409766f1da2c2256524785bbd11cbb845c25f047c047ea","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"393b49c0c83f9f3b27f93fb70e1563ad01988342be12b200d28cf1a0dfb9b05f","oracle":{"probs":[0.004581847471152039,0.4492999702994483,0.5461181822293996],"argmax":"2","top2_margin":0.09681821192995127,"near_tie":false,"group_logz":{"24":-5.4231414794921875,"25":-0.8375530242919922,"26":-0.6424083709716797}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.004700667468325669,0.458867106918084,0.5364322256135903],"max_abs_dp":0.009685956615809288,"argmax_equal":true}}}]},{"id":"tv4sh_02","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":650,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0011","pair_id":"v3-legacy-transfer-deadline-0011","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0011","id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/a","text_sha256":"caa7e35da81a3cb28a58aed070b8862a1b9d3af997e9743b76b3293e18974942","row_sha256":"858badab820e2dfef0501519d23eb6bce180ca7971473cfe705f673611d68915","group_id":"v3-legacy-transfer-deadline-0011","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/a"}},"request_sha256":"754bce86b5139ff26f7df117bd5c751cdbc02f93ae565d58d60f9cd16d865d69","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"afd996a4877156fcabf7bdc874589b2b1121162341300b16c6827ef9e0ad90b7","oracle":{"probs":[0.858864075714667,0.13769569960504943,0.0034402246802834892],"argmax":"0","top2_margin":0.7211683761096176,"near_tie":false,"group_logz":{"24":-0.16533279418945312,"25":-1.9958972930908203,"26":-5.685406684875488}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.8560885028003222,0.1404581275589325,0.0034533696407452554],"max_abs_dp":0.002775572914344826,"argmax_equal":true}}}]},{"id":"tv4sh_03","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":651,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0011","pair_id":"v3-legacy-transfer-deadline-0011","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0011","id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/b","text_sha256":"5c5a0edb17d07ebd90c032acf716143b30671f5316b69e17723fb59ec9bd7c69","row_sha256":"f15a66a6b56cb9946b7968bdb77c53ae81e58f8d5825504b8b20ea286e8cc003","group_id":"v3-legacy-transfer-deadline-0011","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/b"}},"request_sha256":"984998de589bf38f32c8f555af0f6ec51bc1c4ffc51b2a029fafaad80aec1b20","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"eede3379e77626526568f54a71414db4eb2185a952a4265412a67bbb86aa59b0","oracle":{"probs":[0.00401403496959601,0.9366223313195027,0.05936363371090135],"argmax":"1","top2_margin":0.8772586976086013,"near_tie":false,"group_logz":{"24":-5.5338850021362305,"25":-0.08140182495117188,"26":-2.8400001525878906}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.004060492711258252,0.9381271662576476,0.05781234103109418],"max_abs_dp":0.0015512926798071716,"argmax_equal":true}}}]},{"id":"tv4sh_04","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":652,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0012","pair_id":"v3-legacy-transfer-deadline-0012","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0012","id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/a","text_sha256":"a79e54bf80e987aa5dcbe6ac7adecb143981024bf41397e311360bec7a034e3c","row_sha256":"fc9f9d8c34210cdb9a943bfb17443651fe3bff5c41ab5462fd0e85b23ebb9bae","group_id":"v3-legacy-transfer-deadline-0012","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/a"}},"request_sha256":"0168b36c0515466fbb230025d0fb8764bcc6afdcf39ac8b04c22d17ab989f0d6","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"5c97014a91bbfca698a7875842f79d808c44bee265ca970f048478e02414886c","oracle":{"probs":[0.7789573975668233,0.21784515811328203,0.003197444319894609],"argmax":"0","top2_margin":0.5611122394535413,"near_tie":false,"group_logz":{"24":-0.2650604248046875,"25":-1.5392322540283203,"26":-5.760664939880371}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.7749054809257575,0.22185689700273015,0.003237622071512317],"max_abs_dp":0.004051916641065745,"argmax_equal":true}}}]},{"id":"tv4sh_05","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":653,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0012","pair_id":"v3-legacy-transfer-deadline-0012","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0012","id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/b","text_sha256":"a056b9ce266919cbb722895aa4d4432cc469b06fc27ebd0c677238d34200fb8e","row_sha256":"dfcb3fdbcf88be8c45cb2a71d511d19a58245a30c3e8fb7fd1bdc39e0f1fca92","group_id":"v3-legacy-transfer-deadline-0012","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/b"}},"request_sha256":"a2c91fa6c2de2191e0a0641d77ad4ea38bccb95a016d6fef6c5861412d71c1ff","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"2ec7a1025d8f49dddf855f56a43fa140075df1e239029279ccf406907457ba53","oracle":{"probs":[0.007049913219904463,0.7015928778383044,0.2913572089417913],"argmax":"1","top2_margin":0.4102356688965131,"near_tie":false,"group_logz":{"24":-5.002157211303711,"25":-0.40181922912597656,"26":-1.2806224822998047}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.007204710790398134,0.715444319402025,0.2773509698075769],"max_abs_dp":0.014006239134214393,"argmax_equal":true}}}]},{"id":"tv4sh_06","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":654,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0013","pair_id":"v3-legacy-transfer-deadline-0013","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0013","id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/a","text_sha256":"1707ca65b05430e955da5abed9f42eb596b77c27a3e3f8a81d11d63a11235117","row_sha256":"ba18be7789b6139d2981db43e91aa5fa49ee0341070ef987283988963ebedc51","group_id":"v3-legacy-transfer-deadline-0013","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/a"}},"request_sha256":"bfb13370b4c1bbdc866b930656ad56fcef5535b9e26db1e3418ab7f2e4ea303b","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"952e6953812aebea921c7e61fa1cef11b21cac145ea15883d4a92d812b002388","oracle":{"probs":[0.01885313362405331,0.9662898919151223,0.014856974460824455],"argmax":"1","top2_margin":0.947436758291069,"near_tie":false,"group_logz":{"24":-3.9891109466552734,"25":-0.052326202392578125,"26":-4.227320671081543}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.018924138968206265,0.9660525383090411,0.015023322722752546],"max_abs_dp":0.00023735360608112543,"argmax_equal":true}}}]},{"id":"tv4sh_07","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":655,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0013","pair_id":"v3-legacy-transfer-deadline-0013","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0013","id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/b","text_sha256":"f39da0354802898bbde918103b780c20c08561afa8524949e67d37c58f3aaffa","row_sha256":"6f39642be84beb8ea612a1be593599dfd6a81c4b0165a317b7f88580adb32118","group_id":"v3-legacy-transfer-deadline-0013","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/b"}},"request_sha256":"b3fa4bc82daf3ccc4fa851aae42477fd553292d291a1eebafcf995a96fc06e82","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"34039d24dcdb02eaf3517531057d3322e01fda761318659ac2cc02df9ec4b034","oracle":{"probs":[0.019621320574416876,0.931821187141498,0.04855749228408519],"argmax":"1","top2_margin":0.8832636948574127,"near_tie":false,"group_logz":{"24":-4.024018287658691,"25":-0.16349411010742188,"26":-3.117886543273926}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.01992836369426909,0.9315285857118618,0.04854305059386916],"max_abs_dp":0.00030704311985221486,"argmax_equal":true}}}]},{"id":"tv4sh_08","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":656,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0014","pair_id":"v3-legacy-transfer-deadline-0014","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0014","id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/a","text_sha256":"fc9977e9dd9c23766d739280ab1085eaf5d91e7be1a3ec9e0092f7bfa4ebbf6c","row_sha256":"a4d8941e13b548ecf801d5ef8154ecb8944e0879b85db56e725f5c5df8d8ad2d","group_id":"v3-legacy-transfer-deadline-0014","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/a"}},"request_sha256":"791aa4ce1aac1d17c4a8081f57ed43930dfd8b66853e7ea64c5a349ead828816","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"105db773683cb7375cfb15037f392d5c95b72757ac78014e6bfdd607486306f5","oracle":{"probs":[0.15533867970940582,0.8378251343601165,0.006836185930477585],"argmax":"1","top2_margin":0.6824864546507107,"near_tie":false,"group_logz":{"24":-1.8713455200195312,"25":-0.1861438751220703,"26":-4.994723320007324}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.15301541587956036,0.8401731663954479,0.00681141772499172],"max_abs_dp":0.0023480320353314044,"argmax_equal":true}}}]},{"id":"tv4sh_09","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":657,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0014","pair_id":"v3-legacy-transfer-deadline-0014","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0014","id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/b","text_sha256":"e0777025a5b0c26270b3e5955629fbef0413904eeebd12540cde582b64449ec7","row_sha256":"5754f6c7211511e9a1a3dd1dcb09e6a325ed9facc94a9300fc4033c0bf4e37a1","group_id":"v3-legacy-transfer-deadline-0014","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/b"}},"request_sha256":"b78f62502896f103846a620f81afad1845a762750c842f235f39cc5c97de9902","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"8b0ddeb3bac7b771c6c46ccd6ef815e05d53e8715c2ad3a40497d775352685b1","oracle":{"probs":[0.017588326685839,0.8506627831265388,0.1317488901876223],"argmax":"1","top2_margin":0.7189138929389165,"near_tie":false,"group_logz":{"24":-4.222623825073242,"25":-0.3438434600830078,"26":-2.2089614868164062}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.01795885019956309,0.8574548151258197,0.12458633467461722],"max_abs_dp":0.00716255551300507,"argmax_equal":true}}}]},{"id":"tv4sh_10","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":658,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0015","pair_id":"v3-legacy-transfer-deadline-0015","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0015","id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/a","text_sha256":"e698380e12ef3e684f4bbf2bfb49e21b88c13030cc4305f39172002bf75a2581","row_sha256":"ca1846312146ab4b7ede6c0348eb01ccae4a9e484bcd8c355a8f8497414e0479","group_id":"v3-legacy-transfer-deadline-0015","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/a"}},"request_sha256":"9194a3c79162509539d35f50238866267f150f0af6b16f64aa152298121419f7","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"47d00403db2729a8452c230c138ab55b3701eaf89605c8e5ca326f5915ee2f6c","oracle":{"probs":[0.7435471062057365,0.2501220153393281,0.00633087845493543],"argmax":"0","top2_margin":0.4934250908664084,"near_tie":false,"group_logz":{"24":-0.316131591796875,"25":-1.4056148529052734,"26":-5.082124710083008}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.7400274317735935,0.25354518008867116,0.006427388137735216],"max_abs_dp":0.0035196744321429607,"argmax_equal":true}}}]},{"id":"tv4sh_11","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":659,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0015","pair_id":"v3-legacy-transfer-deadline-0015","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0015","id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/b","text_sha256":"7d4d32bb0f70babb9e567683e98f6cfb068535585c37276488434bba5c61d5e1","row_sha256":"90ebc7661f64c68aef69fa8feba55e02ac0c081b22af11b0fb6a33f1770a81fe","group_id":"v3-legacy-transfer-deadline-0015","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/b"}},"request_sha256":"6396d91afc78581f24d3bb69578a1755556f5702ef157655a47b807e59cd4933","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"b834d98ae3cffbaaaa20c7d7eb571a97b1e9ec7b2d10f597a095e1fb8ee92b83","oracle":{"probs":[0.010838330032698743,0.9860848679874247,0.003076801979876621],"argmax":"1","top2_margin":0.9752465379547259,"near_tie":false,"group_logz":{"24":-4.532269477844238,"25":-0.021615982055664062,"26":-5.791467666625977}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.010800220307942102,0.986136289312694,0.0030634903793638948],"max_abs_dp":5.142132526936294e-05,"argmax_equal":true}}}]},{"id":"tv4sh_12","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":660,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0016","pair_id":"v3-legacy-transfer-deadline-0016","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0016","id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/a","text_sha256":"90947706e3d61df0db4c3c44469df55f90dd95289439b06a1b047f9ecb80d3c4","row_sha256":"4609e1f514173bb6cc53fcef82c8b16f934b5b6c8552661a2e01ebf1569a53ac","group_id":"v3-legacy-transfer-deadline-0016","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/a"}},"request_sha256":"47d7d5a14d20bcb4ada04f89d9f28bd36dd1d67d2772143af128e6cdbf5afeee","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"8393d89637e54625177653b005aba8a7d595af3d76f175bd8c2a3329ed15f792","oracle":{"probs":[0.11315607580390194,0.881623995933694,0.005219928262404048],"argmax":"1","top2_margin":0.768467920129792,"near_tie":false,"group_logz":{"24":-2.1871185302734375,"25":-0.13412094116210938,"26":-5.263402938842773}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.11341264710896075,0.8814287025779303,0.005158650313108858],"max_abs_dp":0.00025657130505880865,"argmax_equal":true}}}]},{"id":"tv4sh_13","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":661,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0016","pair_id":"v3-legacy-transfer-deadline-0016","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0016","id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/b","text_sha256":"3345c00abfc381ed9e7d49dd4830b6b691edce70625996b261b66e517cac596f","row_sha256":"67d688120c333a429f9c98361e43b62a00c42acbc93dc99b9b2c76b598f999bf","group_id":"v3-legacy-transfer-deadline-0016","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/b"}},"request_sha256":"4170afe69f99d988b24e73f9c231f0e848a645e4e80d5cf684550106e035bc41","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"ccfee1a4498a4e51208bf318eb3a80606d6593d2bf8955a750198949d756dd3c","oracle":{"probs":[0.007568019152534779,0.9740214696048085,0.01841051124265677],"argmax":"1","top2_margin":0.9556109583621517,"near_tie":false,"group_logz":{"24":-4.944452285766602,"25":-0.08695030212402344,"26":-4.055461883544922}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.00784678958850203,0.9735110136980055,0.01864219671349231],"max_abs_dp":0.0005104559068029868,"argmax_equal":true}}}]},{"id":"tv4sh_14","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":662,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0017","pair_id":"v3-legacy-transfer-deadline-0017","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0017","id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/a","text_sha256":"e6cf0c98a6e024fc739907f5cb3a3fca844d42b56ab463d46c2033ee7a7eb8a7","row_sha256":"cd426821d0d22f03f1de40e3dadb4d88c16109b010c7620270dd516f0bf0b93b","group_id":"v3-legacy-transfer-deadline-0017","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/a"}},"request_sha256":"ef251031e4b529d8e6012fdb8d1baedbddfffbcaea4504aedb06cea346c2408b","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"fae6cfa0bf1d66d8286a7164b81ea9b055aecb9e62bfa50dd7a1f3017ffdaa2e","oracle":{"probs":[0.9287801482631354,0.0693143177938051,0.0019055339430596062],"argmax":"0","top2_margin":0.8594658304693302,"near_tie":false,"group_logz":{"24":-0.08254051208496094,"25":-2.6777610778808594,"26":-6.271650314331055}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.9301814348667717,0.06791965415273508,0.0018989109804933076],"max_abs_dp":0.0014012866036363647,"argmax_equal":true}}}]},{"id":"tv4sh_15","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":663,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0017","pair_id":"v3-legacy-transfer-deadline-0017","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0017","id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/b","text_sha256":"04ab254f4aeb306a27d84d8abc28714ad898019447b8e87ba41f63e5ac185ca7","row_sha256":"623b716f5719dd85aa3dbccab13695554b8131b3380aefb8c3511b23f26b244f","group_id":"v3-legacy-transfer-deadline-0017","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/b"}},"request_sha256":"64905ea1e3620256129b503dd029018fc3a99ecac02769c890f60192179bc988","questions":[{"type":"score","n_options":3,"row_len":121,"slot":120,"row_ids_sha256":"626e51aa437a9c24c318608f93bf6e29fa9d19182bf66c234efefaef2b9bde39","oracle":{"probs":[0.005941741981940223,0.9741032601233788,0.019954997894681106],"argmax":"1","top2_margin":0.9541482622286976,"near_tie":false,"group_logz":{"24":-5.159848213195801,"25":-0.060333251953125,"26":-3.948370933532715}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.005949462322507202,0.9740404737457089,0.020010063931783895],"max_abs_dp":6.278637766987671e-05,"argmax_equal":true}}}]},{"id":"tv4sh_16","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":664,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0018","pair_id":"v3-legacy-transfer-deadline-0018","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0018","id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/a","text_sha256":"3159faf45f6f6cbd83e78bec26087c97cc2a53e96923f3e8543e0d16d8bea505","row_sha256":"648c0e4938dc08cf8bc819191e4105d3f39788b83d126e5b9ffe25c8e44634d7","group_id":"v3-legacy-transfer-deadline-0018","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/a"}},"request_sha256":"290e9bbe66c416209606be83c4e2bc880dce97762f145620313d6b2c56087146","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"43a2bc869dd2aeb6a89d4116a01dea06136cf8c6dbc12593bcf6148b73ee6446","oracle":{"probs":[0.1730645548709054,0.8216576495808657,0.005277795548228909],"argmax":"1","top2_margin":0.6485930947099603,"near_tie":false,"group_logz":{"24":-1.764089584350586,"25":-0.20643043518066406,"26":-5.254245758056641}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.18036876440449065,0.8143569989518851,0.005274236643624221],"max_abs_dp":0.0073042095335852375,"argmax_equal":true}}}]},{"id":"tv4sh_17","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":665,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0018","pair_id":"v3-legacy-transfer-deadline-0018","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0018","id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/b","text_sha256":"7e53a08ce33836290f3ad26ef97343372fcca8d9b3be8ad7fc494d59a4e8c384","row_sha256":"0b0627231d64cb4c8b0ac99d7806214c71ab8700ffc779ca420b6bff1b643064","group_id":"v3-legacy-transfer-deadline-0018","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/b"}},"request_sha256":"68f894ec4caa04349e4149cfe759f6918e84b228198d6415f7c44d6b54f15e2e","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"b34cd190b8ba6491142933b678b941ebf0c06f56e0509fce27aa92e988d60f7a","oracle":{"probs":[0.0178513199630641,0.8964744378431925,0.08567424219374348],"argmax":"1","top2_margin":0.810800195649449,"near_tie":false,"group_logz":{"24":-4.413168907165527,"25":-0.4967765808105469,"26":-2.844694137573242}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.018324031063154993,0.898198444900371,0.08347752403647392],"max_abs_dp":0.0021967181572695588,"argmax_equal":true}}}]},{"id":"tv4sh_18","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":666,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0019","pair_id":"v3-legacy-transfer-deadline-0019","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0019","id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/a","text_sha256":"498d90fc103eca25d611d80676bdcda336cd1ce79841f5d14213f7dacae8c71e","row_sha256":"3fd388a669406a32788e1db619b49c1bf72ebb426c0bb62cb61377d0db98092a","group_id":"v3-legacy-transfer-deadline-0019","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/a"}},"request_sha256":"4696336f4619441d3f47237ac6e85d8d211942f4438380d4d1bec02cdc6075a2","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"4173230f8a90606b1663a4470ecf94bbac9f3491f2897ce5568e936d93c36df5","oracle":{"probs":[0.06548673030791495,0.9279043131156535,0.006608956576431643],"argmax":"1","top2_margin":0.8624175828077385,"near_tie":false,"group_logz":{"24":-2.7353134155273438,"25":-0.08423233032226562,"26":-5.028735160827637}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.06477726926247838,0.9286855023242508,0.006537228413270678],"max_abs_dp":0.0007811892085973771,"argmax_equal":true}}}]},{"id":"tv4sh_19","source":"tv4sh","publication":"reference_only","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":667,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0019","pair_id":"v3-legacy-transfer-deadline-0019","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0019","id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/b","text_sha256":"c9e31ed73d47b668f7343567a07269bca02606c680b8c6ebbbcb35055b8edfa7","row_sha256":"72b880321dae0e88d1ebc8287fa4a9aab2c344959fb6913f203eb27acacf8a60","group_id":"v3-legacy-transfer-deadline-0019","variant":"clean"},"kev_heldout":{"file":"_kev/fixtures/heldout.json","sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df","kev_source":"transfer_v4_dev_score_heldout","meta_id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/b"}},"request_sha256":"8d4ada34e60d889f24622fa47d86f458129c6b7e910c331ccf66eb0f963cc39e","questions":[{"type":"score","n_options":3,"row_len":122,"slot":121,"row_ids_sha256":"13a51a10092cd776a73a1b4f49aa580f07850ddcd2dc7b9393bee6b94d207f76","oracle":{"probs":[0.03936405079217363,0.8531181249483566,0.10751782425946979],"argmax":"1","top2_margin":0.7456003006888868,"near_tie":false,"group_logz":{"24":-3.321537971496582,"25":-0.24549293518066406,"26":-2.3167343139648438}},"graph":{"mac_fp16":null,"iphone_int8mlp":{"p":[0.039356498680048486,0.853941411853882,0.10670208946606966],"max_abs_dp":0.0008232869055253556,"argmax_equal":true}}}]}]},"image_records":[{"id":"img01_shapes_384x384","publication":"publish_text_and_generator","request":{"state":null,"questions":{"count":{"type":"choice","instructions":"How many shapes are in the picture?","criteria":{"one":"One","two":"Two","three":"Three","four_or_more":"Four or more"}},"circle":{"type":"noul","instructions":"Is there a circle in the picture?"}}},"images":["images/img01_shapes_384x384.png"],"gold":{"count":"three","circle":"true"},"note":"drawn here (one crop 24x24, native size); gold from the drawing (meta.json drawn)","picture":{"size":[384,384],"sha256":"37d6e9b1256a91fafcc209be6d2dc9971414bdd062ad661d4d10bf5afe7fba13","rgb_sha256":"96d85daed80c0e225241215bfbe0e1b0373bf4789f53e7bf8b5e9a97f59ff582","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img01","seed":1},"drawn":{"shapes":[{"kind":"circle","color":"red","area":15394},{"kind":"square","color":"blue","area":8100},{"kind":"triangle","color":"green","area":4500}],"n_shapes":3,"has_circle":true,"largest_color":"red","largest_area":15394,"next_other_color_area":8100,"crowded_level":1,"kinds":["circle","square","triangle"],"colors":["blue","green","red"]}},"questions":[{"name":"count","type":"choice","keys":["one","two","three","four_or_more"],"codes":["A","B","C","D"],"gold":"three","row_len":186,"slot":185,"n_image_tokens":144,"row_ids_sha256":"ba05ecc97679459ce5435698e0c5feed85cf13ad457b2dae484318a9e9484f25","oracle":{"probs":[0.003987268477159175,0.005595587608682278,0.9838073286911434,0.006609815223015139],"argmax":"three","top2_margin":0.9771975134681282,"near_tie":false,"group_logz":{"41":-5.532680511474609,"334":-14.288018226623535,"42":-5.193808555603027,"378":-13.479687690734863,"43":-0.024356842041015625,"340":-8.518049240112305,"44":-5.027231216430664,"388":-13.479619026184082}},"graph":{"mac_fp16":{"p":[0.003997599244674326,0.0056150925933302455,0.9837645819187253,0.006622726243270092],"p_bits":["3f705fc9604b53df","3f76ffd9f0d67ced","3fef7affdc49baae","3f7b206e8a00d7fe"],"hidden_sha256":"f036249f69819b7507f098b751b1e380c6da328dfa645582d42be7d44f256f08"},"iphone_int8mlp":{"p":[0.003960481116912126,0.0055757513961472785,0.983843288127606,0.0066204793593345485],"p_bits":["3f7038dd8dfa1d56","3f76d6995e5a6f68","3fef7ba4eb5cb5a4","3f7b1e136550a122"],"hidden_sha256":"afda2041b9356201ef2008fc73889b8985bc7e57c6e4757fae67133f25a9aae9"}}},{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"yes","row_len":170,"slot":169,"n_image_tokens":144,"row_ids_sha256":"23a3eb812f02d310e4ddfedf2696abc458f2a772222fd64d56e3dfaa4c834bfc","oracle":{"probs":[0.9905731368515412,0.00942686314845881],"argmax":"yes","top2_margin":0.9811462737030824,"near_tie":false,"group_logz":{"11683":-0.025640487670898438,"12447":-4.366167068481445,"2243":-4.680360794067383,"4547":-8.433160781860352,"19598":-11.048224449157715}},"graph":{"mac_fp16":{"p":[0.9905790489698281,0.009420951030171935],"p_bits":["3fefb2d2d56db3c4","3f834b4aa4930f02"],"hidden_sha256":"441813969b4e561602ad8dbf9acf432caf8480374a5c75773a39ad16113005f9"},"iphone_int8mlp":{"p":[0.9906451814445757,0.009354818555424246],"p_bits":["3fefb35d8607d271","3f83289e7e0b639e"],"hidden_sha256":"042c1168dfdcd9ff32b7a5f8f6c43bdb3341bc9172da17bccbfc8e3c84d05855"}}}]},{"id":"img02_circle_384x256","publication":"publish_text_and_generator","request":{"state":"A picture from a shape-sorting exercise.","questions":{"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}},"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]}}},"images":["images/img02_circle_384x256.png"],"gold":{"largest":"orange","crowded":"0"},"note":"drawn here (one crop 16x24, native size); gold from the drawing (meta.json drawn)","picture":{"size":[384,256],"sha256":"b13c07b49e5b7b92fa2177fd14b4a4e3ad63550e9e02560a6d58a376d56666a3","rgb_sha256":"44740636cb5b2d9fd147acbe235ba2a5ed188a6699b64a3c23b2dae0e4c7e2d7","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img02","seed":2},"drawn":{"shapes":[{"kind":"circle","color":"orange","area":31416}],"n_shapes":1,"has_circle":true,"largest_color":"orange","largest_area":31416,"next_other_color_area":null,"crowded_level":0,"kinds":["circle"],"colors":["orange"]}},"questions":[{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"orange","row_len":152,"slot":151,"n_image_tokens":96,"row_ids_sha256":"4268405d14447669970b67b18c4a72c2662401d5569ea225d6af3b151ec0f6fd","oracle":{"probs":[0.0055371850104255224,0.9780104640474168,0.005723271615102888,0.004595573082015882,0.0033002705397884124,0.002833235705250595],"argmax":"orange","top2_margin":0.9722871924323139,"near_tie":false,"group_logz":{"41":-5.204669952392578,"334":-13.6603364944458,"42":-0.030635833740234375,"378":-8.213479995727539,"43":-5.1716156005859375,"340":-13.303817749023438,"44":-5.3910627365112305,"388":-13.782194137573242,"45":-5.722151756286621,"407":-13.507107734680176,"46":-5.874736785888672,"410":-14.23171615600586}},"graph":{"mac_fp16":{"p":[0.005512595575463124,0.9781140440705955,0.005695213677473226,0.004571120686883596,0.003285345790209965,0.0028216801993746613],"p_bits":["3f7694601b672bf8","3fef4cb5d2e1540e","3f7753dd47d0cfc6","3f72b92addb32287","3f6ae9de973339a7","3f671d7e05a27cdd"],"hidden_sha256":"6e78e47bbf7114220ed5038ebacde159d666511e423a8dbf1675db175566c401"},"iphone_int8mlp":{"p":[0.005522804072380622,0.978090334073773,0.005706169666381401,0.0045716553063113955,0.0032793317129083703,0.0028297051682450746],"p_bits":["3f769f146df82da1","3fef4c8419ac2da0","3f775f5a41a3fc1b","3f72b9ba6077925a","3f6add41ceb5acc6","3f672e5264f539c5"],"hidden_sha256":"9753e31966d2637a8223f2b85835211472cbde251f38df3d6baddc0cf58deff6"}}},{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"0","row_len":150,"slot":149,"n_image_tokens":96,"row_ids_sha256":"998e115eeed93330e7b444e41e5e100fddc2ec639ad560489d8388a7933af831","oracle":{"probs":[0.9307184601119837,0.06423353518123791,0.005048004706778327],"argmax":"0","top2_margin":0.8664849249307458,"near_tie":false,"group_logz":{"24":-0.08300971984863281,"25":-2.756441116333008,"26":-5.299973487854004}},"graph":{"mac_fp16":{"p":[0.9310048442807036,0.06395890559861186,0.00503625012068458],"p_bits":["3fedcacaabd34d5f","3fb05f9c5fd58052","3f74a0e419014bb1"],"hidden_sha256":"7d5511a68da7d1bda732a53db58c736c37e7fea01d0fa71525fee38b4cddd1ca"},"iphone_int8mlp":{"p":[0.9313106438148724,0.06365384513461479,0.005035511050512779],"p_bits":["3fedcd4bfab34028","3fb04b9e4f1e2967","3f74a01db47d555e"],"hidden_sha256":"7f1022e667020076e85c200608f8bd51f0408f9e570589376dbf8fd6f2dd995e"}}}]},{"id":"img03_bands_256x384","publication":"publish_text_and_generator","request":{"state":null,"questions":{"circle":{"type":"noul","instructions":"Is there a circle in the picture?"}}},"images":["images/img03_bands_256x384.png"],"gold":{"circle":"false"},"note":"drawn here (one crop 24x16, background bands); gold from the drawing (meta.json drawn)","picture":{"size":[256,384],"sha256":"b06a97f5dbb129725c7f18ea4e33dafbd491ec95c3ec8ca7d87b4d2443487ef8","rgb_sha256":"ae1f4b0fe0392798af60147bf9589e48b4409495fdd5750e67c04869b394aca0","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img03","seed":3},"drawn":{"shapes":[{"kind":"triangle","color":"purple","area":9750},{"kind":"triangle","color":"purple","area":4500}],"n_shapes":2,"has_circle":false,"largest_color":"purple","largest_area":9750,"next_other_color_area":null,"crowded_level":0,"kinds":["triangle"],"colors":["purple"]}},"questions":[{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"no","row_len":122,"slot":121,"n_image_tokens":96,"row_ids_sha256":"35d6371e8617261d594b619a6c67867d7fddb90452bb5e9b5d71792d9cafe88d","oracle":{"probs":[0.013220681637372897,0.9867793183626272],"argmax":"no","top2_margin":0.9735586367252543,"near_tie":false,"group_logz":{"11683":-4.346183776855469,"12447":-8.153595924377441,"2243":-0.033519744873046875,"4547":-4.069881439208984,"19598":-8.168102264404297}},"graph":{"mac_fp16":{"p":[0.013204407560893701,0.9867955924391062],"p_bits":["3f8b0ae99518c484","3fef93d459ab9ced"],"hidden_sha256":"ecea28cd1519b7195ff3174a269eda1d380e637e3e581c2a130a8bd487a96dd9"},"iphone_int8mlp":{"p":[0.013226541946731028,0.986773458053269],"p_bits":["3f8b168468cd950c","3fef93a5ee5cc9ac"],"hidden_sha256":"0f3a52d97d0144a4f9ab6384248ca6a777f602180cdb80492a623b5c7ec57230"}}}]},{"id":"img04_scatter_640x480","publication":"publish_text_and_generator","request":{"state":{"source":"drawing","sheet":4},"questions":{"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]},"circle":{"type":"noul","instructions":"Is there a circle in the picture?"},"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}}}},"images":["images/img04_scatter_640x480.png"],"gold":{"crowded":"2","circle":"true","largest":"blue"},"note":"drawn here (one crop 26x36 (the 640x480 shape of the card's COCO example), JSON state); gold from the drawing (meta.json drawn)","picture":{"size":[640,480],"sha256":"e5326e97a0593356dc58181dccb499df20902ca18ecf071dc5c2890c96ca158e","rgb_sha256":"3e662c4200ffe1f4e7e1457a4de55b673a13a2b6fde8e8e4ce33667db6a3d032","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img04","seed":4},"drawn":{"shapes":[{"kind":"square","color":"blue","area":36100},{"kind":"circle","color":"red","area":1521},{"kind":"triangle","color":"green","area":418},{"kind":"circle","color":"red","area":1385},{"kind":"circle","color":"orange","area":1257},{"kind":"circle","color":"yellow","area":1018},{"kind":"triangle","color":"yellow","area":418},{"kind":"square","color":"yellow","area":625},{"kind":"triangle","color":"yellow","area":817},{"kind":"square","color":"yellow","area":576},{"kind":"triangle","color":"yellow","area":464},{"kind":"square","color":"purple","area":961},{"kind":"square","color":"red","area":484},{"kind":"triangle","color":"yellow","area":324},{"kind":"triangle","color":"green","area":576},{"kind":"circle","color":"yellow","area":707},{"kind":"square","color":"orange","area":1024},{"kind":"circle","color":"yellow","area":1018},{"kind":"triangle","color":"yellow","area":324}],"n_shapes":19,"has_circle":true,"largest_color":"blue","largest_area":36100,"next_other_color_area":1521,"crowded_level":2,"kinds":["circle","square","triangle"],"colors":["blue","green","orange","purple","red","yellow"]}},"questions":[{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"2","row_len":296,"slot":295,"n_image_tokens":234,"row_ids_sha256":"15b85f870618f55ce8623e37b9c8f2a55f1be0a68c20f1481daa7dfa2cb1cf6a","oracle":{"probs":[0.015116076045447424,0.10277313942911995,0.8821107845254327],"argmax":"2","top2_margin":0.7793376450963128,"near_tie":false,"group_logz":{"24":-4.268377304077148,"25":-2.351612091064453,"26":-0.20181846618652344}},"graph":{"mac_fp16":{"p":[0.01504563235403744,0.10255256699917689,0.8824018006467856],"p_bits":["3f8ed03e97440e3f","3fba40e29161df4e","3fec3ca2b376b3dd"],"hidden_sha256":"cec43a9a4d0596efc60b8d5326319777ec81e59f9bbdfb6b4bbdcd4bd688665f"},"iphone_int8mlp":{"p":[0.01526817918458112,0.1042524639957458,0.880479356819673],"p_bits":["3f8f44ec522350bf","3fbab04a1b639f0d","3fec2ce30b4afedb"],"hidden_sha256":"1d19adde4ab4ce6e291a410fb14520c8b11997aa92e4d8a365349511bc78837c"}}},{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"yes","row_len":279,"slot":278,"n_image_tokens":234,"row_ids_sha256":"cdbcaf4f78ae6e7da7d68fdbac277581adfc8d862d04aa43b94aae5aebf48251","oracle":{"probs":[0.9766095273090994,0.023390472690900686],"argmax":"yes","top2_margin":0.9532190546181988,"near_tie":false,"group_logz":{"11683":-0.032794952392578125,"12447":-5.79852294921875,"2243":-3.7645530700683594,"4547":-9.242433547973633,"19598":-11.132797241210938}},"graph":{"mac_fp16":{"p":[0.9766446063158594,0.02335539368414073],"p_bits":["3fef40ac307e200a","3f97ea79f03bfedb"],"hidden_sha256":"fdbabcf7a0db8dd2ce34c80cc1fdc31d983d22a630cab82087c639712331b9a8"},"iphone_int8mlp":{"p":[0.9766403845165291,0.023359615483470795],"p_bits":["3fef40a355ee7185","3f97eb954231cf49"],"hidden_sha256":"b212945c374bbff6a51fbf511c2f4fc3c7b3835a77f75dac9f4a3808cdbb2ac8"}}},{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"blue","row_len":298,"slot":297,"n_image_tokens":234,"row_ids_sha256":"0e43d7af28c4797645ed2b5c8e58c313cb62c3d96f89fc3ff5bb1de985f38e96","oracle":{"probs":[0.003477417407012462,0.0026565785198467817,0.003931129105823456,0.00311691089815293,0.9842516730984967,0.002566290970667679],"argmax":"blue","top2_margin":0.9803205439926732,"near_tie":false,"group_logz":{"41":-5.671131134033203,"334":-15.009963989257812,"42":-5.94038200378418,"378":-15.030813217163086,"43":-5.548494338989258,"340":-14.433282852172852,"44":-5.78057861328125,"388":-14.779940605163574,"45":-0.025539398193359375,"407":-8.509668350219727,"46":-5.974959373474121,"410":-14.854576110839844}},"graph":{"mac_fp16":{"p":[0.003484024606306532,0.0026607351669577024,0.00393795680989774,0.0031213367008566975,0.9842250262700336,0.0025709204459478346],"p_bits":["3f6c8a8777c2ef79","3f65cbf750d01491","3f70213f3b629892","3f6991eae1997827","3fef7ec57b778150","3f650f9c678d03b7"],"hidden_sha256":"77e4482995ebbb03cb4aa926bce0cefc9669302639351c38aa456229eb48f982"},"iphone_int8mlp":{"p":[0.0034603140194814893,0.0026478821466265376,0.003926319493694712,0.003095633270550384,0.9843165906877682,0.0025532603818785794],"p_bits":["3f6c58cdf185a37a","3f65b102e7264887","3f70150b5d1ae505","3f695c0375099cd7","3fef7f8581bd40fc","3f64ea933ad3b071"],"hidden_sha256":"77f8c26af0e679fcde997bed6c5a52309d5c68d04dc6ee510836efd3f81fa172"}}}]},{"id":"img05_tall_480x640","publication":"publish_text_and_generator","request":{"state":null,"questions":{"count":{"type":"choice","instructions":"How many shapes are in the picture?","criteria":{"one":"One","two":"Two","three":"Three","four_or_more":"Four or more"}}}},"images":["images/img05_tall_480x640.png"],"gold":{"count":"two"},"note":"drawn here (one crop 36x26); gold from the drawing (meta.json drawn)","picture":{"size":[480,640],"sha256":"eff01475a8c34884abdf0d8c9b94e94c96d08b9f6416c9a13a5f4a3b84e9294e","rgb_sha256":"f02094c57c7d4ea612f118ed119d97d2caf4096663f44de4f8c4855ef033af5e","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img05","seed":5},"drawn":{"shapes":[{"kind":"square","color":"yellow","area":32400},{"kind":"square","color":"yellow","area":12100}],"n_shapes":2,"has_circle":false,"largest_color":"yellow","largest_area":32400,"next_other_color_area":null,"crowded_level":0,"kinds":["square"],"colors":["yellow"]}},"questions":[{"name":"count","type":"choice","keys":["one","two","three","four_or_more"],"codes":["A","B","C","D"],"gold":"two","row_len":276,"slot":275,"n_image_tokens":234,"row_ids_sha256":"06e3a9b74544b602eed14df5c4fb0b917819e8123c769082142657cd6ca55e85","oracle":{"probs":[0.004247893622025314,0.986771452375253,0.004860429837343958,0.0041202241653778325],"argmax":"two","top2_margin":0.9819110225379091,"near_tie":false,"group_logz":{"41":-5.467103958129883,"334":-14.41870403289795,"42":-0.0190887451171875,"378":-8.543045043945312,"43":-5.332400321960449,"340":-13.890076637268066,"44":-5.49761962890625,"388":-14.142672538757324}},"graph":{"mac_fp16":{"p":[0.0042534035178191775,0.9867485762739837,0.004865330014556531,0.004132690193640515],"p_bits":["3f716c04501772ce","3fef9371c00f1511","3f73edab14bf8f48","3f70ed70939e7520"],"hidden_sha256":"588f1b00dc15db0cb79c6af0223963835d02f704214897b10e8223d2a370d81d"},"iphone_int8mlp":{"p":[0.004248871255857676,0.9868163460297953,0.004835778733028172,0.00409900398131889],"p_bits":["3f716743b16bd1d7","3fef93ffdfab65e9","3f73ceae78250fd8","3f70ca1e00bc29ef"],"hidden_sha256":"04938fdbe7d0b300b5d953d13dc474b0159e072e7d9f5e3157df063850089b46"}}}]},{"id":"img06_grid_1024x768","publication":"publish_text_and_generator","request":{"state":"Shapes placed on a grid.","questions":{"count":{"type":"choice","instructions":"How many shapes are in the picture?","criteria":{"one":"One","two":"Two","three":"Three","four_or_more":"Four or more"}},"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}}}},"images":["images/img06_grid_1024x768.png"],"gold":{"count":"four_or_more","largest":"green"},"note":"drawn here (tiles 2x3 + thumbnail 26x36, lattice background); gold from the drawing (meta.json drawn)","picture":{"size":[1024,768],"sha256":"c09748abc5f4789ab50d8c954373447fff6462e098cc211b3dccfc2702a20483","rgb_sha256":"f4fb2c0bb85ebc186a914de9574f4285d6ea19e2065e95ef7b162bc6a0dcc804","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img06","seed":6},"drawn":{"shapes":[{"kind":"circle","color":"green","area":38013},{"kind":"circle","color":"orange","area":6362},{"kind":"square","color":"blue","area":10000},{"kind":"square","color":"red","area":10000},{"kind":"triangle","color":"purple","area":7150},{"kind":"triangle","color":"yellow","area":5500}],"n_shapes":6,"has_circle":true,"largest_color":"green","largest_area":38013,"next_other_color_area":10000,"crowded_level":1,"kinds":["circle","square","triangle"],"colors":["blue","green","orange","purple","red","yellow"]}},"questions":[{"name":"count","type":"choice","keys":["one","two","three","four_or_more"],"codes":["A","B","C","D"],"gold":"four_or_more","row_len":1828,"slot":1827,"n_image_tokens":1770,"row_ids_sha256":"5c15c2eb225711d94963337a0fe917ce4ef56ec7e5725aabee91b4718c1968cc","oracle":{"probs":[0.005160466573883536,0.008349832844306005,0.010264555537775041,0.9762251450440353],"argmax":"four_or_more","top2_margin":0.9659605895062603,"near_tie":false,"group_logz":{"41":-5.280141830444336,"334":-13.580739974975586,"42":-4.798927307128906,"378":-12.830342292785645,"43":-4.592472076416016,"340":-12.775665283203125,"44":-0.0374755859375,"388":-7.978921890258789}},"graph":{"mac_fp16":{"p":[0.005169633299737985,0.008366050682529264,0.010292186783491936,0.9761721292342408],"p_bits":["3f752cc0df466619","3f81223850a008bd","3f851411ed1d88ea","3fef3ccd554a7ced"],"hidden_sha256":"1a2c5a569b9e0c94515501fb446e4848ef7e73f05baee94f6da7ed8ab427f62f"},"iphone_int8mlp":{"p":[0.005092778744430971,0.008221616527452435,0.010136891938681152,0.9765487127894356],"p_bits":["3f74dc2a627313b4","3f80d67eb0d864f3","3f84c2a69ae0b5b6","3fef3fe3160c356f"],"hidden_sha256":"4b0a23c12c6cae394a600752f47b986b451cd2af0652c8bc1ecda16630434fc7"}}},{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"green","row_len":1831,"slot":1830,"n_image_tokens":1770,"row_ids_sha256":"16711537feed397dfb7d4a3f1b27bcd8af8003a7d5e0cfa3870f80066d3c4c2b","oracle":{"probs":[0.0032296869230975544,0.002979577777734019,0.00244482915888146,0.9872517832414944,0.0027082962841533404,0.001385826614639096],"argmax":"green","top2_margin":0.9840220963183969,"near_tie":false,"group_logz":{"41":-5.74156379699707,"334":-14.279082298278809,"42":-5.82216739654541,"378":-14.326109886169434,"43":-6.0199737548828125,"340":-14.650787353515625,"44":-0.019023895263671875,"388":-8.56236743927002,"45":-5.917629241943359,"407":-14.36526870727539,"46":-6.587652206420898,"410":-15.023429870605469}},"graph":{"mac_fp16":{"p":[0.003232154628328715,0.0029801721557537156,0.002444897092840588,0.9872387242128412,0.0027151142069374618,0.0013889377032982177],"p_bits":["3f6a7a51cd93c20d","3f6869dfbe40d0b9","3f64075221c9682a","3fef9775aa3ad867","3f663e01d7288fc9","3f56c1a080c21abe"],"hidden_sha256":"1c3cb58a1469609c03da4e8109739f842d2e41fcdad6849073fae4d66c026837"},"iphone_int8mlp":{"p":[0.0032234884303883886,0.0029519906448188145,0.0024424073434255495,0.9872942682606195,0.0027071575841338576,0.0013806877366139235],"p_bits":["3f6a68252c66d481","3f682ec5e8e20e68","3f640219753b9489","3fef97ea2636a4c1","3f662d52293f44e8","3f569f062b2f05eb"],"hidden_sha256":"26dbbb8207db94304648e8dbef5b8e918098ce9955fca1e7ba6b7df1d2613604"}}}]},{"id":"img07_portrait_768x1024","publication":"publish_text_and_generator","request":{"state":null,"questions":{"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}},"circle":{"type":"noul","instructions":"Is there a circle in the picture?"},"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]}}},"images":["images/img07_portrait_768x1024.png"],"gold":{"largest":"yellow","circle":"true","crowded":"0"},"note":"drawn here (tiles 3x2 + thumbnail 36x26); gold from the drawing (meta.json drawn)","picture":{"size":[768,1024],"sha256":"1b3a3898c49c7ca95e1355abfdbf046bb59ed3c77785930a2631ceaac82b5f4f","rgb_sha256":"6b09195452f616c4e376133a683b0e70cbe5df3b401b83811ab0db4015ea4966","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img07","seed":7},"drawn":{"shapes":[{"kind":"circle","color":"yellow","area":196350},{"kind":"rectangle","color":"blue","area":17000}],"n_shapes":2,"has_circle":true,"largest_color":"yellow","largest_area":196350,"next_other_color_area":17000,"crowded_level":0,"kinds":["circle","rectangle"],"colors":["blue","yellow"]}},"questions":[{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"yellow","row_len":1822,"slot":1821,"n_image_tokens":1770,"row_ids_sha256":"af34e804b98058f338bf16aebc5b210a8a64f8ae3787e0f66b4869d04d72f5ca","oracle":{"probs":[0.003182285936973088,0.0032756587916714735,0.985631225640902,0.003005933541108987,0.003128261809821408,0.001776634279523084],"argmax":"yellow","top2_margin":0.9823555668492305,"near_tie":false,"group_logz":{"41":-5.756852149963379,"334":-15.057003021240234,"42":-5.727932929992676,"378":-14.636351585388184,"43":-0.021169662475585938,"340":-9.086481094360352,"44":-5.813863754272461,"388":-15.043502807617188,"45":-5.773974418640137,"407":-14.684759140014648,"46":-6.339731216430664,"410":-15.419922828674316}},"graph":{"mac_fp16":{"p":[0.0031826161395078273,0.0032785055990471484,0.9856219463166752,0.00300814648317325,0.0031296610302367755,0.0017791244313599135],"p_bits":["3f6a126e07845d8e","3f6adb864a7c31a5","3fef8a370934ccb3","3f68a48a5889add5","3f69a35ff8b7c49c","3f5d26304fe29902"],"hidden_sha256":"56c265a41501400b212a5d03827fdd57d77a550c05404f5a2cf781055e2a1960"},"iphone_int8mlp":{"p":[0.00315661438441143,0.0032248244209072308,0.985756810993092,0.0029869151600415917,0.003103975187377304,0.0017708598541703996],"p_bits":["3f69dbe671821754","3f6a6af26d8a5a29","3fef8b51de20c963","3f687803ddb4ad32","3f696d81fd5b2f62","3f5d03864a349cbb"],"hidden_sha256":"8254fe4a528c9dd4b5c6b10f2e8dd37f334f3ca049151ccf486c7e9766ead0de"}}},{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"yes","row_len":1803,"slot":1802,"n_image_tokens":1770,"row_ids_sha256":"eec8f97f91af51023c9cfc228dd2e8a81377688f0a1f708d47a1844cdd9fcea4","oracle":{"probs":[0.9902871214890723,0.009712878510927657],"argmax":"yes","top2_margin":0.9805742429781447,"near_tie":false,"group_logz":{"11683":-0.018253326416015625,"12447":-5.207640647888184,"2243":-4.642795562744141,"4547":-9.219873428344727,"19598":-11.319833755493164}},"graph":{"mac_fp16":{"p":[0.9903034348425473,0.00969656515745265],"p_bits":["3fefb090d4387c95","3f83dbcaf1e0dabf"],"hidden_sha256":"85654f37043a225d683c5e17c3b6c0aa274e23e4254e275374540b63813ff3bf"},"iphone_int8mlp":{"p":[0.990321862555446,0.00967813744455401],"p_bits":["3fefb0b779860ff4","3f83d2219e7c02f1"],"hidden_sha256":"162d31762262778332fd8e2549c4ee6be340cc978b8e123da777e88dc58112b5"}}},{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"0","row_len":1820,"slot":1819,"n_image_tokens":1770,"row_ids_sha256":"03b3e0184865df90c45d5b2137c4540fe6f9e83af96192af53ae878b257a571e","oracle":{"probs":[0.1580374755576089,0.8228552564252372,0.01910726801715408],"argmax":"1","top2_margin":0.6648177808676283,"near_tie":false,"group_logz":{"24":-1.8562583923339844,"25":-0.20631027221679688,"26":-3.969021797180176}},"graph":{"mac_fp16":{"p":[0.15761062652579505,0.8233424244015182,0.01904694907268674],"p_bits":["3fc42c95c3371608","3fea58d23646d80c","3f93810b1d6c4e34"],"hidden_sha256":"379433ea7410e262aea7a7fe0ce86eda92c482c1f790516490a57aa7f2bc64d9"},"iphone_int8mlp":{"p":[0.15827566621395517,0.8226634921622417,0.019060841623803244],"p_bits":["3fc4426085121e4c","3fea5342634e663d","3f9384af6da24615"],"hidden_sha256":"41f5340b695893ce99b81170337d777e40e191770966b2a6685fbdbb68802fc4"}}}]},{"id":"img08_square_1024x1024","publication":"publish_text_and_generator","request":{"state":null,"questions":{"circle":{"type":"noul","instructions":"Is there a circle in the picture?"},"count":{"type":"choice","instructions":"How many shapes are in the picture?","criteria":{"one":"One","two":"Two","three":"Three","four_or_more":"Four or more"}}}},"images":["images/img08_square_1024x1024.png"],"gold":{"circle":"false","count":"one"},"note":"drawn here (tiles 2x2 + thumbnail 32x32); gold from the drawing (meta.json drawn)","picture":{"size":[1024,1024],"sha256":"f564ac216d02f2dca26218686806db4f08175211bfdadcf24bc9da2c09351e79","rgb_sha256":"928e42f3cc4abbb0b5bc0d39e0c88853073e1bfe08101a3385cc65b85688e357","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img08","seed":8},"drawn":{"shapes":[{"kind":"triangle","color":"red","area":25200}],"n_shapes":1,"has_circle":false,"largest_color":"red","largest_area":25200,"next_other_color_area":null,"crowded_level":0,"kinds":["triangle"],"colors":["red"]}},"questions":[{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"no","row_len":1311,"slot":1310,"n_image_tokens":1280,"row_ids_sha256":"aaea4256d382b0c6518744d196fca3d01d37ff01a74b662a1077022f1c4a48e1","oracle":{"probs":[0.013552634781805203,0.9864473652181948],"argmax":"no","top2_margin":0.9728947304363895,"near_tie":false,"group_logz":{"11683":-4.310791015625,"12447":-8.842643737792969,"2243":-0.02326202392578125,"4547":-4.935722351074219,"19598":-8.66185474395752}},"graph":{"mac_fp16":{"p":[0.01358322661935177,0.9864167733806484],"p_bits":["3f8bd185d0d58cd1","3fef90b9e8bca9ce"],"hidden_sha256":"714026941fc6e00907c729aa1193eecf045072a51c52c1328a58fbcb4ec862b6"},"iphone_int8mlp":{"p":[0.01361618583303794,0.9863838141669622],"p_bits":["3f8be2cd86cb128c","3fef9074c9e4d3b7"],"hidden_sha256":"078033e75b02fd209ba466dc16eed3a4743cfc1889e678feec590f06ab93dd7a"}}},{"name":"count","type":"choice","keys":["one","two","three","four_or_more"],"codes":["A","B","C","D"],"gold":"one","row_len":1327,"slot":1326,"n_image_tokens":1280,"row_ids_sha256":"7337ce353ea2f30ec1b906cf551695ef65a47764cf9a83f4d26ca972ce4814d8","oracle":{"probs":[0.9533452836300587,0.021589021995286804,0.012794659083944969,0.012271035290709475],"argmax":"one","top2_margin":0.9317562616347719,"near_tie":false,"group_logz":{"41":-0.07125663757324219,"334":-8.223333358764648,"42":-3.859048843383789,"378":-11.815285682678223,"43":-4.382205963134766,"340":-12.362977981567383,"44":-4.423992156982422,"388":-12.310473442077637}},"graph":{"mac_fp16":{"p":[0.9534422494790313,0.021534231154914697,0.012777166763482084,0.01224635260257178],"p_bits":["3fee82995204633a","3f960d11ca382dfd","3f8a2aea4b13da4d","3f89149d9f62fafc"],"hidden_sha256":"158605cb43e8681a94f6ed4bb64ba9fbb046a85a431eddbda631c5a029565f24"},"iphone_int8mlp":{"p":[0.9534041779530579,0.02164656479261322,0.012842179283157309,0.012107077971171562],"p_bits":["3fee82497a85b363","3f962a845f6b8a0d","3f8a4d00203e862e","3f88cb987f7d8ce5"],"hidden_sha256":"7ccdabb4fde0e755bdadaf1ad735663a406a62472fc2c7e25be6df1c84605bb8"}}}]},{"id":"img09_wide_1600x1200","publication":"publish_text_and_generator","request":{"state":"A drawing on a large canvas.","questions":{"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}},"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]}}},"images":["images/img09_wide_1600x1200.png"],"gold":{"largest":"purple","crowded":"1"},"note":"drawn here (cap_pixels to 1182x886, then tiles 2x3 + thumbnail 26x36); gold from the drawing (meta.json drawn)","picture":{"size":[1600,1200],"sha256":"016b571dd6d5deae0146af62c1e2bdcd518b723522f8229c7983323e31fae7c9","rgb_sha256":"b12583ec66a72274fdb0ae3f2ff4cc74840f0cb7c64cea24431b7e7d027dbd1a","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img09","seed":9},"drawn":{"shapes":[{"kind":"circle","color":"orange","area":45239},{"kind":"circle","color":"orange","area":38013},{"kind":"square","color":"purple","area":160000},{"kind":"triangle","color":"green","area":28600},{"kind":"triangle","color":"green","area":24000}],"n_shapes":5,"has_circle":true,"largest_color":"purple","largest_area":160000,"next_other_color_area":45239,"crowded_level":1,"kinds":["circle","square","triangle"],"colors":["green","orange","purple"]}},"questions":[{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"purple","row_len":1831,"slot":1830,"n_image_tokens":1770,"row_ids_sha256":"eb62f61aa72f9206153d7f59146fa32ae18f02a89508730a29e3e397c78278d7","oracle":{"probs":[0.004199383645067041,0.005821693762424988,0.0035959367086589233,0.013209594133010606,0.00449284243960815,0.9686805493112304],"argmax":"purple","top2_margin":0.9554709551782198,"near_tie":false,"group_logz":{"41":-5.483979225158691,"334":-14.607152938842773,"42":-5.157325744628906,"378":-14.055221557617188,"43":-5.63911247253418,"340":-14.699807167053223,"44":-4.337973594665527,"388":-13.13774585723877,"45":-5.416431427001953,"407":-14.146599769592285,"46":-0.04298210144042969,"410":-8.767805099487305}},"graph":{"mac_fp16":{"p":[0.004209691886361318,0.0058309019726622245,0.0035972709474502654,0.013244248433315516,0.004493390026274905,0.9686244967339358],"p_bits":["3f713e2e8fa6896f","3f77e224d475d37c","3f6d7806225f6f74","3f8b1fccef0cad04","3f7267a9336059f7","3feefef8ccf2746f"],"hidden_sha256":"3f738e9a7cd83f75448ebe798090ca55d0954a4241d675e1c61dae9c1d2b2fe6"},"iphone_int8mlp":{"p":[0.004195023777063119,0.005759450844200215,0.003596496892390475,0.013005574112581747,0.00449098888935643,0.968952465484408],"p_bits":["3f712ecd1edad7d5","3f779738d04cd6c7","3f6d7666910e31c8","3f8aa2aa9bd57d43","3f726524a680f6a8","3fef01a899d44a8f"],"hidden_sha256":"34e557d14674dd36868e7360b8c0c2f65fac1dc82bfb36031c4953694b7b03e4"}}},{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"1","row_len":1829,"slot":1828,"n_image_tokens":1770,"row_ids_sha256":"9e2c345e26b7edeff605e8de90590bac462a64d244304158973d95941bcb0f1c","oracle":{"probs":[0.021594703881890498,0.8215222893573665,0.15688300676074304],"argmax":"1","top2_margin":0.6646392825966234,"near_tie":false,"group_logz":{"24":-3.885716438293457,"25":-0.24700546264648438,"26":-1.9026641845703125}},"graph":{"mac_fp16":{"p":[0.02154821207025002,0.8216791754445367,0.15677261248521326],"p_bits":["3f9610bc0884179b","3fea4b32204ad540","3fc4111ffdc4280d"],"hidden_sha256":"a5cb3e9ef4b48e87e707023e5c0a3d278ca33245ad8aaeb1231891d41285b18b"},"iphone_int8mlp":{"p":[0.021612387931733288,0.8239398359396014,0.1544477761286652],"p_bits":["3f96218ecd6bc929","3fea5db713276e21","3fc3c4f1d9b4ce53"],"hidden_sha256":"8fbd43c29d2c620cdb3028d8195f06d155e39459400de77b93a5b3c0af6929d6"}}}]},{"id":"img10_big_2048x1536","publication":"publish_text_and_generator","request":{"state":null,"questions":{"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]},"circle":{"type":"noul","instructions":"Is there a circle in the picture?"}}},"images":["images/img10_big_2048x1536.png"],"gold":{"crowded":"2","circle":"true"},"note":"drawn here (cap_pixels to 1182x886, then tiles 2x3 + thumbnail 26x36); gold from the drawing (meta.json drawn)","picture":{"size":[2048,1536],"sha256":"a7624d95a02ccf2d483c7c2a92a93627675d5f16c65f405620a7ae78789264dd","rgb_sha256":"c0d4bba074aca90d7bdf546e8bde590c19c3066c2fd4e17b9db48994ec5f008b","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img10","seed":10},"drawn":{"shapes":[{"kind":"circle","color":"red","area":282743},{"kind":"square","color":"purple","area":6561},{"kind":"square","color":"green","area":3136},{"kind":"circle","color":"purple","area":11310},{"kind":"circle","color":"green","area":9503},{"kind":"triangle","color":"green","area":2442},{"kind":"square","color":"yellow","area":8464},{"kind":"square","color":"yellow","area":7225},{"kind":"square","color":"purple","area":5776},{"kind":"circle","color":"yellow","area":8495},{"kind":"triangle","color":"green","area":2765},{"kind":"square","color":"purple","area":8100},{"kind":"square","color":"purple","area":3481},{"kind":"square","color":"yellow","area":3969},{"kind":"square","color":"purple","area":3249},{"kind":"circle","color":"blue","area":14957},{"kind":"circle","color":"orange","area":10568},{"kind":"square","color":"blue","area":5776},{"kind":"triangle","color":"purple","area":5390},{"kind":"circle","color":"yellow","area":6362},{"kind":"square","color":"blue","area":5929},{"kind":"triangle","color":"orange","area":3188},{"kind":"circle","color":"orange","area":7854},{"kind":"square","color":"yellow","area":6561},{"kind":"square","color":"orange","area":6084},{"kind":"square","color":"blue","area":3969},{"kind":"triangle","color":"green","area":2765},{"kind":"square","color":"orange","area":7569},{"kind":"circle","color":"blue","area":15394},{"kind":"square","color":"orange","area":6400},{"kind":"square","color":"blue","area":3844}],"n_shapes":31,"has_circle":true,"largest_color":"red","largest_area":282743,"next_other_color_area":15394,"crowded_level":2,"kinds":["circle","square","triangle"],"colors":["blue","green","orange","purple","red","yellow"]}},"questions":[{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"2","row_len":1820,"slot":1819,"n_image_tokens":1770,"row_ids_sha256":"bdc5e910fcd9115bb0f7ea1fa606bb817c7f930d46861c9d13108985004a9dcd","oracle":{"probs":[0.011868847900688491,0.058766295267709226,0.9293648568316023],"argmax":"2","top2_margin":0.870598561563893,"near_tie":false,"group_logz":{"24":-4.508676528930664,"25":-2.909025192260742,"26":-0.14809226989746094}},"graph":{"mac_fp16":{"p":[0.011867308145199226,0.05885032776858092,0.9292823640862199],"p_bits":["3f884de32300610b","3fae21a1523ff793","3fedbcae5e4fff03"],"hidden_sha256":"0672b5b2bde11856b0b442f5902a75a4a48975c0d089c5262b0d5969d2acb161"},"iphone_int8mlp":{"p":[0.012020457222772465,0.05986307224678415,0.9281164705304434],"p_bits":["3f889e2e753cfa88","3faea65f631377c6","3fedb3214ff9d49a"],"hidden_sha256":"5bf2d7c5eef2380542bb41d87183100f71fd5c6681154ea009d015d704687d29"}}},{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"yes","row_len":1803,"slot":1802,"n_image_tokens":1770,"row_ids_sha256":"94ba930c387c63712d0bb9059265c594ecc59607008d9b41dc70e8273ca23ba6","oracle":{"probs":[0.9197362152728423,0.0802637847271577],"argmax":"yes","top2_margin":0.8394724305456847,"near_tie":false,"group_logz":{"11683":-0.10693931579589844,"12447":-4.5660810470581055,"2243":-2.5457077026367188,"4547":-6.762487411499023,"19598":-9.763647079467773}},"graph":{"mac_fp16":{"p":[0.9197730032638497,0.08022699673615016],"p_bits":["3fed6ec7cb1861c7","3fb489c1a73cf1be"],"hidden_sha256":"c9b3ecc6a87e06520f31a9846c3bc5c5d684c830df223c6e80f4c75fcda89fae"},"iphone_int8mlp":{"p":[0.9201390679509838,0.0798609320490161],"p_bits":["3fed71c77c93e27c","3fb471c41b60ec17"],"hidden_sha256":"5fb69c1daec2941ff167f756f6ac178cfa78edbf8c5956a2d86d5c36a67dad98"}}}]},{"id":"img11_banner_1200x400","publication":"publish_text_and_generator","request":{"state":"A banner.","questions":{"count":{"type":"choice","instructions":"How many shapes are in the picture?","criteria":{"one":"One","two":"Two","three":"Three","four_or_more":"Four or more"}}}},"images":["images/img11_banner_1200x400.png"],"gold":{"count":"three"},"note":"drawn here (one crop 18x54 (wide; 384x1216 after rounding is under the tile threshold)); gold from the drawing (meta.json drawn)","picture":{"size":[1200,400],"sha256":"c1a366a1e934e4668f00c6f6d754277d4f38522b25af90ce62d91a7775859b82","rgb_sha256":"1b7e4c8b04f6bb83c8e007d89741e5da19a68aba841857b9e3fe8499a22cd557","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img11","seed":11},"drawn":{"shapes":[{"kind":"circle","color":"blue","area":45239},{"kind":"circle","color":"blue","area":45239},{"kind":"circle","color":"blue","area":45239}],"n_shapes":3,"has_circle":true,"largest_color":"blue","largest_area":45239,"next_other_color_area":null,"crowded_level":1,"kinds":["circle"],"colors":["blue"]}},"questions":[{"name":"count","type":"choice","keys":["one","two","three","four_or_more"],"codes":["A","B","C","D"],"gold":"three","row_len":290,"slot":289,"n_image_tokens":243,"row_ids_sha256":"ee9cac00d35305d3426127ac7a24f80f0a0b5d61307593995d0c8640a860dd86","oracle":{"probs":[0.004284066866144311,0.003794973408164274,0.9879601791782866,0.003960780547404802],"argmax":"three","top2_margin":0.9836761123121422,"near_tie":false,"group_logz":{"41":-5.45919132232666,"334":-14.600542068481445,"42":-5.580416679382324,"378":-14.306473731994629,"43":-0.018451690673828125,"340":-8.944646835327148,"44":-5.537652969360352,"388":-14.474013328552246}},"graph":{"mac_fp16":{"p":[0.004274706247979268,0.0037873687268970705,0.987987053077796,0.00395087194732762],"p_bits":["3f71825ab88fadc3","3f6f06b01a3cc0a2","3fef9d97063ae5c9","3f702eca1cdf0d37"],"hidden_sha256":"767119b3c677b054fb849a5f04065263673dbc44f541a6bb0857567d8def282b"},"iphone_int8mlp":{"p":[0.004237352883011047,0.003756810734088383,0.9880585180286492,0.003947318354251403],"p_bits":["3f715b2fc0ddc5d9","3f6ec69a67af7bd8","3fef9e2ce5aef3ea","3f702b1033d08757"],"hidden_sha256":"841f6f1f856171a371e522b8a09c41f0cf71c51abe6198abd29f8948336fdc28"}}}]},{"id":"img12_small_300x300","publication":"publish_text_and_generator","request":{"state":null,"questions":{"largest":{"type":"choice","instructions":"What color is the largest shape?","criteria":{"red":"Red","orange":"Orange","yellow":"Yellow","green":"Green","blue":"Blue","purple":"Purple"}},"circle":{"type":"noul","instructions":"Is there a circle in the picture?"},"crowded":{"type":"score","instructions":"How crowded is the picture?","criteria":["Nearly empty","A few shapes","Packed with shapes"]}}},"images":["images/img12_small_300x300.png"],"gold":{"largest":"green","circle":"false","crowded":"0"},"note":"drawn here (one crop 18x18 (300 -> 288)); gold from the drawing (meta.json drawn)","picture":{"size":[300,300],"sha256":"cae01c3e8d3ba1702a853621ec8f135b76bde68c83050f8e8ed659fdef7cbb1e","rgb_sha256":"ca971b3b9f9452e5387169c93ff1c45dd27cc368bb1dd9cd4b7135386fd0328f","generator":{"script":"conversion/d1/make_fixture_images.py","function":"img12","seed":12},"drawn":{"shapes":[{"kind":"square","color":"green","area":14400}],"n_shapes":1,"has_circle":false,"largest_color":"green","largest_area":14400,"next_other_color_area":null,"crowded_level":0,"kinds":["square"],"colors":["green"]}},"questions":[{"name":"largest","type":"choice","keys":["red","orange","yellow","green","blue","purple"],"codes":["A","B","C","D","E","F"],"gold":"green","row_len":126,"slot":125,"n_image_tokens":81,"row_ids_sha256":"7b5c590f2e6bf8258c0014e99af703f25da0afeef59329560b044582fca4b671","oracle":{"probs":[0.004135820024461702,0.003623825382808803,0.004116997755498961,0.9818531429773566,0.0038742210609728107,0.0023959927989010764],"argmax":"green","top2_margin":0.9777173229528949,"near_tie":false,"group_logz":{"41":-5.498040199279785,"334":-14.034358978271484,"42":-5.630195617675781,"378":-14.072786331176758,"43":-5.502601623535156,"340":-14.040789604187012,"44":-0.028284072875976562,"388":-8.48050594329834,"45":-5.563381195068359,"407":-13.89963436126709,"46":-6.043928146362305,"410":-14.54266357421875}},"graph":{"mac_fp16":{"p":[0.0041302356214616715,0.00362110086204924,0.004112026125488952,0.9818671592686815,0.003872029700761175,0.002397448421557483],"p_bits":["3f70eaddaeb40722","3f6da9ffb8e587e1","3f70d7c59bab19ca","3fef6b74ad4269c3","3f6fb83c1de2970b","3f63a3d0520fdc87"],"hidden_sha256":"c2fe851db76f524234737457c167af96491937a1497dfe264cf00882b4cbbaa3"},"iphone_int8mlp":{"p":[0.0041306659887015785,0.0035967525845887523,0.004097273043686424,0.9819345157869487,0.0038548223043557255,0.002385970291718974],"p_bits":["3f70eb5135509469","3f6d76efd71fa38c","3f70c84d5b9b4d5b","3fef6c01ef03cb15","3f6f9425f754c7d8","3f638bbe0be8bd4a"],"hidden_sha256":"b75089c5b3e0a8a050bc659c6159a3cd2479b9c38020ed51c82267094c7658e3"}}},{"name":"circle","type":"noul","keys":["yes","no"],"codes":null,"gold":"no","row_len":107,"slot":106,"n_image_tokens":81,"row_ids_sha256":"d8b946423fb5b0272710d9dfce728dfbe481368a88b9338f4ce12300374c99ea","oracle":{"probs":[0.0127470484248935,0.9872529515751065],"argmax":"no","top2_margin":0.974505903150213,"near_tie":false,"group_logz":{"11683":-4.376117706298828,"12447":-8.530147552490234,"2243":-0.026491165161132812,"4547":-4.501176834106445,"19598":-8.50324535369873}},"graph":{"mac_fp16":{"p":[0.012737989166160072,0.98726201083384],"p_bits":["3f8a165ff7156bf5","3fef97a68023aa51"],"hidden_sha256":"3ef08ffc20e6e76a92e40e7aa3331f04422ee09979025e41ad380e8d2f64e510"},"iphone_int8mlp":{"p":[0.012796820569495052,0.9872031794305051],"p_bits":["3f8a35382eb5c882","3fef972b1f4528df"],"hidden_sha256":"0df1adaeaa61548889f5cf8fc1874e5ca9859860bfd69c912c7fec36711583e0"}}},{"name":"crowded","type":"score","keys":["0","1","2"],"codes":null,"gold":"0","row_len":124,"slot":123,"n_image_tokens":81,"row_ids_sha256":"9e4c1ad1e9568d0eb1a00236a582d08c0bbe637ffc6e72ed1c9f9d77a944f5fa","oracle":{"probs":[0.9544319592134891,0.03991382290161922,0.005654217884891648],"argmax":"0","top2_margin":0.9145181363118698,"near_tie":false,"group_logz":{"24":-0.05789375305175781,"25":-3.2322874069213867,"26":-5.18660831451416}},"graph":{"mac_fp16":{"p":[0.9544599042030659,0.039883873717676946,0.005656222079257192],"p_bits":["3fee8aef7f3ca79e","3fa46ba8ba82abba","3f772afa8d96d376"],"hidden_sha256":"3ae617ea9b904925a9efcfcd12692bc27fc6edc8ff53d0eabca76e65f2f596df"},"iphone_int8mlp":{"p":[0.954868944877524,0.03950137265828657,0.005629682464189243],"p_bits":["3fee8e495146e138","3fa439861f6dcc50","3f770f26612100e4"],"hidden_sha256":"b69da829b50d56c066c29e5f4bfe093a9519fb0d30d5af51aac14900eb32f654"}}}]},{"id":"card_cats","publication":"not_published","note":"the provider card's photo example (a COCO URL): never fetched, not in the gates"}],"red_arms":{"what":"round 4's five red arms (fixtures/red_arms_r4.json): each must move the oracle by the gate's bar and move the graph by the same amount","arms":[{"id":"red_word_tv4_001","kind":"word","base":"tv4_001","question":"answer","source":"round 4: red_word_tv4_000's change on the next tv4 record (replacement candidate)","change":{"field":"questions.answer.instructions","what":"one word of the question's instructions replaced ('correctly' -> 'incorrectly')","text":"held back (the base record is a reference-only transfer-v4 row)"}},{"id":"red_not_qnli_00","kind":"not","base":"tv4x_qnli_00","question":"answers","source":null,"change":{"field":"questions.answers.instructions","what":"a grammatical 'not' inserted in the question's instructions","text":"held back (the base record is a reference-only transfer-v4 row)"}},{"id":"red_not_paws_00","kind":"not","base":"tv4x_paws_00","question":"paraphrase","source":null,"change":{"field":"questions.paraphrase.instructions","what":"a grammatical 'not' inserted in the question's instructions","text":"held back (the base record is a reference-only transfer-v4 row)"}},{"id":"state_swap_tv4_000__tv4_001","kind":"state_swap","base":null,"question":null,"source":null},{"id":"state_swap_semif_a3f18f3a63d45345942b__semif_0b43ea8e24e74d621f1b","kind":"state_swap","base":null,"question":null,"source":null}]},"summary":{"records":361,"questions":393,"by_source":{"fixture_records":361,"fixture_questions":394,"fixture_publish_text":{"records":149,"questions":157},"fixture_reference_only":{"records":200,"questions":200},"fixture_withhold_text":{"records":12,"questions":37},"heldout_reference_only":{"records":120,"questions":120},"pictures_published":12,"picture_questions_published":24,"red_arms_with_change_text":1},"near_ties":["tv4_013:?"],"heldout_records":120,"heldout_questions":120,"image_records_published":12,"graph_gate":{"fixture":{"questions":393,"questions_non_near_tie":392,"argmax_equal_non_near_tie":392,"near_tie_questions":1,"argmax_equal_near_tie":1,"max_abs_dp":0.019660274378349274,"mean_of_run_mean_abs_dp":0.0012615584218342564,"worst_run":{"id":"semif_33d6e5da58f0fee2490d","name":"answer","max_abs_dp":0.019660274378349274},"finite_all":true},"heldout":{"questions":120,"questions_non_near_tie":118,"argmax_equal_non_near_tie":118,"near_tie_questions":2,"argmax_equal_near_tie":2,"max_abs_dp":0.014006239134214393,"mean_of_run_mean_abs_dp":0.0010123085973413758,"worst_run":{"id":"tv4sh_05","name":"decision","max_abs_dp":0.014006239134214393},"finite_all":true},"pictures":{"questions":24,"argmax_equal":24,"questions_non_near_tie":24,"argmax_equal_non_near_tie":24,"near_ties":[],"max_abs_dp":0.002435230632077845,"worst":"img09_wide_1600x1200:crowded","mean_of_run_mean_abs_dp":0.0002337234971274742,"pass":true}}},"trimmed":{"from_lane_files":{"fixtures/records.json":{"bytes":597770,"sha256":"2284bb22e8cdc02a4d1c4a787e34981d2cee4044b32c3b4452abcd61136ef4dd"},"fixtures/heldout.json":{"bytes":215330,"sha256":"7b853f53b97b9f20209e942cf9733a7441a81152dc150bf89f6ec409fdc80804"},"fixtures/image_records.json":{"bytes":11284,"sha256":"50dec6e837d315da610979cb176facca9955315c9a7d81ae978029f6b3c597e0"},"fixtures/images/meta.json":{"bytes":16143,"sha256":"348275565f64b64795fb9fe162af391c03f359a9a6a103aa64676d857bcec973"},"fixtures/red_arms_r4.json":{"bytes":7418,"sha256":"a1b8533efc0cea5812e1b8bb25c1d7d8ae5841b75a57359b7907873465610443"},"fixtures/LICENSE-SemIf-MIT.txt":{"bytes":1066,"sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be"},"oracle/records_oracle.json":{"bytes":1240186,"sha256":"067a4f5658937da0e20e307deaa15a38deb55d0f12b2f9993b609461d0430af1"},"oracle/heldout/records_oracle.json":{"bytes":294381,"sha256":"db8b5bf0efd303e4614043c94ec8b5e8134a0673d040fc8d51618ac74e969a45"},"oracle/records_oracle_images.json":{"bytes":223233,"sha256":"b3d6afac66c6edf4c132b72058da0b04b12cc7039b35c5628df85229275d6d37"},"results/r6a_readout_int8mlp_pf64.json":{"bytes":572833,"sha256":"16f28d47362ccdc5533fa3851c1113a6a4c261638e63d37b6bd2cd2b8f526dc5"},"results/r11_readout_int8mlp_pf64_heldout.json":{"bytes":170437,"sha256":"01a3758476898f057b5f86efa9b4424b507e8ab4e0a2039fad2771881891ccdb"},"results/r7_images_int8mlp_pf64.json":{"bytes":16552,"sha256":"c42b187a2ec743bafa682cb65cc122469fe4756b6d8f357a20732606e404f600"},"results/r11_fixture_publication.json":{"bytes":16926,"sha256":"98024034ea008436930f8d42bab472ff82c5afd8e50cd48e49a19bd13f9d1c69"}},"removed":["the transfer-v4 records' requests, rendered prompts, keys, codes and gold labels","the own_* records' requests, keys, codes, question names and gold labels","every row's token ids and rendered prompt text (row_ids_sha256 keeps the ids checkable)","the oracle's per-question timing and the hidden-row dumps"],"serialization":"compact json (no indentation)"}}