{"what":"The held-out set (30 records written after round 4, 40 runs, 188 questions; its own fp32 oracle run): the fp16 and int8mix decoders (S = 64) alone on the Mac GPU, the same gate as the fixture readouts. int8mix's layer set was chosen on the fixture; this is its test.","sources":[{"file":"results/readout_heldout_fp16_pf64.json","bytes":223141,"sha256":"49a2553cbe8ff955e47a524c1f8b8ce1f151e9998cae7ebb141a096a52c3b5e9"},{"file":"results/readout_heldout_int8mix_pf64.json","bytes":251618,"sha256":"2c0a1fd895a927c9f9ad0d0ca100d0d14d4a417c5baf1fbec15896395b945d2e"}],"trimmed":{"removed":["both transcripts: runs[].call_ms (per-call ms; the timing block keeps the medians)","both transcripts: runs[].npz / hidden_npz / npz_key (lane file names)","both transcripts: runs[].hidden.lowest_positions (each run keeps min_pos_cos, its index and the counts below 0.99 / 0.9 / 0.5)","both transcripts: runs[].questions[].probs_oracle (the oracle's probabilities are in fixtures-clef-flash.json)","both transcripts: oracle.npz_sha256 (sha256 of the lane's per-run oracle npz files)","both transcripts: processes[].npz / npz_sha256 / descriptor (the descriptor is the top-level one: descriptor_same_in_every_process)"],"serialization":"compact json (no indentation)","full_copies":"the source files above, in the lane directory $ZOO_WORK_ROOT/_clefflash/ (not in the repository)"},"fp16_pf64":{"schema":"clef-flash-decoder-readout-gate/1","gate":"decoder alone on the Mac GPU (oracle ids, oracle fp32 image_embeds -> fp16, zero-padded to 1024 rows), the fp16 hidden read through the author's fp32 head","bundle":{"bundle":"<lane>/exports/bundles/clef_flash_decode_fp16_pf64","name":"clef_flash_decode_fp16_pf64","kind":"decision-backbone","compression":null,"prefill_chunk":64,"metadata_sha256":"bbd4401e72dd055c9fb9998aecf3fcd1b43ea789698bec9040c0d89412a77ae3","main_mlirb":{"bytes":15880253283,"sha256":"0b7dfa30a0597ead745d8f3ed9ed11fa5ac639a4c578a33a7cca7ced11b9bea8"},"tokenizer_sha256":{"chat_template.jinja":"a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715","tokenizer.json":"06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523","tokenizer_config.json":"91a08f825d370d085d692e04cf117cdd7faad7bf18e996f1e6031b6dab03db72"},"aimodelc":{"path":"<lane>/exports/bundles_aotc/clef_flash_decode_fp16_pf64.h16c.aimodelc","bytes":29731951203,"tree_sha256":"196c089dca5446037f583f9957a5fa5eac5717324e7b49dfce7cc9b7fd49ec9e","files":{"main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist":"bbf348d5a22a3d8b3e86844153f727c37e891c6f4e2d9f6fc5408413c2947104","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/original_model_0.mpsgraph":"6ce9393c561b16b7b0fe36956f3ba3746597ba9d7b27fc9542ded93258bec117","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin":"f602c49d232fd7d08d9b02a276cfcbc033fe27920f1f50d5fa645ad7a1093273","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_1.mpsgraph":"e60f6614c8f00859c2a8dd86e0b8e8275956de95b7530c1c94692111cba0d556","main-h16c.mlirb":"0d85c1b244198a3102e1b37d1ad76b9ecba06424607c232697c7ab66c60a08ad","main.hash":"5a75a90e1c5d061241566adf4a940ede53aea86252b35ac78d0afd8889ba401b","metadata.json":"2c6acb63c7baf6f6072fb2d06b15d6163e95d83c56c05e302c79c90060ffe786","stats.json":"c4eac63e4b03f3d77179ecfe949163a5b64281cca93484f54a2a9ba9c813e50c"}}},"chunk":64,"max_ctx":4096,"readout":"fresh zero states; a row of T ids runs as ceil(T / 64) calls of 'main' (static S = 64): call k gets ids[64k : 64k + 64] with position_ids 0..64k+63; the last call is padded with <|endoftext|> (248044) and the hidden rows of the padded positions are discarded (causal: they cannot reach a real position). The hidden rows [1, 64, 4096] of every call, concatenated and cut to T, are the backbone's final-norm last_hidden_state [T, 4096] the head reads.","descriptor":{"function":"main","inputs":{"input_ids":[[1,64],"int32"],"position_ids":[[1,-1],"int32"],"image_embeds":[[1024,4096],"float16"],"image_rc":[[1024,2],"int32"],"rope_shift_start":[[1],"int32"],"rope_shift_amount":[[1],"int32"]},"outputs":{"hidden":[[1,64,4096],"float16"]},"states":{"keyCache":[[8,1,4,-1,256],"float16"],"valueCache":[[8,1,4,-1,256],"float16"],"convState":[[24,1,8192,3],"float16"],"recState":[[24,1,32,128,128],"float16"]}},"descriptor_same_in_every_process":true,"contract":{"inputs":{"input_ids":[[1,64],"int32"],"position_ids":[[1,-1],"int32"],"image_embeds":[[1024,4096],"float16"],"image_rc":[[1024,2],"int32"],"rope_shift_start":[[1],"int32"],"rope_shift_amount":[[1],"int32"]},"outputs":{"hidden":[[1,64,4096],"float16"]},"states":{"keyCache":[[8,1,4,-1,256],"float16"],"valueCache":[[8,1,4,-1,256],"float16"],"convState":[[24,1,8192,3],"float16"],"recState":[[24,1,32,128,128],"float16"]}},"oracle":{"path":"<lane>/oracle/heldout/records_oracle.json","sha256":"85e4dda56890dff47a163bdd74c85291f5d3be0944403dd1f59ba2f5b9faaa3f","source":"Cloudflare/clef-flash@17f0b0ad64efb65d273590632833508766b2aae6","versions":{"python":"3.12.11","torch":"2.9.0","torchvision":"0.24.0","transformers":"5.17.0","pillow":"12.3.0","numpy":"2.5.3","huggingface_hub":"1.33.0"}},"subset":"all","runs_file":null,"script":{"path":"conversion/clef_flash/readout_gate.py","sha256":"ae633622829f02e83a202d522994559c77ab25a8e08dc759ec14435c983c50f0"},"skipped":[],"bar":{"max_abs_dp":0.02,"mean_of_run_mean_abs_dp":0.002,"near_tie_top2_margin":0.02,"argmax":"every question with an oracle top-2 margin above 0.02; near-ties listed apart","max_abs_dp_applies_to":"every option of every question, near-ties included","reset":"bit-equal hidden rows in every process","mean_of_run_mean_abs_dp_definition":"mean over runs of the mean |dp| over all (question, option) entries of the run"},"environment":{"versions":{"python":"3.11.13","numpy":"2.3.5","coreai-core":"1.0.0b2","coreai-torch":"0.4.1","coreai-models":"0.1.0","torch":"2.9.0","safetensors":"0.7.0"},"platform":"macOS-27.0-arm64-arm-64bit","macos_build":"26A428","chip":"Apple M4 Max","runtime":"coreai python runtime, AOT h16c GPU .aimodelc, SpecializationOptions.default(), no JIT","head":"the author's JointSchemaHead, fp32, CPU (parity_decoder_torch.AuthorHead)","gpu":"shared with other sessions, no _GPU_LOCK held (times are contended reference values)"},"other_gpu_processes":{"note":"process listing removed before publication (4 lines); it showed other sessions' GPU work at the time = the times are contended"},"processes":[{"shard":"shard_00","runs":["ho_t01/text/base","ho_t02/text/base","ho_t03/text/base","ho_t04/text/base","ho_t05/text/base","ho_t06/text/base","ho_t07/text/base","ho_t08/text/base","ho_t09/text/base","ho_t10/text/base","ho_j01/text/base","ho_j02/text/base","ho_j03/text/base","ho_j04/text/base","ho_j05/text/base","ho_j06/text/base","ho_j07/text/base","ho_j08/text/base","ho_l01/text/base","ho_l02/text/base","ho_i01/g256/base","ho_i01/g448/base","ho_i02/g256/base","ho_i02/g448/base","ho_i03/g256/base","ho_i03/g448/base","ho_i04/g256/base","ho_i04/g448/base","ho_i05/g256/base","ho_i05/g448/base","ho_i06/g256/base","ho_i06/g448/base","ho_i07/g256/base","ho_i07/g448/base","ho_i08/g256/base","ho_i08/g448/base","ho_p01/g256/base","ho_p01/g448/base","ho_p02/g256/base","ho_p02/g448/base"],"prompts":41,"pid":12568,"load_seconds":8.623037499986822,"load_split_seconds":{"model":0.005894374975468963,"main":8.617143125011353},"function_names":["main"],"process_wall_seconds":85.41850479098503,"reset_check":{"run":["ho_t01","text","base"],"bit_equal":true,"hidden_max_abs_diff":0.0,"wall_seconds":1.621496500010835},"contract_mismatch":[]}],"summary":{"runs":40,"expected_runs":40,"missing_runs":[],"questions":188,"argmax_equal":188,"questions_non_near_tie":186,"argmax_equal_non_near_tie":186,"near_tie_questions":2,"argmax_equal_near_tie":2,"max_abs_dp":0.008473306894302368,"max_abs_dp_non_near_tie":0.008473306894302368,"mean_of_run_mean_abs_dp":0.00036112793286965494,"mean_question_max_abs_dp":0.0006599323068586911,"max_abs_dlogit":0.050017595291137695,"min_pos_cos":0.8900131817819378,"positions":32093,"positions_below":{"0.99":15,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":6,"runs_with_positions_below_0.99":["ho_t03/text","ho_t10/text","ho_j06/text","ho_l01/text","ho_i01/g448","ho_i02/g448","ho_i07/g256","ho_i07/g448","ho_p01/g256","ho_p01/g448","ho_p02/g448"],"hidden_max_abs_diff":8.786064147949219,"hidden_max_rel_diff":0.17143305011423837,"worst_run":{"id":"ho_i06","arm":"g448","max_abs_dp":0.008473306894302368},"finite_all":true,"tokens":32093,"calls":521,"ms_per_call_median":134.34666700777598,"ms_per_call_warm_median":134.35833400581032},"by_source_arm":{"heldout_image/g256":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":31,"argmax_equal":31,"questions_non_near_tie":31,"argmax_equal_non_near_tie":31,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.0019000768661499023,"max_abs_dp_non_near_tie":0.0019000768661499023,"mean_of_run_mean_abs_dp":0.0002463807746733368,"mean_question_max_abs_dp":0.00048813929090336445,"max_abs_dlogit":0.021762549877166748,"min_pos_cos":0.8900131817819378,"positions":4522,"positions_below":{"0.99":2,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":2,"runs_with_positions_below_0.99":["ho_i07/g256"],"hidden_max_abs_diff":7.7289628982543945,"hidden_max_rel_diff":0.15996380624724557,"worst_run":{"id":"ho_i05","arm":"g256","max_abs_dp":0.0019000768661499023},"finite_all":true,"tokens":4522,"calls":74,"ms_per_call_median":134.25412500509992,"ms_per_call_warm_median":134.25412500509992},"heldout_image/g448":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":31,"argmax_equal":31,"questions_non_near_tie":31,"argmax_equal_non_near_tie":31,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.008473306894302368,"max_abs_dp_non_near_tie":0.008473306894302368,"mean_of_run_mean_abs_dp":0.00029478437627831805,"mean_question_max_abs_dp":0.0006076814306359138,"max_abs_dlogit":0.04421865940093994,"min_pos_cos":0.9776798395939983,"positions":5578,"positions_below":{"0.99":3,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":2,"runs_with_positions_below_0.99":["ho_i01/g448","ho_i02/g448","ho_i07/g448"],"hidden_max_abs_diff":8.786064147949219,"hidden_max_rel_diff":0.17143305011423837,"worst_run":{"id":"ho_i06","arm":"g448","max_abs_dp":0.008473306894302368},"finite_all":true,"tokens":5578,"calls":91,"ms_per_call_median":134.3372499977704,"ms_per_call_warm_median":134.3372499977704},"heldout_json/text":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":36,"argmax_equal":36,"questions_non_near_tie":35,"argmax_equal_non_near_tie":35,"near_tie_questions":1,"argmax_equal_near_tie":1,"max_abs_dp":0.008229732513427734,"max_abs_dp_non_near_tie":0.008229732513427734,"mean_of_run_mean_abs_dp":0.0005812588865798626,"mean_question_max_abs_dp":0.0009995844836036365,"max_abs_dlogit":0.050017595291137695,"min_pos_cos":0.9796648257196925,"positions":5621,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_j06/text"],"hidden_max_abs_diff":2.8965320587158203,"hidden_max_rel_diff":0.05994868614131057,"worst_run":{"id":"ho_j06","arm":"text","max_abs_dp":0.008229732513427734},"finite_all":true,"tokens":5621,"calls":93,"ms_per_call_median":134.37154097482562,"ms_per_call_warm_median":134.37154097482562},"heldout_photo/g256":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":10,"argmax_equal":10,"questions_non_near_tie":10,"argmax_equal_non_near_tie":10,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.0019177794456481934,"max_abs_dp_non_near_tie":0.0019177794456481934,"mean_of_run_mean_abs_dp":0.00025799359633203815,"mean_question_max_abs_dp":0.00043056868016719816,"max_abs_dlogit":0.04926559329032898,"min_pos_cos":0.9883229252244917,"positions":1254,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_p01/g256"],"hidden_max_abs_diff":4.187688827514648,"hidden_max_rel_diff":0.08667122690775941,"worst_run":{"id":"ho_p02","arm":"g256","max_abs_dp":0.0019177794456481934},"finite_all":true,"tokens":1254,"calls":20,"ms_per_call_median":134.22193750739098,"ms_per_call_warm_median":134.22193750739098},"heldout_photo/g448":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":10,"argmax_equal":10,"questions_non_near_tie":10,"argmax_equal_non_near_tie":10,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.0004940032958984375,"max_abs_dp_non_near_tie":0.0004940032958984375,"mean_of_run_mean_abs_dp":0.0001009333833102502,"mean_question_max_abs_dp":0.00016010329127311707,"max_abs_dlogit":0.01087939739227295,"min_pos_cos":0.955415087593718,"positions":1518,"positions_below":{"0.99":3,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":2,"runs_with_positions_below_0.99":["ho_p01/g448","ho_p02/g448"],"hidden_max_abs_diff":7.3300697803497314,"hidden_max_rel_diff":0.15170833387941582,"worst_run":{"id":"ho_p02","arm":"g448","max_abs_dp":0.0004940032958984375},"finite_all":true,"tokens":1518,"calls":24,"ms_per_call_median":134.23629150202032,"ms_per_call_warm_median":134.23629150202032},"heldout_text/text":{"runs":10,"expected_runs":10,"missing_runs":[],"questions":56,"argmax_equal":56,"questions_non_near_tie":55,"argmax_equal_non_near_tie":55,"near_tie_questions":1,"argmax_equal_near_tie":1,"max_abs_dp":0.006222903728485107,"max_abs_dp_non_near_tie":0.0036199092864990234,"mean_of_run_mean_abs_dp":0.0004393599390266821,"mean_question_max_abs_dp":0.0007833995623514056,"max_abs_dlogit":0.03148818016052246,"min_pos_cos":0.9790879466144176,"positions":8019,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_t03/text","ho_t10/text"],"hidden_max_abs_diff":5.011299133300781,"hidden_max_rel_diff":0.10319023781995637,"worst_run":{"id":"ho_t07","arm":"text","max_abs_dp":0.006222903728485107},"finite_all":true,"tokens":8019,"calls":131,"ms_per_call_median":134.26937500480562,"ms_per_call_warm_median":134.30916701327078},"heldout_text_long/text":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":14,"argmax_equal":14,"questions_non_near_tie":14,"argmax_equal_non_near_tie":14,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.0012440979480743408,"max_abs_dp_non_near_tie":0.0012440979480743408,"mean_of_run_mean_abs_dp":0.00017713583249133079,"mean_question_max_abs_dp":0.0003096210504216807,"max_abs_dlogit":0.015716075897216797,"min_pos_cos":0.9802225236587327,"positions":5581,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_l01/text"],"hidden_max_abs_diff":3.334559440612793,"hidden_max_rel_diff":0.06551305641071224,"worst_run":{"id":"ho_l01","arm":"text","max_abs_dp":0.0012440979480743408},"finite_all":true,"tokens":5581,"calls":88,"ms_per_call_median":134.91210449137725,"ms_per_call_warm_median":134.91210449137725}},"near_ties":[{"id":"ho_t07","arm":"text","question_id":"questions_asked","oracle_top2_margin":0.011684805154800415,"argmax_equal":true,"max_abs_dp":0.006222903728485107,"probs":[0.009835845790803432,0.4039638638496399,0.40109482407569885,0.18510544300079346],"probs_oracle":[0.009856987744569778,0.410186767578125,0.3985019624233246,0.18145422637462616]},{"id":"ho_j01","arm":"text","question_id":"late_start","oracle_top2_margin":0.0010575354099273682,"argmax_equal":true,"max_abs_dp":0.00018170475959777832,"probs":[0.5003470778465271,0.4996529221534729],"probs_oracle":[0.5005287528038025,0.4994712173938751]}],"checks":{"all_runs":true,"argmax_non_near_tie":true,"max_abs_dp":true,"mean_of_run_mean_abs_dp":true,"finite":true,"reset_bit_equal_all_processes":true},"result":"PASS","red_arm":null,"timing":{"contended":true,"S":64,"load_seconds":{"first_process":8.623037499986822,"median":8.623037499986822,"all":[8.623037499986822]},"ms_per_call_warm_runs":{"n":509,"median":134.35833400581032,"p10":133.72677498846315,"p90":135.24105800315738},"ms_per_call_all_runs":{"n":521,"median":134.34666700777598,"p10":133.728249988053,"p90":135.23995800642297},"first_call_of_process_ms":[2649.355542002013],"ms_per_token_warm_runs":{"n":39,"median":2.190111302479636,"p10":2.109990507124648,"p90":2.2654852917878667},"run_decode_seconds_warm_runs":{"n":39,"median":1.609269749984378,"p10":1.2086832168104593,"p90":1.8861357831920031},"run_wall_seconds_warm_runs":{"n":39,"median":1.619079416996101,"p10":1.2214540914108512,"p90":1.8976226754195522},"by_source_arm_warm":{"heldout_image/g256":{"runs":8,"median_tokens":555.5,"median_calls":9.0,"ms_per_call_median":134.25412500509992,"run_decode_seconds_median":1.2094309585081646},"heldout_image/g448":{"runs":8,"median_tokens":687.5,"median_calls":11.0,"ms_per_call_median":134.3372499977704,"run_decode_seconds_median":1.4778196460101753},"heldout_json/text":{"runs":8,"median_tokens":772.0,"median_calls":13.0,"ms_per_call_median":134.37154097482562,"run_decode_seconds_median":1.7450739375053672},"heldout_photo/g256":{"runs":2,"median_tokens":627.0,"median_calls":10.0,"ms_per_call_median":134.22193750739098,"run_decode_seconds_median":1.3433640625153203},"heldout_photo/g448":{"runs":2,"median_tokens":759.0,"median_calls":12.0,"ms_per_call_median":134.23629150202032,"run_decode_seconds_median":1.6108319999912055},"heldout_text/text":{"runs":9,"median_tokens":800.0,"median_calls":13.0,"ms_per_call_median":134.30916701327078,"run_decode_seconds_median":1.7520890419837087},"heldout_text_long/text":{"runs":2,"median_tokens":2790.5,"median_calls":44.0,"ms_per_call_median":134.91210449137725,"run_decode_seconds_median":5.933971542006475}}},"seconds":{"gpu_processes":85.44988358300179,"scoring":21.25672766601201,"total":106.86155579201295},"runs":[{"id":"ho_t01","arm":"text","variant":"base","tokens":708,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":4.125098082993645,"wall_seconds":4.138910750014475,"finite":true,"index":0,"shard":"shard_00","first_in_process":true,"source":"heldout_text","hidden":{"max_abs_diff":1.8131752014160156,"ref_absmax":50.3668098449707,"rel_max_abs_diff":0.03599940530275747,"min_pos_cos":0.9962073355757706,"min_pos_cos_index":413,"mean_pos_cos":0.999950617327217,"positions":708,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"leave_days","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.312131881713867e-05,"mean_abs_dp":3.1546223908662796e-05,"max_abs_dlogit":0.004570960998535156,"oracle_top2_margin":0.9403126761317253,"near_tie":false,"probs":[0.011944801546633244,0.014395558275282383,0.9570424556732178,0.01661723665893078]},{"question_id":"cover_arranged","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0002962872385978699,"mean_abs_dp":0.00029626116156578064,"max_abs_dlogit":0.009436845779418945,"oracle_top2_margin":0.8687373250722885,"near_tie":false,"probs":[0.9346649050712585,0.06533505767583847]},{"question_id":"days_left_after","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0023886561393737793,"mean_abs_dp":0.0011943653225898743,"max_abs_dlogit":0.02327561378479004,"oracle_top2_margin":0.6971522346138954,"near_tie":false,"probs":[0.056872110813856125,0.0639915019273758,0.09284339100122452,0.7862930297851562]},{"question_id":"reason","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7404556274414062e-05,"mean_abs_dp":8.705887012183666e-06,"max_abs_dlogit":0.004341602325439453,"oracle_top2_margin":0.9823572798632085,"near_tie":false,"probs":[0.9869105219841003,0.004575112368911505,0.003969552926719189,0.00454482389613986]},{"question_id":"ease_of_approval","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":9.685009717941284e-05,"mean_abs_dp":6.45791490872701e-05,"max_abs_dlogit":0.005325794219970703,"oracle_top2_margin":0.680185854434967,"near_tie":false,"probs":[0.07241052389144897,0.12361104041337967,0.803978443145752]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0023886561393737793,"mean_abs_dp":0.0003367487943786032,"max_abs_dlogit":0.02327561378479004},{"id":"ho_t02","arm":"text","variant":"base","tokens":746,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6088319580012467,"wall_seconds":1.6215286660008132,"finite":true,"index":1,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.8298606872558594,"ref_absmax":56.83818817138672,"rel_max_abs_diff":0.03219421213319114,"min_pos_cos":0.9904090600350529,"min_pos_cos_index":508,"mean_pos_cos":0.9999175662788405,"positions":746,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"in_warranty","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.001874387264251709,"mean_abs_dp":0.001874387264251709,"max_abs_dlogit":0.01286935806274414,"oracle_top2_margin":0.1861058473587036,"near_tie":false,"probs":[0.4050726890563965,0.5949273109436035]},{"question_id":"faulty_part","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00046122074127197266,"mean_abs_dp":0.00018450673669576645,"max_abs_dlogit":0.01139611005783081,"oracle_top2_margin":0.9428951684385538,"near_tie":false,"probs":[0.9563832879066467,0.009478136897087097,0.012917276471853256,0.009035065770149231,0.012186125852167606]},{"question_id":"possible_damage","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00022876262664794922,"mean_abs_dp":0.00022874027490615845,"max_abs_dlogit":0.012531042098999023,"oracle_top2_margin":0.9125267155468464,"near_tie":false,"probs":[0.9560346007347107,0.04396536573767662]},{"question_id":"remedy","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.990795135498047e-05,"mean_abs_dp":9.960494935512543e-06,"max_abs_dlogit":0.0012264251708984375,"oracle_top2_margin":0.9695126404985785,"near_tie":false,"probs":[0.0066662998870015144,0.9788604378700256,0.005094022490084171,0.00937930028885603]},{"question_id":"claim_strength","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0005040913820266724,"mean_abs_dp":0.00033608078956604004,"max_abs_dlogit":0.005591392517089844,"oracle_top2_margin":0.4374786615371704,"near_tie":false,"probs":[0.2032291740179062,0.6416352987289429,0.1551356017589569]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.001874387264251709,"mean_abs_dp":0.0003860545693896711,"max_abs_dlogit":0.01286935806274414},{"id":"ho_t03","arm":"text","variant":"base","tokens":842,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.878324374993099,"wall_seconds":1.888170666003134,"finite":true,"index":2,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":5.011299133300781,"ref_absmax":48.56369400024414,"rel_max_abs_diff":0.10319023781995637,"min_pos_cos":0.9877670402008326,"min_pos_cos_index":359,"mean_pos_cos":0.9999182654068006,"positions":842,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"asks_about_pet","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.458559513092041e-05,"mean_abs_dp":3.4578144550323486e-05,"max_abs_dlogit":0.006014764308929443,"oracle_top2_margin":0.9131554178893566,"near_tie":false,"probs":[0.9566122889518738,0.04338771477341652]},{"question_id":"bedrooms","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.516580909490585e-05,"mean_abs_dp":1.850258558988571e-05,"max_abs_dlogit":0.002500772476196289,"oracle_top2_margin":0.9704894721508026,"near_tie":false,"probs":[0.008544296957552433,0.9791004061698914,0.0063462527468800545,0.006009072531014681]},{"question_id":"needs_parking","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.817941665649414e-05,"mean_abs_dp":1.8168240785598755e-05,"max_abs_dlogit":0.0033111572265625,"oracle_top2_margin":0.9770000539720058,"near_tie":false,"probs":[0.011518161743879318,0.988481879234314]},{"question_id":"viewing_time","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00020956993103027344,"mean_abs_dp":0.00010480498895049095,"max_abs_dlogit":0.006494045257568359,"oracle_top2_margin":0.8937774673104286,"near_tie":false,"probs":[0.029394201934337616,0.023489434272050858,0.9228388667106628,0.024277495220303535]},{"question_id":"other_question","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.9729137420654297e-05,"mean_abs_dp":9.845360182225704e-06,"max_abs_dlogit":0.009613990783691406,"oracle_top2_margin":0.9933792083756998,"near_tie":false,"probs":[0.001778837526217103,0.001176453661173582,0.0018183194333687425,0.9952263832092285]},{"question_id":"match","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0024101734161376953,"mean_abs_dp":0.0012760385870933533,"max_abs_dlogit":0.006142675876617432,"oracle_top2_margin":0.204206645488739,"near_tie":false,"probs":[0.08794065564870834,0.3150675594806671,0.5236930847167969,0.07329871505498886]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0024101734161376953,"mean_abs_dp":0.00028711294289678334,"max_abs_dlogit":0.009613990783691406},{"id":"ho_t04","arm":"text","variant":"base","tokens":658,"calls":11,"padded_tokens":704,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.4745385000132956,"wall_seconds":1.4840032079955563,"finite":true,"index":3,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.4534721374511719,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.030082092383894046,"min_pos_cos":0.9970379774072641,"min_pos_cos_index":269,"mean_pos_cos":0.9999663904348551,"positions":658,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_problem","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.571676254272461e-05,"mean_abs_dp":3.2578781247138974e-05,"max_abs_dlogit":0.004703044891357422,"oracle_top2_margin":0.9411561395972967,"near_tie":false,"probs":[0.012424403801560402,0.9576114416122437,0.01644536852836609,0.006579216569662094,0.006939692888408899]},{"question_id":"would_return","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":9.300000965595245e-05,"mean_abs_dp":9.299162775278091e-05,"max_abs_dlogit":0.005877971649169922,"oracle_top2_margin":0.9438176397234201,"near_tie":false,"probs":[0.9720017910003662,0.027998168021440506]},{"question_id":"food_left","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.386529326438904e-06,"mean_abs_dp":4.339031875133514e-06,"max_abs_dlogit":0.01032876968383789,"oracle_top2_margin":0.9789652321487665,"near_tie":false,"probs":[0.010513022541999817,0.9894869327545166]},{"question_id":"tone","type":"score","n_options":7,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0008855462074279785,"mean_abs_dp":0.00025300176015922,"max_abs_dlogit":0.0046086907386779785,"oracle_top2_margin":0.6996366679668427,"near_tie":false,"probs":[0.02182287536561489,0.01760968193411827,0.01954798772931099,0.8011665344238281,0.10007244348526001,0.02042539417743683,0.019355110824108124]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008855462074279785,"mean_abs_dp":0.000133035471662879,"max_abs_dlogit":0.01032876968383789},{"id":"ho_t05","arm":"text","variant":"base","tokens":727,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6094995000166818,"wall_seconds":1.619079416996101,"finite":true,"index":4,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.2555294036865234,"ref_absmax":51.32048797607422,"rel_max_abs_diff":0.024464486858968595,"min_pos_cos":0.9965155174791962,"min_pos_cos_index":273,"mean_pos_cos":0.9999543115264456,"positions":727,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"defect_kind","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.748603820800781e-05,"mean_abs_dp":4.917895421385765e-05,"max_abs_dlogit":0.0040018558502197266,"oracle_top2_margin":0.8769160024821758,"near_tie":false,"probs":[0.9176120162010193,0.028877438977360725,0.040834009647369385,0.012676535174250603]},{"question_id":"regression","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":9.553134441375732e-05,"mean_abs_dp":9.547919034957886e-05,"max_abs_dlogit":0.005087375640869141,"oracle_top2_margin":0.8895891383290291,"near_tie":false,"probs":[0.944890022277832,0.05510992556810379]},{"question_id":"loses_work","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001087188720703125,"mean_abs_dp":0.00010871514678001404,"max_abs_dlogit":0.009530305862426758,"oracle_top2_margin":0.9126163460314274,"near_tie":false,"probs":[0.9561994671821594,0.04380055144429207]},{"question_id":"small_notebooks","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.470348358154297e-06,"mean_abs_dp":4.4568441808223724e-06,"max_abs_dlogit":0.004508495330810547,"oracle_top2_margin":0.9837353955954313,"near_tie":false,"probs":[0.008136768825352192,0.9918632507324219]},{"question_id":"priority","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00024646520614624023,"mean_abs_dp":0.0001645013689994812,"max_abs_dlogit":0.005550980567932129,"oracle_top2_margin":0.47177982330322266,"near_tie":false,"probs":[0.06414894014596939,0.14608590304851532,0.6309285163879395,0.15883664786815643]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00024646520614624023,"mean_abs_dp":9.085883253387042e-05,"max_abs_dlogit":0.009530305862426758},{"id":"ho_t06","arm":"text","variant":"base","tokens":721,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.609269749984378,"wall_seconds":1.6189237909857184,"finite":true,"index":5,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.847712516784668,"ref_absmax":49.36119079589844,"rel_max_abs_diff":0.037432494779647815,"min_pos_cos":0.9979020044158756,"min_pos_cos_index":390,"mean_pos_cos":0.9999610960583869,"positions":721,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"problem","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.060434341430664e-05,"mean_abs_dp":2.0241178572177886e-05,"max_abs_dlogit":0.0028939247131347656,"oracle_top2_margin":0.9249128699302673,"near_tie":false,"probs":[0.011490913107991219,0.017635377123951912,0.9424965977668762,0.01479051448404789,0.013586538843810558]},{"question_id":"was_home","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00020547956228256226,"mean_abs_dp":0.00020543858408927917,"max_abs_dlogit":0.0075531005859375,"oracle_top2_margin":0.7574861496686935,"near_tie":false,"probs":[0.878537654876709,0.121462382376194]},{"question_id":"outcome_wanted","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00016993284225463867,"mean_abs_dp":8.498365059494972e-05,"max_abs_dlogit":0.004993081092834473,"oracle_top2_margin":0.9236043281853199,"near_tie":false,"probs":[0.017240185290575027,0.017940619960427284,0.020479943603277206,0.9443392157554626]},{"question_id":"time_sensitive","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001277327537536621,"mean_abs_dp":0.00012770667672157288,"max_abs_dlogit":0.004230499267578125,"oracle_top2_margin":0.8349910154938698,"near_tie":false,"probs":[0.9176232218742371,0.08237679302692413]},{"question_id":"mood","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009757280349731445,"mean_abs_dp":0.00048786774277687073,"max_abs_dlogit":0.0038973093032836914,"oracle_top2_margin":0.3556882441043854,"near_tie":false,"probs":[0.06522830575704575,0.1972900629043579,0.5543664693832397,0.1831151843070984]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0009757280349731445,"mean_abs_dp":0.00017993541105705148,"max_abs_dlogit":0.0075531005859375},{"id":"ho_t07","arm":"text","variant":"base","tokens":1086,"calls":17,"padded_tokens":1088,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":2.282930125016719,"wall_seconds":2.2925983330060262,"finite":true,"index":6,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.236626625061035,"ref_absmax":50.011722564697266,"rel_max_abs_diff":0.04472204735934942,"min_pos_cos":0.9948426124176317,"min_pos_cos_index":1018,"mean_pos_cos":0.999932367875766,"positions":1086,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"factor","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0014156699180603027,"mean_abs_dp":0.0005662722513079643,"max_abs_dlogit":0.01681828498840332,"oracle_top2_margin":0.7617465928196907,"near_tie":false,"probs":[0.055047400295734406,0.07823587208986282,0.8423219919204712,0.014116236008703709,0.0102785499766469]},{"question_id":"oats_total","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":9.918212890625e-05,"mean_abs_dp":5.344988312572241e-05,"max_abs_dlogit":0.03148818016052246,"oracle_top2_margin":0.848384864628315,"near_tie":false,"probs":[0.0637769103050232,0.01800268329679966,0.005904154386371374,0.9123162031173706]},{"question_id":"syrup_total","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00021392852067947388,"mean_abs_dp":0.00010696914978325367,"max_abs_dlogit":0.013095855712890625,"oracle_top2_margin":0.9178646951913834,"near_tie":false,"probs":[0.9514525532722473,0.0339689776301384,0.005346966441720724,0.009231582283973694]},{"question_id":"oats_one_weighing","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0017340779304504395,"mean_abs_dp":0.001734033226966858,"max_abs_dlogit":0.009246408939361572,"oracle_top2_margin":0.4905860126018524,"near_tie":false,"probs":[0.25297296047210693,0.7470270395278931]},{"question_id":"asks_swap","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.2158873081207275e-06,"mean_abs_dp":7.184222340583801e-06,"max_abs_dlogit":0.00932455062866211,"oracle_top2_margin":0.9517021421343088,"near_tie":false,"probs":[0.9758439064025879,0.024156132712960243]},{"question_id":"oven_temperature","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001221299171447754,"mean_abs_dp":6.106146611273289e-05,"max_abs_dlogit":0.0053060054779052734,"oracle_top2_margin":0.9683727771043777,"near_tie":false,"probs":[0.007792615331709385,0.007581804413348436,0.9765888452529907,0.008036743849515915]},{"question_id":"questions_asked","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.006222903728485107,"mean_abs_dp":0.0031220309901982546,"max_abs_dlogit":0.02948552370071411,"oracle_top2_margin":0.011684805154800415,"near_tie":true,"probs":[0.009835845790803432,0.4039638638496399,0.40109482407569885,0.18510544300079346]},{"question_id":"difficulty","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008483529090881348,"mean_abs_dp":0.0006217928603291512,"max_abs_dlogit":0.010395169258117676,"oracle_top2_margin":0.6591825112700462,"near_tie":false,"probs":[0.03822702914476395,0.0833800882101059,0.7689084410667419,0.10948438942432404]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.006222903728485107,"mean_abs_dp":0.0007646556397707298,"max_abs_dlogit":0.03148818016052246},{"id":"ho_t08","arm":"text","variant":"base","tokens":864,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.8848449579963926,"wall_seconds":1.8946287500148173,"finite":true,"index":7,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.448288917541504,"ref_absmax":54.1589241027832,"rel_max_abs_diff":0.045205641694342436,"min_pos_cos":0.9914866257458187,"min_pos_cos_index":455,"mean_pos_cos":0.9999107593143647,"positions":864,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"policy_section","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001487135887145996,"mean_abs_dp":6.597917526960373e-05,"max_abs_dlogit":0.009495735168457031,"oracle_top2_margin":0.9393905811011791,"near_tie":false,"probs":[0.016814585775136948,0.013858890160918236,0.9559402465820312,0.007540860213339329,0.005845440085977316]},{"question_id":"costs_total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011175870895385742,"mean_abs_dp":5.585385952144861e-05,"max_abs_dlogit":0.008122682571411133,"oracle_top2_margin":0.9614982744678855,"near_tie":false,"probs":[0.006006152369081974,0.9761126041412354,0.0148171903565526,0.0030640624463558197]},{"question_id":"payout","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0025677084922790527,"mean_abs_dp":0.0012838654220104218,"max_abs_dlogit":0.016382932662963867,"oracle_top2_margin":0.5377697646617889,"near_tie":false,"probs":[0.6887291669845581,0.15500478446483612,0.09448111802339554,0.0617850087583065]},{"question_id":"airline_paid","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00040245428681373596,"mean_abs_dp":0.00040245242416858673,"max_abs_dlogit":0.0060787200927734375,"oracle_top2_margin":0.901993878185749,"near_tie":false,"probs":[0.04860060289502144,0.9513993859291077]},{"question_id":"baggage_lost","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.0984560251235962e-05,"mean_abs_dp":2.098269760608673e-05,"max_abs_dlogit":0.005304574966430664,"oracle_top2_margin":0.9824593458324671,"near_tie":false,"probs":[0.00879129208624363,0.9912086725234985]},{"question_id":"clarity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0010387450456619263,"mean_abs_dp":0.0006924668947855631,"max_abs_dlogit":0.011318027973175049,"oracle_top2_margin":0.43105728924274445,"near_tie":false,"probs":[0.19851040840148926,0.6281619668006897,0.17332766950130463]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0025677084922790527,"mean_abs_dp":0.00043065219651907685,"max_abs_dlogit":0.016382932662963867},{"id":"ho_t09","arm":"text","variant":"base","tokens":867,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.8912990839744452,"wall_seconds":1.9015820410277229,"finite":true,"index":8,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.6349287033081055,"ref_absmax":50.86228942871094,"rel_max_abs_diff":0.03214422161628483,"min_pos_cos":0.9920508418920219,"min_pos_cos_index":791,"mean_pos_cos":0.9999523772153907,"positions":867,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"bartholomew_chemistry","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0032618045806884766,"mean_abs_dp":0.002174516518910726,"max_abs_dlogit":0.013718843460083008,"oracle_top2_margin":0.23581808805465698,"near_tie":false,"probs":[0.5033227801322937,0.27328044176101685,0.22339683771133423]},{"question_id":"ingrid_laptop","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011505186557769775,"mean_abs_dp":0.00011504441499710083,"max_abs_dlogit":0.012132644653320312,"oracle_top2_margin":0.9835674799978733,"near_tie":false,"probs":[0.008331310003995895,0.991668701171875]},{"question_id":"within_limit","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0013744831085205078,"mean_abs_dp":0.0013744756579399109,"max_abs_dlogit":0.010404467582702637,"oracle_top2_margin":0.5680407285690308,"near_tie":false,"probs":[0.7826458811759949,0.21735410392284393]},{"question_id":"moved_lesson","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4526769518852234e-05,"mean_abs_dp":7.2678085416555405e-06,"max_abs_dlogit":0.006979942321777344,"oracle_top2_margin":0.9909667111933231,"near_tie":false,"probs":[0.002658052369952202,0.9936457872390747,0.0013699467526748776,0.0023262170143425465]},{"question_id":"maths_length","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0013377666473388672,"mean_abs_dp":0.0006688878638669848,"max_abs_dlogit":0.018673062324523926,"oracle_top2_margin":0.835874080657959,"near_tie":false,"probs":[0.04632118344306946,0.0071938675828278065,0.05403481423854828,0.8924500942230225]},{"question_id":"planning_effort","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.002474486827850342,"mean_abs_dp":0.0016496529181798298,"max_abs_dlogit":0.009485781192779541,"oracle_top2_margin":0.03400421142578125,"near_tie":false,"probs":[0.0822102427482605,0.4735564887523651,0.4442332983016968]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0032618045806884766,"mean_abs_dp":0.0009531206192655696,"max_abs_dlogit":0.018673062324523926},{"id":"ho_t10","arm":"text","variant":"base","tokens":800,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7520890419837087,"wall_seconds":1.7625706659746356,"finite":true,"index":9,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.5124778747558594,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.03130325774565889,"min_pos_cos":0.9790879466144176,"min_pos_cos_index":445,"mean_pos_cos":0.9999232878583946,"positions":800,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"most_wanted","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.606910705566406e-05,"mean_abs_dp":3.4451810643076895e-05,"max_abs_dlogit":0.0037708282470703125,"oracle_top2_margin":0.9730223435908556,"near_tie":false,"probs":[0.005044577643275261,0.9819677472114563,0.008813758380711079,0.002095437841489911,0.0020784379448741674]},{"question_id":"mistake_reported","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0007430911064147949,"mean_abs_dp":0.0007430315017700195,"max_abs_dlogit":0.008177995681762695,"oracle_top2_margin":0.1922793686389923,"near_tie":false,"probs":[0.40460339188575745,0.5953966975212097]},{"question_id":"all_positive","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.8848648071289062e-05,"mean_abs_dp":2.8794631361961365e-05,"max_abs_dlogit":0.006183743476867676,"oracle_top2_margin":0.955326372757554,"near_tie":false,"probs":[0.022308045998215675,0.9776920080184937]},{"question_id":"digital_readers","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0010222792625427246,"mean_abs_dp":0.0005111312493681908,"max_abs_dlogit":0.00769495964050293,"oracle_top2_margin":0.8206045627593994,"near_tie":false,"probs":[0.020736631006002426,0.06450798362493515,0.8835906386375427,0.03116474114358425]},{"question_id":"reply_count","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0036199092864990234,"mean_abs_dp":0.0018099332228302956,"max_abs_dlogit":0.008512556552886963,"oracle_top2_margin":0.3369687795639038,"near_tie":false,"probs":[0.02973908744752407,0.3059113621711731,0.6359784603118896,0.0283710528165102]},{"question_id":"reception","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0030693411827087402,"mean_abs_dp":0.0016149384900927544,"max_abs_dlogit":0.010805070400238037,"oracle_top2_margin":0.38388296961784363,"near_tie":false,"probs":[0.023392600938677788,0.6616883277893066,0.28381016850471497,0.031108934432268143]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0036199092864990234,"mean_abs_dp":0.0008314249127925862,"max_abs_dlogit":0.010805070400238037},{"id":"ho_j01","arm":"text","variant":"base","tokens":769,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7555385830055457,"wall_seconds":1.7663747499755118,"finite":true,"index":10,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.060110092163086,"ref_absmax":51.30471420288086,"rel_max_abs_diff":0.040154401484754916,"min_pos_cos":0.9966138527562396,"min_pos_cos_index":447,"mean_pos_cos":0.9999732259272985,"positions":769,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"sick_days","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.422136306762695e-05,"mean_abs_dp":4.210416227579117e-05,"max_abs_dlogit":0.005269765853881836,"oracle_top2_margin":0.9564286395907402,"near_tie":false,"probs":[0.012769991531968117,0.9693126678466797,0.010300783440470695,0.007616570685058832]},{"question_id":"longest_day","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0005373954772949219,"mean_abs_dp":0.000281487125903368,"max_abs_dlogit":0.011201620101928711,"oracle_top2_margin":0.5593627244234085,"near_tie":false,"probs":[0.18487763404846191,0.05709221959114075,0.7435615062713623,0.01446870993822813]},{"question_id":"approved","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.52346420288086e-06,"mean_abs_dp":8.514616638422012e-06,"max_abs_dlogit":0.002603769302368164,"oracle_top2_margin":0.9902706993743777,"near_tie":false,"probs":[0.0048561617732048035,0.9951438903808594]},{"question_id":"late_start","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00018170475959777832,"mean_abs_dp":0.00018168985843658447,"max_abs_dlogit":0.0034098029136657715,"oracle_top2_margin":0.0010575354099273682,"near_tie":true,"probs":[0.5003470778465271,0.4996529221534729]},{"question_id":"hours_concern","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00025403499603271484,"mean_abs_dp":0.00016934672991434732,"max_abs_dlogit":0.0035909414291381836,"oracle_top2_margin":0.3177451193332672,"near_tie":false,"probs":[0.30271419882774353,0.6209272742271423,0.07635857164859772]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0005373954772949219,"mean_abs_dp":0.0001455209528406461,"max_abs_dlogit":0.011201620101928711},{"id":"ho_j02","arm":"text","variant":"base","tokens":335,"calls":6,"padded_tokens":384,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":0.8075559579883702,"wall_seconds":0.8182549579942133,"finite":true,"index":11,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":0.794621467590332,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.016446019492502684,"min_pos_cos":0.9988922547222996,"min_pos_cos_index":254,"mean_pos_cos":0.9999834448699089,"positions":335,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"beyond_tolerance","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0011948943138122559,"mean_abs_dp":0.0004763215159376462,"max_abs_dlogit":0.013053178787231445,"oracle_top2_margin":0.10029461979866028,"near_tie":false,"probs":[0.024087771773338318,0.06434983015060425,0.36784034967422485,0.46595343947410583,0.058615297079086304,0.01915331557393074]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0011948943138122559,"mean_abs_dp":0.0004763215159376462,"max_abs_dlogit":0.013053178787231445},{"id":"ho_j03","arm":"text","variant":"base","tokens":775,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7496338339988142,"wall_seconds":1.7617956250032876,"finite":true,"index":12,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.6736774444580078,"ref_absmax":49.201229095458984,"rel_max_abs_diff":0.034016984437742016,"min_pos_cos":0.9956219539598031,"min_pos_cos_index":414,"mean_pos_cos":0.99994291201972,"positions":775,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"excursion","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0003343448042869568,"mean_abs_dp":0.0003343336284160614,"max_abs_dlogit":0.005454063415527344,"oracle_top2_margin":0.8218574672937393,"near_tie":false,"probs":[0.9105944037437439,0.0894056037068367]},{"question_id":"first_excursion","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00011730194091796875,"mean_abs_dp":4.040263593196869e-05,"max_abs_dlogit":0.006276607513427734,"oracle_top2_margin":0.9088389798998833,"near_tie":false,"probs":[0.011960510164499283,0.014732557348906994,0.9267274141311646,0.014499710872769356,0.014327709563076496,0.017752142623066902]},{"question_id":"cause","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00010404549539089203,"mean_abs_dp":6.75637274980545e-05,"max_abs_dlogit":0.016313552856445312,"oracle_top2_margin":0.9396330676972866,"near_tie":false,"probs":[0.01574067212641239,0.955205500125885,0.014264329336583614,0.014789433218538761]},{"question_id":"recovered","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00030450522899627686,"mean_abs_dp":0.0003044828772544861,"max_abs_dlogit":0.006945610046386719,"oracle_top2_margin":0.8434159979224205,"near_tie":false,"probs":[0.9214035272598267,0.07859649509191513]},{"question_id":"risk","type":"score","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00186997652053833,"mean_abs_dp":0.0007480032742023468,"max_abs_dlogit":0.012205660343170166,"oracle_top2_margin":0.36235082149505615,"near_tie":false,"probs":[0.058309707790613174,0.10404159873723984,0.5659739375114441,0.20080268383026123,0.07087203860282898]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00186997652053833,"mean_abs_dp":0.00029106947936509783,"max_abs_dlogit":0.016313552856445312},{"id":"ho_j04","arm":"text","variant":"base","tokens":610,"calls":10,"padded_tokens":640,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.3454358750022948,"wall_seconds":1.3564840420149267,"finite":true,"index":13,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.1756305694580078,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.02433163971761246,"min_pos_cos":0.9968758207498357,"min_pos_cos_index":333,"mean_pos_cos":0.9999484543019828,"positions":610,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"free_cancellation","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00146406888961792,"mean_abs_dp":0.001464053988456726,"max_abs_dlogit":0.050017595291137695,"oracle_top2_margin":0.640770673751831,"near_tie":false,"probs":[0.18107870221138,0.8189212679862976]},{"question_id":"party","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001469254493713379,"mean_abs_dp":7.346505299210548e-05,"max_abs_dlogit":0.0077207088470458984,"oracle_top2_margin":0.9523126352578402,"near_tie":false,"probs":[0.011366274207830429,0.012053066864609718,0.9645645022392273,0.012016133405268192]},{"question_id":"kitchen_need","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00016069412231445312,"mean_abs_dp":8.036918006837368e-05,"max_abs_dlogit":0.015035390853881836,"oracle_top2_margin":0.9280225206166506,"near_tie":false,"probs":[0.025876101106405258,0.9541028738021851,0.010352594777941704,0.009668393060564995]},{"question_id":"special_occasion","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0007643848657608032,"mean_abs_dp":0.0007643476128578186,"max_abs_dlogit":0.015790939331054688,"oracle_top2_margin":0.8431091755628586,"near_tie":false,"probs":[0.9207902550697327,0.07920977473258972]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00146406888961792,"mean_abs_dp":0.00042267834457258385,"max_abs_dlogit":0.050017595291137695},{"id":"ho_j05","arm":"text","variant":"base","tokens":775,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7461837500159163,"wall_seconds":1.7562715419917367,"finite":true,"index":14,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.181201934814453,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.045143619950042624,"min_pos_cos":0.9922488611460909,"min_pos_cos_index":169,"mean_pos_cos":0.9999583407594332,"positions":775,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"missed_month","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00011843442916870117,"mean_abs_dp":4.736874252557755e-05,"max_abs_dlogit":0.007516801357269287,"oracle_top2_margin":0.9646178442053497,"near_tie":false,"probs":[0.007305780425667763,0.007578104734420776,0.9723471403121948,0.0060436618514359,0.006725321989506483]},{"question_id":"july_late","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001289844512939453,"mean_abs_dp":0.00012895837426185608,"max_abs_dlogit":0.00994729995727539,"oracle_top2_margin":0.8146922215819359,"near_tie":false,"probs":[0.9072171449661255,0.09278284013271332]},{"question_id":"double_payment","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0006235837936401367,"mean_abs_dp":0.0006235837936401367,"max_abs_dlogit":0.006321579217910767,"oracle_top2_margin":0.33765122294425964,"near_tie":false,"probs":[0.6694492101669312,0.33055081963539124]},{"question_id":"owed","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00179213285446167,"mean_abs_dp":0.0008960254490375519,"max_abs_dlogit":0.009110808372497559,"oracle_top2_margin":0.13490816950798035,"near_tie":false,"probs":[0.09724918007850647,0.15396375954151154,0.307841032743454,0.44094595313072205]},{"question_id":"reliability","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0009071230888366699,"mean_abs_dp":0.0005416534841060638,"max_abs_dlogit":0.0035625696182250977,"oracle_top2_margin":0.2541508674621582,"near_tie":false,"probs":[0.16789165139198303,0.5055078864097595,0.24960309267044067,0.07699737697839737]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00179213285446167,"mean_abs_dp":0.00044074375182390213,"max_abs_dlogit":0.00994729995727539},{"id":"ho_j06","arm":"text","variant":"base","tokens":733,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6094128330005333,"wall_seconds":1.6191715419990942,"finite":true,"index":15,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.8965320587158203,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.05994868614131057,"min_pos_cos":0.9796648257196925,"min_pos_cos_index":497,"mean_pos_cos":0.9998733390874744,"positions":733,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"hides_endpoints","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.9163435101509094e-05,"mean_abs_dp":2.9155053198337555e-05,"max_abs_dlogit":0.002924799919128418,"oracle_top2_margin":0.9664727952331305,"near_tie":false,"probs":[0.01673441380262375,0.9832655191421509]},{"question_id":"route_audience","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.0338764190673828e-05,"mean_abs_dp":2.0256265997886658e-05,"max_abs_dlogit":0.01099538803100586,"oracle_top2_margin":0.9730440806597471,"near_tie":false,"probs":[0.010806945152580738,0.9838067293167114,0.005386390723288059]},{"question_id":"marketing","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.950910806655884e-06,"mean_abs_dp":4.949048161506653e-06,"max_abs_dlogit":0.0014917850494384766,"oracle_top2_margin":0.9910305761732161,"near_tie":false,"probs":[0.004479759838432074,0.9955202341079712]},{"question_id":"zone_boundaries","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.008229732513427734,"mean_abs_dp":0.00423861388117075,"max_abs_dlogit":0.03548622131347656,"oracle_top2_margin":0.3823526203632355,"near_tie":false,"probs":[0.6353611946105957,0.26932480931282043,0.05512790381908417,0.0401860773563385]},{"question_id":"unit_mismatch","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.004162311553955078,"mean_abs_dp":0.004162296652793884,"max_abs_dlogit":0.02244621515274048,"oracle_top2_margin":0.3890989124774933,"near_tie":false,"probs":[0.3012882471084595,0.6987117528915405]},{"question_id":"privacy_level","type":"score","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0004755258560180664,"mean_abs_dp":0.00019322289153933525,"max_abs_dlogit":0.0037175267934799194,"oracle_top2_margin":0.1093018651008606,"near_tie":false,"probs":[0.1468321979045868,0.46613043546676636,0.35696667432785034,0.024699851870536804,0.005370866972953081]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.008229732513427734,"mean_abs_dp":0.0014652300160378218,"max_abs_dlogit":0.03548622131347656},{"id":"ho_j07","arm":"text","variant":"base","tokens":826,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7439641249948181,"wall_seconds":1.7538186660094652,"finite":true,"index":16,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.3351507186889648,"ref_absmax":50.477657318115234,"rel_max_abs_diff":0.026450330495227062,"min_pos_cos":0.9968847454494709,"min_pos_cos_index":391,"mean_pos_cos":0.9999489170904176,"positions":826,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"unassigned","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0015503168106079102,"mean_abs_dp":0.0006201189011335373,"max_abs_dlogit":0.034036874771118164,"oracle_top2_margin":0.928688820451498,"near_tie":false,"probs":[0.9435779452323914,0.01372026652097702,0.013585325330495834,0.012066518887877464,0.01704994961619377]},{"question_id":"open_high","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0004330873489379883,"mean_abs_dp":0.00017323102802038192,"max_abs_dlogit":0.019780218601226807,"oracle_top2_margin":0.9467990919947624,"near_tie":false,"probs":[0.0076020085252821445,0.016195770353078842,0.9635508060455322,0.007992143742740154,0.004659306723624468]},{"question_id":"shared_theme","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.0609626770019531e-05,"mean_abs_dp":7.106777047738433e-06,"max_abs_dlogit":0.00351715087890625,"oracle_top2_margin":0.9929788664449006,"near_tie":false,"probs":[0.0014187389751896262,0.9950202703475952,0.0020343700889497995,0.0015265322290360928]},{"question_id":"all_closed","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.863739013671875e-05,"mean_abs_dp":4.862062633037567e-05,"max_abs_dlogit":0.005819082260131836,"oracle_top2_margin":0.9899770021438599,"near_tie":false,"probs":[0.004962895065546036,0.9950371384620667]},{"question_id":"wider_incident","type":"score","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0014390349388122559,"mean_abs_dp":0.0014390051364898682,"max_abs_dlogit":0.010492324829101562,"oracle_top2_margin":0.13046076893806458,"near_tie":false,"probs":[0.43333062529563904,0.5666694045066833]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0015503168106079102,"mean_abs_dp":0.000387246015533391,"max_abs_dlogit":0.034036874771118164},{"id":"ho_j08","arm":"text","variant":"base","tokens":798,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.746532916004071,"wall_seconds":1.7565760410216171,"finite":true,"index":17,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.136301040649414,"ref_absmax":52.01266860961914,"rel_max_abs_diff":0.021846620660399423,"min_pos_cos":0.9966376317564863,"min_pos_cos_index":224,"mean_pos_cos":0.9999771206618357,"positions":798,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"all_pass","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00108414888381958,"mean_abs_dp":0.0010841190814971924,"max_abs_dlogit":0.0036080479621887207,"oracle_top2_margin":0.521710604429245,"near_tie":false,"probs":[0.23806056380271912,0.7619394063949585]},{"question_id":"failing_dimension","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0043204426765441895,"mean_abs_dp":0.0017603799700737,"max_abs_dlogit":0.013166308403015137,"oracle_top2_margin":0.3418823480606079,"near_tie":false,"probs":[0.528283953666687,0.08245014399290085,0.17526927590370178,0.17918777465820312,0.034808866679668427]},{"question_id":"failing_sample","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0019401013851165771,"mean_abs_dp":0.001288481056690216,"max_abs_dlogit":0.010736942291259766,"oracle_top2_margin":0.039329737424850464,"near_tie":false,"probs":[0.3735421597957611,0.12779861688613892,0.1678875833749771,0.33077162504196167]},{"question_id":"disposition","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0006949901580810547,"mean_abs_dp":0.000463336706161499,"max_abs_dlogit":0.00504070520401001,"oracle_top2_margin":0.41586053371429443,"near_tie":false,"probs":[0.25248998403549194,0.6695806384086609,0.07792939245700836]},{"question_id":"capability","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0004343688488006592,"mean_abs_dp":0.00021715648472309113,"max_abs_dlogit":0.003127574920654297,"oracle_top2_margin":0.03322687745094299,"near_tie":false,"probs":[0.05930859223008156,0.31059756875038147,0.2868969440460205,0.3431968688964844]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0043204426765441895,"mean_abs_dp":0.0010212610165278118,"max_abs_dlogit":0.013166308403015137},{"id":"ho_l01","arm":"text","variant":"base","tokens":2850,"calls":45,"padded_tokens":2880,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":6.07242804201087,"wall_seconds":6.083007209002972,"finite":true,"index":18,"shard":"shard_00","first_in_process":false,"source":"heldout_text_long","hidden":{"max_abs_diff":3.334559440612793,"ref_absmax":50.8991584777832,"rel_max_abs_diff":0.06551305641071224,"min_pos_cos":0.9802225236587327,"min_pos_cos_index":1103,"mean_pos_cos":0.9999214187412877,"positions":2850,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"root_cause","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001665353775024414,"mean_abs_dp":6.663268432021141e-05,"max_abs_dlogit":0.010225296020507812,"oracle_top2_margin":0.9651294462382793,"near_tie":false,"probs":[0.9759581089019775,0.011116624809801579,0.004219430964440107,0.004265964962542057,0.004439882468432188]},{"question_id":"peak_temperature","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.259845733642578e-05,"mean_abs_dp":3.6287237890064716e-05,"max_abs_dlogit":0.003400564193725586,"oracle_top2_margin":0.9787671356461942,"near_tie":false,"probs":[0.9862902760505676,0.007630165666341782,0.003344635246321559,0.0027348955627530813]},{"question_id":"first_text_reached","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00018006563186645508,"mean_abs_dp":0.00018005073070526123,"max_abs_dlogit":0.00792837142944336,"oracle_top2_margin":0.9262152798473835,"near_tie":false,"probs":[0.037072401493787766,0.9629275798797607]},{"question_id":"dough_outcome","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.928598016500473e-05,"mean_abs_dp":1.962174428626895e-05,"max_abs_dlogit":0.0052852630615234375,"oracle_top2_margin":0.9787769615650177,"near_tie":false,"probs":[0.0040815104730427265,0.9855572581291199,0.006839132867753506,0.003522203303873539]},{"question_id":"standby_ready","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.0617077350616455e-05,"mean_abs_dp":1.0613352060317993e-05,"max_abs_dlogit":0.01117086410522461,"oracle_top2_margin":0.9688542913645506,"near_tie":false,"probs":[0.015583479776978493,0.9844165444374084]},{"question_id":"above_limit","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0012440979480743408,"mean_abs_dp":0.0006220964714884758,"max_abs_dlogit":0.015716075897216797,"oracle_top2_margin":0.15288570523262024,"near_tie":false,"probs":[0.1733592301607132,0.46890565752983093,0.31730780005455017,0.04042721167206764]},{"question_id":"seriousness","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0009382963180541992,"mean_abs_dp":0.0004795733839273453,"max_abs_dlogit":0.002811431884765625,"oracle_top2_margin":0.03678572177886963,"near_tie":false,"probs":[0.037833355367183685,0.47070199251174927,0.4320511519908905,0.05941356346011162]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0012440979480743408,"mean_abs_dp":0.00021379227750003337,"max_abs_dlogit":0.015716075897216797},{"id":"ho_l02","arm":"text","variant":"base","tokens":2731,"calls":43,"padded_tokens":2752,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":5.79551504200208,"wall_seconds":5.806020583026111,"finite":true,"index":19,"shard":"shard_00","first_in_process":false,"source":"heldout_text_long","hidden":{"max_abs_diff":1.1159286499023438,"ref_absmax":56.4263916015625,"rel_max_abs_diff":0.019776714729202047,"min_pos_cos":0.995852622480905,"min_pos_cos_index":2718,"mean_pos_cos":0.9999706118723315,"positions":2731,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"break_notice","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0003107786178588867,"mean_abs_dp":0.00012431256473064423,"max_abs_dlogit":0.007065951824188232,"oracle_top2_margin":0.9312390778213739,"near_tie":false,"probs":[0.023194188252091408,0.005245206877589226,0.9549199342727661,0.011006011627614498,0.005634656175971031]},{"question_id":"review_direction","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.5670738220214844e-05,"mean_abs_dp":2.7840840630233288e-05,"max_abs_dlogit":0.0030336380004882812,"oracle_top2_margin":0.9693590383976698,"near_tie":false,"probs":[0.007077674847096205,0.004143795929849148,0.9790263175964355,0.009752172976732254]},{"question_id":"sublet_part","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.655122756958008e-05,"mean_abs_dp":4.653260111808777e-05,"max_abs_dlogit":0.012472748756408691,"oracle_top2_margin":0.9782952684909105,"near_tie":false,"probs":[0.010898908600211143,0.9891011118888855]},{"question_id":"roof","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.519918441772461e-05,"mean_abs_dp":8.871604222804308e-06,"max_abs_dlogit":0.004299163818359375,"oracle_top2_margin":0.985643103485927,"near_tie":false,"probs":[0.003887577448040247,0.9895433187484741,0.003305186051875353,0.0032639189157634974]},{"question_id":"deposit","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0006784796714782715,"mean_abs_dp":0.00033926067408174276,"max_abs_dlogit":0.008603394031524658,"oracle_top2_margin":0.8725584335625172,"near_tie":false,"probs":[0.0570484921336174,0.9283623695373535,0.010813150554895401,0.00377603224478662]},{"question_id":"kilns","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.0751187801361084e-05,"mean_abs_dp":1.0740011930465698e-05,"max_abs_dlogit":0.0032312870025634766,"oracle_top2_margin":0.9571442604064941,"near_tie":false,"probs":[0.9785828590393066,0.02141711860895157]},{"question_id":"balance","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0005657672882080078,"mean_abs_dp":0.0003771682580312093,"max_abs_dlogit":0.001558065414428711,"oracle_top2_margin":0.31222064793109894,"near_tie":false,"probs":[0.5570178627967834,0.24388115108013153,0.19910092651844025]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0006784796714782715,"mean_abs_dp":0.00014047938748262823,"max_abs_dlogit":0.012472748756408691},{"id":"ho_i01","arm":"g256","variant":"base","tokens":529,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2086345840070862,"wall_seconds":1.2213366250216495,"finite":true,"index":20,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.2411069869995117,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.025686785651378254,"min_pos_cos":0.9979878547764754,"min_pos_cos_index":85,"mean_pos_cos":0.9999763531693528,"positions":529,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_slice","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00022429227828979492,"mean_abs_dp":0.00011216499842703342,"max_abs_dlogit":0.00624828040599823,"oracle_top2_margin":0.9263720586895943,"near_tie":false,"probs":[0.031659677624702454,0.0056935823522508144,0.9584200382232666,0.0042266808450222015]},{"question_id":"over_half","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0018739700317382812,"mean_abs_dp":0.0018739551305770874,"max_abs_dlogit":0.00823509693145752,"oracle_top2_margin":0.4712936580181122,"near_tie":false,"probs":[0.266227126121521,0.733772873878479]},{"question_id":"smallest_label","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00010734796524047852,"mean_abs_dp":5.365267861634493e-05,"max_abs_dlogit":0.008925259113311768,"oracle_top2_margin":0.9675928661599755,"near_tie":false,"probs":[0.004588468931615353,0.9776032567024231,0.009860311634838581,0.007948043756186962]},{"question_id":"blue_share","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.648098886013031e-05,"mean_abs_dp":3.766020139058431e-05,"max_abs_dlogit":0.0020971298217773438,"oracle_top2_margin":0.9218912087380886,"near_tie":false,"probs":[0.026796994730830193,0.9486523270606995,0.02455066330730915]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0018739700317382812,"mean_abs_dp":0.00034801242873072624,"max_abs_dlogit":0.008925259113311768},{"id":"ho_i01","arm":"g448","variant":"base","tokens":661,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4768362919858191,"wall_seconds":1.4932392920018174,"finite":true,"index":21,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.7296022176742554,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.056493787508959045,"min_pos_cos":0.9861703468594533,"min_pos_cos_index":154,"mean_pos_cos":0.9999259722092505,"positions":661,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"biggest_slice","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001589059829711914,"mean_abs_dp":7.945823017507792e-05,"max_abs_dlogit":0.0037369132041931152,"oracle_top2_margin":0.9189082495868206,"near_tie":false,"probs":[0.03545631468296051,0.005241437815129757,0.9546483755111694,0.004653864540159702]},{"question_id":"over_half","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0006813406944274902,"mean_abs_dp":0.0006813406944274902,"max_abs_dlogit":0.001997232437133789,"oracle_top2_margin":0.3951268196105957,"near_tie":false,"probs":[0.30311793088912964,0.6968820691108704]},{"question_id":"smallest_label","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.8287450075149536e-05,"mean_abs_dp":9.124749340116978e-06,"max_abs_dlogit":0.007399559020996094,"oracle_top2_margin":0.9696091758087277,"near_tie":false,"probs":[0.0037097756285220385,0.9805256724357605,0.010911080054938793,0.004853470250964165]},{"question_id":"blue_share","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":9.554624557495117e-05,"mean_abs_dp":6.37490302324295e-05,"max_abs_dlogit":0.002695322036743164,"oracle_top2_margin":0.9033529125154018,"near_tie":false,"probs":[0.028239261358976364,0.9376481175422668,0.03411255404353142]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0006813406944274902,"mean_abs_dp":0.00014678926135484988,"max_abs_dlogit":0.007399559020996094},{"id":"ho_i02","arm":"g256","variant":"base","tokens":570,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2077074169937987,"wall_seconds":1.2200976669846568,"finite":true,"index":22,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.3426694869995117,"ref_absmax":51.21390151977539,"rel_max_abs_diff":0.04574284359286568,"min_pos_cos":0.9951902225291607,"min_pos_cos_index":96,"mean_pos_cos":0.9999504294571424,"positions":570,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"harlowick_platform","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.18832015991211e-05,"mean_abs_dp":3.589334664866328e-05,"max_abs_dlogit":0.006119251251220703,"oracle_top2_margin":0.9853119025938213,"near_tie":false,"probs":[0.9892055988311768,0.003663947805762291,0.003988931886851788,0.003141425782814622]},{"question_id":"earliest","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0002453327178955078,"mean_abs_dp":0.00012266496196389198,"max_abs_dlogit":0.01115494966506958,"oracle_top2_margin":0.9508627159520984,"near_tie":false,"probs":[0.010028918273746967,0.965150773525238,0.013942291028797626,0.01087803766131401]},{"question_id":"platform_3","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0002872347831726074,"mean_abs_dp":0.00014361622743308544,"max_abs_dlogit":0.004090309143066406,"oracle_top2_margin":0.9140067957341671,"near_tie":false,"probs":[0.014001891016960144,0.027373982593417168,0.941866934299469,0.016757167875766754]},{"question_id":"after_nine","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":9.066425263881683e-05,"mean_abs_dp":9.066145867109299e-05,"max_abs_dlogit":0.003988981246948242,"oracle_top2_margin":0.943212391808629,"near_tie":false,"probs":[0.9715155363082886,0.028484467417001724]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0002872347831726074,"mean_abs_dp":9.92872186803392e-05,"max_abs_dlogit":0.01115494966506958},{"id":"ho_i02","arm":"g448","variant":"base","tokens":702,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4770559169992339,"wall_seconds":1.4943944169790484,"finite":true,"index":23,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":8.786064147949219,"ref_absmax":51.250701904296875,"rel_max_abs_diff":0.17143305011423837,"min_pos_cos":0.9814472838289593,"min_pos_cos_index":227,"mean_pos_cos":0.999926627711488,"positions":702,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"harlowick_platform","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.045269012451172e-05,"mean_abs_dp":3.5232631489634514e-05,"max_abs_dlogit":0.009176492691040039,"oracle_top2_margin":0.9784979093819857,"near_tie":false,"probs":[0.9842733144760132,0.005676173605024815,0.00567203713580966,0.004378491546958685]},{"question_id":"earliest","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00026530027389526367,"mean_abs_dp":0.00013624830171465874,"max_abs_dlogit":0.008460044860839844,"oracle_top2_margin":0.9492904739454389,"near_tie":false,"probs":[0.010568870231509209,0.9632666110992432,0.013490218669176102,0.01267428882420063]},{"question_id":"platform_3","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0007925629615783691,"mean_abs_dp":0.0003962907940149307,"max_abs_dlogit":0.013582944869995117,"oracle_top2_margin":0.8928603697568178,"near_tie":false,"probs":[0.018450524657964706,0.02966676652431488,0.9237014055252075,0.028181303292512894]},{"question_id":"after_nine","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00010813958942890167,"mean_abs_dp":0.00010813120752573013,"max_abs_dlogit":0.008587807416915894,"oracle_top2_margin":0.9381010439246893,"near_tie":false,"probs":[0.9691586494445801,0.030841343104839325]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0007925629615783691,"mean_abs_dp":0.00017766780885202543,"max_abs_dlogit":0.013582944869995117},{"id":"ho_i03","arm":"g256","variant":"base","tokens":541,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2091545000148471,"wall_seconds":1.2214834580081515,"finite":true,"index":24,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.0517630577087402,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.021768035762587228,"min_pos_cos":0.9989964629327115,"min_pos_cos_index":90,"mean_pos_cos":0.9999690032018445,"positions":541,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"limit","type":"choice","n_options":10,"argmax":6,"argmax_oracle":6,"argmax_equal":true,"max_abs_dp":8.004903793334961e-05,"mean_abs_dp":1.6002031043171882e-05,"max_abs_dlogit":0.002167224884033203,"oracle_top2_margin":0.9458053046837449,"near_tie":false,"probs":[0.005396993365138769,0.004849626682698727,0.00459259795024991,0.004830899182707071,0.004765314981341362,0.006067626178264618,0.952551007270813,0.0068295602686703205,0.005205813329666853,0.004910476505756378]},{"question_id":"outline","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.86102294921875e-05,"mean_abs_dp":1.4300458133220673e-05,"max_abs_dlogit":0.0010749101638793945,"oracle_top2_margin":0.9205611497163773,"near_tie":false,"probs":[0.021614016965031624,0.9422078132629395,0.018053455278277397,0.01812472753226757]},{"question_id":"seventy_ok","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00014767050743103027,"mean_abs_dp":0.00014762580394744873,"max_abs_dlogit":0.017756521701812744,"oracle_top2_margin":0.955428808927536,"near_tie":false,"probs":[0.022137939929962158,0.9778620004653931]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00014767050743103027,"mean_abs_dp":3.2029609428718686e-05,"max_abs_dlogit":0.017756521701812744},{"id":"ho_i03","arm":"g448","variant":"base","tokens":673,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4785833750211168,"wall_seconds":1.4952473330195062,"finite":true,"index":25,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.8633193969726562,"ref_absmax":48.58998107910156,"rel_max_abs_diff":0.05892818505756042,"min_pos_cos":0.9943495326188867,"min_pos_cos_index":190,"mean_pos_cos":0.9999479106991135,"positions":673,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"limit","type":"choice","n_options":10,"argmax":6,"argmax_oracle":6,"argmax_equal":true,"max_abs_dp":6.085634231567383e-05,"mean_abs_dp":1.2174528092145919e-05,"max_abs_dlogit":0.002861499786376953,"oracle_top2_margin":0.947433871217072,"near_tie":false,"probs":[0.005208222195506096,0.004797281697392464,0.004542327951639891,0.00465904176235199,0.004611376207321882,0.005547111853957176,0.9541276097297668,0.00661504827439785,0.005157226696610451,0.004734693095088005]},{"question_id":"outline","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00018352270126342773,"mean_abs_dp":9.174062870442867e-05,"max_abs_dlogit":0.003849327564239502,"oracle_top2_margin":0.9329899717122316,"near_tie":false,"probs":[0.018189504742622375,0.9509569406509399,0.015048368833959103,0.015805132687091827]},{"question_id":"seventy_ok","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.8715858459472656e-05,"mean_abs_dp":1.8697232007980347e-05,"max_abs_dlogit":0.002751290798187256,"oracle_top2_margin":0.9579436834901571,"near_tie":false,"probs":[0.021009476855397224,0.9789905548095703]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00018352270126342773,"mean_abs_dp":3.288139123469591e-05,"max_abs_dlogit":0.003849327564239502},{"id":"ho_i04","arm":"g256","variant":"base","tokens":437,"calls":7,"padded_tokens":448,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":0.938212624983862,"wall_seconds":0.9508122919942252,"finite":true,"index":26,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":0.5114932060241699,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.010586226925777677,"min_pos_cos":0.9991486865141898,"min_pos_cos_index":319,"mean_pos_cos":0.9999803924095471,"positions":437,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"reading","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0004235506057739258,"mean_abs_dp":0.00021177972666919231,"max_abs_dlogit":0.010123491287231445,"oracle_top2_margin":0.8822150155901909,"near_tie":false,"probs":[0.011420130729675293,0.043428417295217514,0.9249376058578491,0.02021390199661255]},{"question_id":"below_zero","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.249392077326775e-06,"mean_abs_dp":4.2406609281897545e-06,"max_abs_dlogit":0.0009744167327880859,"oracle_top2_margin":0.9933206476271152,"near_tie":false,"probs":[0.003335454734042287,0.9966645836830139]},{"question_id":"warmth","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009417235851287842,"mean_abs_dp":0.0004708436317741871,"max_abs_dlogit":0.006553769111633301,"oracle_top2_margin":0.25455665588378906,"near_tie":false,"probs":[0.013862171210348606,0.21713554859161377,0.5108689069747925,0.2581333816051483]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0009417235851287842,"mean_abs_dp":0.0002738974755629897,"max_abs_dlogit":0.010123491287231445},{"id":"ho_i04","arm":"g448","variant":"base","tokens":569,"calls":9,"padded_tokens":576,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.2086953750113025,"wall_seconds":1.225602958991658,"finite":true,"index":27,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.316802978515625,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.047950113994184584,"min_pos_cos":0.99465770900687,"min_pos_cos_index":161,"mean_pos_cos":0.9999554621593844,"positions":569,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"reading","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008000731468200684,"mean_abs_dp":0.00040003610774874687,"max_abs_dlogit":0.010277748107910156,"oracle_top2_margin":0.8055376335978508,"near_tie":false,"probs":[0.015552516095340252,0.08556576818227768,0.8896944522857666,0.00918728020042181]},{"question_id":"below_zero","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-06,"mean_abs_dp":3.5617267712950706e-06,"max_abs_dlogit":0.001918792724609375,"oracle_top2_margin":0.9941885098814964,"near_tie":false,"probs":[0.0029093108605593443,0.9970906972885132]},{"question_id":"warmth","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008200705051422119,"mean_abs_dp":0.0004100289661437273,"max_abs_dlogit":0.008714914321899414,"oracle_top2_margin":0.4384543299674988,"near_tie":false,"probs":[0.010842874646186829,0.2053760439157486,0.6432780027389526,0.14050304889678955]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008200705051422119,"mean_abs_dp":0.0003247383749112487,"max_abs_dlogit":0.010277748107910156},{"id":"ho_i05","arm":"g256","variant":"base","tokens":701,"calls":11,"padded_tokens":704,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.475328584027011,"wall_seconds":1.4882766249938868,"finite":true,"index":28,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.3831844329833984,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.028627368096602757,"min_pos_cos":0.9946991359852169,"min_pos_cos_index":613,"mean_pos_cos":0.9999520462006172,"positions":701,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"first_weekday","type":"choice","n_options":7,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0017037391662597656,"mean_abs_dp":0.00048679273043360027,"max_abs_dlogit":0.021762549877166748,"oracle_top2_margin":0.686411127448082,"near_tie":false,"probs":[0.031058581545948982,0.7600809335708618,0.024353358894586563,0.027982749044895172,0.034381698817014694,0.07142974436283112,0.050712913274765015]},{"question_id":"circled","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00012093782424926758,"mean_abs_dp":6.0492311604321e-05,"max_abs_dlogit":0.010085105895996094,"oracle_top2_margin":0.9759097397327423,"near_tie":false,"probs":[0.9827781319618225,0.0070185293443500996,0.003922770265489817,0.006280578672885895]},{"question_id":"month_length","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0004776120185852051,"mean_abs_dp":0.0002387790009379387,"max_abs_dlogit":0.021387577056884766,"oracle_top2_margin":0.9243375789374113,"near_tie":false,"probs":[0.9493640065193176,0.0258026160299778,0.013062977232038975,0.011770332232117653]},{"question_id":"shaded_weekend","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0019000768661499023,"mean_abs_dp":0.0019000545144081116,"max_abs_dlogit":0.01703500747680664,"oracle_top2_margin":0.7392615079879761,"near_tie":false,"probs":[0.12846921384334564,0.8715308308601379]},{"question_id":"shaded_count","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0007143914699554443,"mean_abs_dp":0.0003571966663002968,"max_abs_dlogit":0.012671709060668945,"oracle_top2_margin":0.181378573179245,"near_tie":false,"probs":[0.03513988107442856,0.033213093876838684,0.3757212162017822,0.5559257864952087]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0019000768661499023,"mean_abs_dp":0.00046826333605817387,"max_abs_dlogit":0.021762549877166748},{"id":"ho_i05","arm":"g448","variant":"base","tokens":833,"calls":14,"padded_tokens":896,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.879971167014446,"wall_seconds":1.8966328340175096,"finite":true,"index":29,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.347686767578125,"ref_absmax":48.98490905761719,"rel_max_abs_diff":0.0479267352485379,"min_pos_cos":0.9948125111793189,"min_pos_cos_index":587,"mean_pos_cos":0.9999507885090786,"positions":833,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"first_weekday","type":"choice","n_options":7,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0003172755241394043,"mean_abs_dp":9.063072502613068e-05,"max_abs_dlogit":0.01622319221496582,"oracle_top2_margin":0.8434231374412775,"near_tie":false,"probs":[0.02005254290997982,0.8739320635795593,0.015106946229934692,0.021111659705638885,0.018282871693372726,0.030020810663700104,0.021493185311555862]},{"question_id":"circled","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":9.966548532247543e-06,"mean_abs_dp":4.959292709827423e-06,"max_abs_dlogit":0.004548549652099609,"oracle_top2_margin":0.9703278709203005,"near_tie":false,"probs":[0.9790627956390381,0.008731276728212833,0.005634982138872147,0.006570847705006599]},{"question_id":"month_length","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.928775787353516e-05,"mean_abs_dp":5.367700941860676e-05,"max_abs_dlogit":0.008618354797363281,"oracle_top2_margin":0.9584462745115161,"near_tie":false,"probs":[0.9711371660232544,0.012762051075696945,0.008602489717304707,0.007498221006244421]},{"question_id":"shaded_weekend","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0002703666687011719,"mean_abs_dp":0.00027036480605602264,"max_abs_dlogit":0.005084991455078125,"oracle_top2_margin":0.904242217540741,"near_tie":false,"probs":[0.04760855808854103,0.952391505241394]},{"question_id":"shaded_count","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008838772773742676,"mean_abs_dp":0.0004419642500579357,"max_abs_dlogit":0.011832386255264282,"oracle_top2_margin":0.703109622001648,"near_tie":false,"probs":[0.03451697900891304,0.01996718905866146,0.8250739574432373,0.12044178694486618]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008838772773742676,"mean_abs_dp":0.00015131175695430664,"max_abs_dlogit":0.01622319221496582},{"id":"ho_i06","arm":"g256","variant":"base","tokens":628,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3415860829991288,"wall_seconds":1.3542118330078665,"finite":true,"index":30,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.891185760498047,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.059838035353617494,"min_pos_cos":0.9959093077201088,"min_pos_cos_index":397,"mean_pos_cos":0.999960559612027,"positions":628,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"third_digit","type":"choice","n_options":10,"argmax":9,"argmax_oracle":9,"argmax_equal":true,"max_abs_dp":0.001293480396270752,"mean_abs_dp":0.00025868690572679044,"max_abs_dlogit":0.016123056411743164,"oracle_top2_margin":0.821707833558321,"near_tie":false,"probs":[0.015095977112650871,0.006126738153398037,0.03629738837480545,0.03051779232919216,0.021892530843615532,0.007094890344887972,0.005725577939301729,0.00589499156922102,0.015074197202920914,0.8562798500061035]},{"question_id":"digit_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009009242057800293,"mean_abs_dp":0.0005383454263210297,"max_abs_dlogit":0.010759353637695312,"oracle_top2_margin":0.27862802147865295,"near_tie":false,"probs":[0.03816869482398033,0.20241671800613403,0.4801221191883087,0.1986413598060608,0.08065114170312881]},{"question_id":"has_seven","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00013855379074811935,"mean_abs_dp":0.0001385076902806759,"max_abs_dlogit":0.0138702392578125,"oracle_top2_margin":0.9801594745367765,"near_tie":false,"probs":[0.009781732223927975,0.9902182221412659]},{"question_id":"red_last","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.8190592527389526e-05,"mean_abs_dp":1.8185004591941833e-05,"max_abs_dlogit":0.0019114017486572266,"oracle_top2_margin":0.9475219752639532,"near_tie":false,"probs":[0.9737427830696106,0.02625717781484127]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.001293480396270752,"mean_abs_dp":0.0002943148199272783,"max_abs_dlogit":0.016123056411743164},{"id":"ho_i06","arm":"g448","variant":"base","tokens":760,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6100081249896903,"wall_seconds":1.6261184999893885,"finite":true,"index":31,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":0.8797869682312012,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.018208661883572064,"min_pos_cos":0.9988930439616879,"min_pos_cos_index":208,"mean_pos_cos":0.9999662138343889,"positions":760,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"third_digit","type":"choice","n_options":10,"argmax":9,"argmax_oracle":9,"argmax_equal":true,"max_abs_dp":0.008473306894302368,"mean_abs_dp":0.001705024321563542,"max_abs_dlogit":0.04421865940093994,"oracle_top2_margin":0.41684913635253906,"near_tie":false,"probs":[0.008047224022448063,0.004506577271968126,0.2753506600856781,0.010135216638445854,0.005591263063251972,0.004233275540173054,0.0025083180516958237,0.0033295955508947372,0.010936909355223179,0.6753609776496887]},{"question_id":"digit_count","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.001235663890838623,"mean_abs_dp":0.0004942573606967926,"max_abs_dlogit":0.008478760719299316,"oracle_top2_margin":0.577194482088089,"near_tie":false,"probs":[0.10626096278429031,0.6949970126152039,0.11970958113670349,0.04624678194522858,0.03278565779328346]},{"question_id":"has_seven","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.0265807658433914e-05,"mean_abs_dp":5.0256261602044106e-05,"max_abs_dlogit":0.0042877197265625,"oracle_top2_margin":0.9850624282844365,"near_tie":false,"probs":[0.0074185458943247795,0.9925814867019653]},{"question_id":"red_last","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00012637116014957428,"mean_abs_dp":0.0001263665035367012,"max_abs_dlogit":0.008639335632324219,"oracle_top2_margin":0.9580337218940258,"near_tie":false,"probs":[0.9788904786109924,0.02110948972404003]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.008473306894302368,"mean_abs_dp":0.001046040818389309,"max_abs_dlogit":0.04421865940093994},{"id":"ho_i07","arm":"g256","variant":"base","tokens":535,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.209707417001482,"wall_seconds":1.2222211670014076,"finite":true,"index":32,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":7.7289628982543945,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.15996380624724557,"min_pos_cos":0.8900131817819378,"min_pos_cos_index":94,"mean_pos_cos":0.9996865144889615,"positions":535,"positions_below":{"0.99":2,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":2},"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":6.448570638895035e-05,"mean_abs_dp":3.6450393963605165e-05,"max_abs_dlogit":0.006285429000854492,"oracle_top2_margin":0.9713920271024108,"near_tie":false,"probs":[0.009325029328465462,0.9806095957756042,0.007327680941671133,0.002737798960879445]},{"question_id":"paid_by","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00015115737915039062,"mean_abs_dp":0.00010076056544979413,"max_abs_dlogit":0.00592041015625,"oracle_top2_margin":0.9541426971554756,"near_tie":false,"probs":[0.9744500517845154,0.02056698314845562,0.004982904531061649]},{"question_id":"priciest","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008234977722167969,"mean_abs_dp":0.0004117293865419924,"max_abs_dlogit":0.0049440860748291016,"oracle_top2_margin":0.7043779790401459,"near_tie":false,"probs":[0.14030548930168152,0.0034494209103286266,0.8430893421173096,0.013155732303857803]},{"question_id":"change_shown","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0006443262100219727,"mean_abs_dp":0.000644296407699585,"max_abs_dlogit":0.015843629837036133,"oracle_top2_margin":0.9286240413784981,"near_tie":false,"probs":[0.03633224219083786,0.9636676907539368]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008234977722167969,"mean_abs_dp":0.0002602764333669956,"max_abs_dlogit":0.015843629837036133},{"id":"ho_i07","arm":"g448","variant":"base","tokens":667,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.476836292014923,"wall_seconds":1.4928488750010729,"finite":true,"index":33,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.5784826278686523,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.07406281980367242,"min_pos_cos":0.9776798395939983,"min_pos_cos_index":489,"mean_pos_cos":0.999891780798054,"positions":667,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.712841033935547e-05,"mean_abs_dp":4.905910464003682e-05,"max_abs_dlogit":0.015193462371826172,"oracle_top2_margin":0.9620198309421539,"near_tie":false,"probs":[0.01267833448946476,0.9748263359069824,0.008923189714550972,0.003572120564058423]},{"question_id":"paid_by","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00014197826385498047,"mean_abs_dp":9.468104690313339e-05,"max_abs_dlogit":0.004295825958251953,"oracle_top2_margin":0.9557577352970839,"near_tie":false,"probs":[0.9751165509223938,0.01911584846675396,0.005767569411545992]},{"question_id":"priciest","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0015791058540344238,"mean_abs_dp":0.0007895565358921885,"max_abs_dlogit":0.010650634765625,"oracle_top2_margin":0.7944317236542702,"near_tie":false,"probs":[0.09113100171089172,0.003228570334613323,0.8884333372116089,0.017207061871886253]},{"question_id":"change_shown","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0002822801470756531,"mean_abs_dp":0.00028225407004356384,"max_abs_dlogit":0.0038824081420898438,"oracle_top2_margin":0.9117454215884209,"near_tie":false,"probs":[0.04440957307815552,0.9555904865264893]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0015791058540344238,"mean_abs_dp":0.00032330875714810996,"max_abs_dlogit":0.015193462371826172},{"id":"ho_i08","arm":"g256","variant":"base","tokens":581,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3448243330058176,"wall_seconds":1.3582350000215229,"finite":true,"index":34,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.3537960052490234,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.02801912430895563,"min_pos_cos":0.9979086331913061,"min_pos_cos_index":475,"mean_pos_cos":0.9999801909924504,"positions":581,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_space","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0007407069206237793,"mean_abs_dp":0.0002962859347462654,"max_abs_dlogit":0.005896270275115967,"oracle_top2_margin":0.76869260892272,"near_tie":false,"probs":[0.062023162841796875,0.048660725355148315,0.8316782712936401,0.04347800463438034,0.014159808866679668]},{"question_id":"entrance","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001920461654663086,"mean_abs_dp":7.682330906391143e-05,"max_abs_dlogit":0.005518496036529541,"oracle_top2_margin":0.9482293706387281,"near_tie":false,"probs":[0.9599367380142212,0.008844105526804924,0.011446913704276085,0.008699139580130577,0.011073123663663864]},{"question_id":"wc_store_wall","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.7027827501297e-05,"mean_abs_dp":7.701851427555084e-05,"max_abs_dlogit":0.002847135066986084,"oracle_top2_margin":0.8891508653759956,"near_tie":false,"probs":[0.9446524381637573,0.05534753575921059]},{"question_id":"space_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0005881041288375854,"mean_abs_dp":0.0002589639276266098,"max_abs_dlogit":0.006864786148071289,"oracle_top2_margin":0.6871495172381401,"near_tie":false,"probs":[0.018477315083146095,0.1012430414557457,0.7895553708076477,0.07036285847425461,0.020361393690109253]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0007407069206237793,"mean_abs_dp":0.00019496487563147266,"max_abs_dlogit":0.006864786148071289},{"id":"ho_i08","arm":"g448","variant":"base","tokens":713,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6102055410156026,"wall_seconds":1.6264970419870224,"finite":true,"index":35,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.1049206983298063,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.04356493480430078,"min_pos_cos":0.996743181094126,"min_pos_cos_index":191,"mean_pos_cos":0.9999599558031741,"positions":713,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_space","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00011962652206420898,"mean_abs_dp":8.610114455223083e-05,"max_abs_dlogit":0.00690913200378418,"oracle_top2_margin":0.7769975811243057,"near_tie":false,"probs":[0.060006819665431976,0.04828376695513725,0.8372204899787903,0.04024982824921608,0.014239093288779259]},{"question_id":"entrance","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00027626752853393555,"mean_abs_dp":0.00011049937456846237,"max_abs_dlogit":0.005533695220947266,"oracle_top2_margin":0.946435839869082,"near_tie":false,"probs":[0.9580685496330261,0.009133038111031055,0.01189134269952774,0.008898665197193623,0.012008336372673512]},{"question_id":"wc_store_wall","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00019624829292297363,"mean_abs_dp":0.0001961737871170044,"max_abs_dlogit":0.009441852569580078,"oracle_top2_margin":0.9023991413414478,"near_tie":false,"probs":[0.9510034322738647,0.048996638506650925]},{"question_id":"space_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0006317570805549622,"mean_abs_dp":0.00025375522673130033,"max_abs_dlogit":0.008303403854370117,"oracle_top2_margin":0.7507094368338585,"near_tie":false,"probs":[0.017197759822010994,0.07116487622261047,0.822692334651947,0.07030973583459854,0.018635345622897148]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0006317570805549622,"mean_abs_dp":0.00015553684138199862,"max_abs_dlogit":0.009441852569580078},{"id":"ho_p01","arm":"g256","variant":"base","tokens":634,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3435257500095759,"wall_seconds":1.3560709159937687,"finite":true,"index":36,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":4.187688827514648,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.08667122690775941,"min_pos_cos":0.9883229252244917,"min_pos_cos_index":135,"mean_pos_cos":0.999945576859364,"positions":634,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"lamp_on","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00013526901602745056,"mean_abs_dp":0.00013522617518901825,"max_abs_dlogit":0.0031261444091796875,"oracle_top2_margin":0.8864576034247875,"near_tie":false,"probs":[0.9430935978889465,0.05690644681453705]},{"question_id":"time_of_day","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":3.49879264831543e-05,"mean_abs_dp":1.7483020201325417e-05,"max_abs_dlogit":0.0038100481033325195,"oracle_top2_margin":0.9142597317695618,"near_tie":false,"probs":[0.00642296951264143,0.007459139451384544,0.03590979054570198,0.9502080678939819]},{"question_id":"sea_behind","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001443624496459961,"mean_abs_dp":0.0001443624496459961,"max_abs_dlogit":0.0035381317138671875,"oracle_top2_margin":0.8596098273992538,"near_tie":false,"probs":[0.9296605587005615,0.07033945620059967]},{"question_id":"tower_colour","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.3942203521728516e-05,"mean_abs_dp":2.693571150302887e-05,"max_abs_dlogit":0.0010194778442382812,"oracle_top2_margin":0.9164357371628284,"near_tie":false,"probs":[0.013549230061471462,0.02824067696928978,0.9445960521697998,0.013613991439342499]},{"question_id":"remoteness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00023353099822998047,"mean_abs_dp":0.00015568484862645468,"max_abs_dlogit":0.0057179927825927734,"oracle_top2_margin":0.7929675057530403,"near_tie":false,"probs":[0.04184069484472275,0.08282273262739182,0.8753365874290466]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00023353099822998047,"mean_abs_dp":8.026044815778732e-05,"max_abs_dlogit":0.0057179927825927734},{"id":"ho_p01","arm":"g448","variant":"base","tokens":766,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6107339999871328,"wall_seconds":1.6273917090147734,"finite":true,"index":37,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":2.762266159057617,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.05716982365509356,"min_pos_cos":0.9890133799464225,"min_pos_cos_index":123,"mean_pos_cos":0.9999366568153724,"positions":766,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"lamp_on","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0002488754689693451,"mean_abs_dp":0.0002488624304533005,"max_abs_dlogit":0.002545595169067383,"oracle_top2_margin":0.8949422650039196,"near_tie":false,"probs":[0.9472222924232483,0.05277775228023529]},{"question_id":"time_of_day","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":6.121397018432617e-05,"mean_abs_dp":3.058509901165962e-05,"max_abs_dlogit":0.002817690372467041,"oracle_top2_margin":0.9366869516670704,"near_tie":false,"probs":[0.00627144193276763,0.0070229824632406235,0.02495509386062622,0.9617505073547363]},{"question_id":"sea_behind","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0002384185791015625,"mean_abs_dp":0.00023841485381126404,"max_abs_dlogit":0.0035486221313476562,"oracle_top2_margin":0.8574466854333878,"near_tie":false,"probs":[0.9284849166870117,0.07151506096124649]},{"question_id":"tower_colour","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.9266270101070404e-05,"mean_abs_dp":1.1027790606021881e-05,"max_abs_dlogit":0.0014095306396484375,"oracle_top2_margin":0.924212034791708,"near_tie":false,"probs":[0.013330970890820026,0.024474576115608215,0.948686957359314,0.013507517985999584]},{"question_id":"remoteness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0002251267433166504,"mean_abs_dp":0.0001500782867272695,"max_abs_dlogit":0.004874229431152344,"oracle_top2_margin":0.8101317957043648,"near_tie":false,"probs":[0.03947177156805992,0.07541834563016891,0.8851099014282227]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0002488754689693451,"mean_abs_dp":0.00010608273247877756,"max_abs_dlogit":0.004874229431152344},{"id":"ho_p02","arm":"g256","variant":"base","tokens":620,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3432023750210647,"wall_seconds":1.3545361660071649,"finite":true,"index":38,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":1.2797183990478516,"ref_absmax":51.6200065612793,"rel_max_abs_diff":0.024791132049327976,"min_pos_cos":0.9965837929715334,"min_pos_cos_index":189,"mean_pos_cos":0.9999707405699881,"positions":620,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"petal_colour","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0019177794456481934,"mean_abs_dp":0.0009588899556547403,"max_abs_dlogit":0.04926559329032898,"oracle_top2_margin":0.9628678802400827,"near_tie":false,"probs":[0.010003997012972832,0.010204903781414032,0.009330852888524532,0.9704602360725403]},{"question_id":"insect","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00021219253540039062,"mean_abs_dp":0.00021218881011009216,"max_abs_dlogit":0.028897523880004883,"oracle_top2_margin":0.937197607010603,"near_tie":false,"probs":[0.03118899092078209,0.9688109755516052]},{"question_id":"centre_colour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0005959272384643555,"mean_abs_dp":0.0002979678101837635,"max_abs_dlogit":0.02546793222427368,"oracle_top2_margin":0.9540995731949806,"near_tie":false,"probs":[0.008579184301197529,0.9670826196670532,0.013775321654975414,0.010562862269580364]},{"question_id":"setting","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0004239678382873535,"mean_abs_dp":0.00021201546769589186,"max_abs_dlogit":0.012824565172195435,"oracle_top2_margin":0.9648665515705943,"near_tie":false,"probs":[0.9745664596557617,0.007539974059909582,0.007623306941241026,0.010270291939377785]},{"question_id":"density","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0005537271499633789,"mean_abs_dp":0.00036916136741638184,"max_abs_dlogit":0.015019536018371582,"oracle_top2_margin":0.37225431203842163,"near_tie":false,"probs":[0.06715317815542221,0.27982097864151,0.6530258655548096]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0019177794456481934,"mean_abs_dp":0.00043572674450628897,"max_abs_dlogit":0.04926559329032898},{"id":"ho_p02","arm":"g448","variant":"base","tokens":752,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6109299999952782,"wall_seconds":1.6269084170053247,"finite":true,"index":39,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":7.3300697803497314,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.15170833387941582,"min_pos_cos":0.955415087593718,"min_pos_cos_index":324,"mean_pos_cos":0.9998578214922478,"positions":752,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"petal_colour","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":9.500980377197266e-05,"mean_abs_dp":4.750152584165335e-05,"max_abs_dlogit":0.00678369402885437,"oracle_top2_margin":0.9763394580222666,"near_tie":false,"probs":[0.005962590221315622,0.005907433107495308,0.005700533278286457,0.9824295043945312]},{"question_id":"insect","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.95125961303711e-05,"mean_abs_dp":7.950607687234879e-05,"max_abs_dlogit":0.00571441650390625,"oracle_top2_margin":0.9645282085984945,"near_tie":false,"probs":[0.017815392464399338,0.9821845889091492]},{"question_id":"centre_colour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.266187220811844e-05,"mean_abs_dp":1.438905019313097e-05,"max_abs_dlogit":0.006194591522216797,"oracle_top2_margin":0.9650700585916638,"near_tie":false,"probs":[0.006407671142369509,0.9755882024765015,0.010495670139789581,0.007508367300033569]},{"question_id":"setting","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001169443130493164,"mean_abs_dp":5.84461959078908e-05,"max_abs_dlogit":0.01027834415435791,"oracle_top2_margin":0.9797865096479654,"near_tie":false,"probs":[0.9859593510627747,0.004280447494238615,0.003751187352463603,0.006009064149111509]},{"question_id":"density","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0004940032958984375,"mean_abs_dp":0.0003293231129646301,"max_abs_dlogit":0.01087939739227295,"oracle_top2_margin":0.24045434594154358,"near_tie":false,"probs":[0.07810893654823303,0.34030091762542725,0.5815901756286621]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0004940032958984375,"mean_abs_dp":9.578403414172285e-05,"max_abs_dlogit":0.01087939739227295}],"note":"round 5 held-out: 30 records written after round 4 (fixtures/heldout), never used to select anything; fp16 S=64, the ship candidate","generated_at":"2026-10-03T05:14:59+00:00"},"int8mix_pf64":{"schema":"clef-flash-decoder-readout-gate/1","gate":"decoder alone on the Mac GPU (oracle ids, oracle fp32 image_embeds -> fp16, zero-padded to 1024 rows), the fp16 hidden read through the author's fp32 head","bundle":{"bundle":"<lane>/exports/bundles/clef_flash_decode_int8mix_pf64","name":"clef_flash_decode_int8mix_pf64","kind":"decision-backbone","compression":{"scheme":"int8mix","linear":"int8 per-block-32 symmetric_with_clipping (weight only)","excluded":"fp16: the embedding table, the GDN conv1d, every RMSNorm (RMSNorm, RMSNormPlusOne, RMSNormGated), SDPA, RoPE, the GDN A_log / dt_bias","excluded_types":["coreai_models.primitives.macos.rms_norm.RMSNorm","coreai_models.primitives.macos.rms_norm.RMSNormGated","coreai_models.primitives.macos.rms_norm.RMSNormPlusOne","coreai_models.primitives.macos.rope.RoPE","coreai_models.primitives.macos.sdpa.SDPA","torch.nn.modules.conv.Conv1d","torch.nn.modules.sparse.Embedding"],"int8_linear_modules":155,"fp16_layers":[0,1,2,3,4,5,6,7,8,9,10,11],"fp16_linear_modules":["model.layers.0.linear_attn.in_proj_qkv","model.layers.0.linear_attn.in_proj_z","model.layers.0.linear_attn.in_proj_b","model.layers.0.linear_attn.in_proj_a","model.layers.0.linear_attn.out_proj","model.layers.0.mlp.gate_proj","model.layers.0.mlp.up_proj","model.layers.0.mlp.down_proj","model.layers.1.linear_attn.in_proj_qkv","model.layers.1.linear_attn.in_proj_z","model.layers.1.linear_attn.in_proj_b","model.layers.1.linear_attn.in_proj_a","model.layers.1.linear_attn.out_proj","model.layers.1.mlp.gate_proj","model.layers.1.mlp.up_proj","model.layers.1.mlp.down_proj","model.layers.2.linear_attn.in_proj_qkv","model.layers.2.linear_attn.in_proj_z","model.layers.2.linear_attn.in_proj_b","model.layers.2.linear_attn.in_proj_a","model.layers.2.linear_attn.out_proj","model.layers.2.mlp.gate_proj","model.layers.2.mlp.up_proj","model.layers.2.mlp.down_proj","model.layers.3.self_attn.q_proj","model.layers.3.self_attn.k_proj","model.layers.3.self_attn.v_proj","model.layers.3.self_attn.o_proj","model.layers.3.mlp.gate_proj","model.layers.3.mlp.up_proj","model.layers.3.mlp.down_proj","model.layers.4.linear_attn.in_proj_qkv","model.layers.4.linear_attn.in_proj_z","model.layers.4.linear_attn.in_proj_b","model.layers.4.linear_attn.in_proj_a","model.layers.4.linear_attn.out_proj","model.layers.4.mlp.gate_proj","model.layers.4.mlp.up_proj","model.layers.4.mlp.down_proj","model.layers.5.linear_attn.in_proj_qkv","model.layers.5.linear_attn.in_proj_z","model.layers.5.linear_attn.in_proj_b","model.layers.5.linear_attn.in_proj_a","model.layers.5.linear_attn.out_proj","model.layers.5.mlp.gate_proj","model.layers.5.mlp.up_proj","model.layers.5.mlp.down_proj","model.layers.6.linear_attn.in_proj_qkv","model.layers.6.linear_attn.in_proj_z","model.layers.6.linear_attn.in_proj_b","model.layers.6.linear_attn.in_proj_a","model.layers.6.linear_attn.out_proj","model.layers.6.mlp.gate_proj","model.layers.6.mlp.up_proj","model.layers.6.mlp.down_proj","model.layers.7.self_attn.q_proj","model.layers.7.self_attn.k_proj","model.layers.7.self_attn.v_proj","model.layers.7.self_attn.o_proj","model.layers.7.mlp.gate_proj","model.layers.7.mlp.up_proj","model.layers.7.mlp.down_proj","model.layers.8.linear_attn.in_proj_qkv","model.layers.8.linear_attn.in_proj_z","model.layers.8.linear_attn.in_proj_b","model.layers.8.linear_attn.in_proj_a","model.layers.8.linear_attn.out_proj","model.layers.8.mlp.gate_proj","model.layers.8.mlp.up_proj","model.layers.8.mlp.down_proj","model.layers.9.linear_attn.in_proj_qkv","model.layers.9.linear_attn.in_proj_z","model.layers.9.linear_attn.in_proj_b","model.layers.9.linear_attn.in_proj_a","model.layers.9.linear_attn.out_proj","model.layers.9.mlp.gate_proj","model.layers.9.mlp.up_proj","model.layers.9.mlp.down_proj","model.layers.10.linear_attn.in_proj_qkv","model.layers.10.linear_attn.in_proj_z","model.layers.10.linear_attn.in_proj_b","model.layers.10.linear_attn.in_proj_a","model.layers.10.linear_attn.out_proj","model.layers.10.mlp.gate_proj","model.layers.10.mlp.up_proj","model.layers.10.mlp.down_proj","model.layers.11.self_attn.q_proj","model.layers.11.self_attn.k_proj","model.layers.11.self_attn.v_proj","model.layers.11.self_attn.o_proj","model.layers.11.mlp.gate_proj","model.layers.11.mlp.up_proj","model.layers.11.mlp.down_proj"],"head":"none in the graph (the untied lm_head table stays on the host)"},"prefill_chunk":64,"metadata_sha256":"1eafaedf47981c83c82bf4f609615e2a33f28093b8fab3d89f86dec0e520fc36","main_mlirb":{"bytes":11826520241,"sha256":"2c19a0e765a659e0f726b4b7099a62490fed6ab2e4db1246eb63fd70996a67b9"},"tokenizer_sha256":{"chat_template.jinja":"a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715","tokenizer.json":"06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523","tokenizer_config.json":"91a08f825d370d085d692e04cf117cdd7faad7bf18e996f1e6031b6dab03db72"},"aimodelc":{"path":"<lane>/exports/bundles_aotc/clef_flash_decode_int8mix_pf64.h16c.aimodelc","bytes":25678382366,"tree_sha256":"e80f419a8a9af23390bd540270510413234361ef5f164659878f393f0b12681a","files":{"main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist":"fc51b9aeb51a86c9fdb4061c6a6911def64e74a44008547d5e6e706d7423e4fe","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/original_model_0.mpsgraph":"5ac972e3a1bf9e3d91452395b2d3f874a3f2ce285694f7b50298522261226553","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin":"f022d2d941944adfcfe6351d5ee94fc72b618eac18bd99ddb3935dc72ec6676f","main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_1.mpsgraph":"a4aef1c29efc3b7193da1c26b06495284c01311d7b4b12fa47e2b14a02ee847b","main-h16c.mlirb":"9bce40915b0251124de6bbed06bfd64755fde5ba20a11c29bcdfdf5784468ab2","main.hash":"3e20a00e80a801ac947f24ee223029c8688932ebb85e1c53369e5e42bed4ef82","metadata.json":"0d1604a74a754f6d673e475d48ce70591ab407fd83cf5a7a379055a56fd17c11","stats.json":"0c48677700237e6e276aff023c19d2ca080c23ab0b7b8631d3b959c83015c0e5"}}},"chunk":64,"max_ctx":4096,"readout":"fresh zero states; a row of T ids runs as ceil(T / 64) calls of 'main' (static S = 64): call k gets ids[64k : 64k + 64] with position_ids 0..64k+63; the last call is padded with <|endoftext|> (248044) and the hidden rows of the padded positions are discarded (causal: they cannot reach a real position). The hidden rows [1, 64, 4096] of every call, concatenated and cut to T, are the backbone's final-norm last_hidden_state [T, 4096] the head reads.","descriptor":{"function":"main","inputs":{"input_ids":[[1,64],"int32"],"position_ids":[[1,-1],"int32"],"image_embeds":[[1024,4096],"float16"],"image_rc":[[1024,2],"int32"],"rope_shift_start":[[1],"int32"],"rope_shift_amount":[[1],"int32"]},"outputs":{"hidden":[[1,64,4096],"float16"]},"states":{"keyCache":[[8,1,4,-1,256],"float16"],"valueCache":[[8,1,4,-1,256],"float16"],"convState":[[24,1,8192,3],"float16"],"recState":[[24,1,32,128,128],"float16"]}},"descriptor_same_in_every_process":true,"contract":{"inputs":{"input_ids":[[1,64],"int32"],"position_ids":[[1,-1],"int32"],"image_embeds":[[1024,4096],"float16"],"image_rc":[[1024,2],"int32"],"rope_shift_start":[[1],"int32"],"rope_shift_amount":[[1],"int32"]},"outputs":{"hidden":[[1,64,4096],"float16"]},"states":{"keyCache":[[8,1,4,-1,256],"float16"],"valueCache":[[8,1,4,-1,256],"float16"],"convState":[[24,1,8192,3],"float16"],"recState":[[24,1,32,128,128],"float16"]}},"oracle":{"path":"<lane>/oracle/heldout/records_oracle.json","sha256":"85e4dda56890dff47a163bdd74c85291f5d3be0944403dd1f59ba2f5b9faaa3f","source":"Cloudflare/clef-flash@17f0b0ad64efb65d273590632833508766b2aae6","versions":{"python":"3.12.11","torch":"2.9.0","torchvision":"0.24.0","transformers":"5.17.0","pillow":"12.3.0","numpy":"2.5.3","huggingface_hub":"1.33.0"}},"subset":"all","runs_file":null,"script":{"path":"conversion/clef_flash/readout_gate.py","sha256":"ae633622829f02e83a202d522994559c77ab25a8e08dc759ec14435c983c50f0"},"skipped":[],"bar":{"max_abs_dp":0.02,"mean_of_run_mean_abs_dp":0.002,"near_tie_top2_margin":0.02,"argmax":"every question with an oracle top-2 margin above 0.02; near-ties listed apart","max_abs_dp_applies_to":"every option of every question, near-ties included","reset":"bit-equal hidden rows in every process","mean_of_run_mean_abs_dp_definition":"mean over runs of the mean |dp| over all (question, option) entries of the run"},"environment":{"versions":{"python":"3.11.13","numpy":"2.3.5","coreai-core":"1.0.0b2","coreai-torch":"0.4.1","coreai-models":"0.1.0","torch":"2.9.0","safetensors":"0.7.0"},"platform":"macOS-27.0-arm64-arm-64bit","macos_build":"26A428","chip":"Apple M4 Max","runtime":"coreai python runtime, AOT h16c GPU .aimodelc, SpecializationOptions.default(), no JIT","head":"the author's JointSchemaHead, fp32, CPU (parity_decoder_torch.AuthorHead)","gpu":"shared with other sessions, no _GPU_LOCK held (times are contended reference values)"},"other_gpu_processes":{"note":"process listing removed before publication (4 lines); it showed other sessions' GPU work at the time = the times are contended"},"processes":[{"shard":"shard_00","runs":["ho_t01/text/base","ho_t02/text/base","ho_t03/text/base","ho_t04/text/base","ho_t05/text/base","ho_t06/text/base","ho_t07/text/base","ho_t08/text/base","ho_t09/text/base","ho_t10/text/base","ho_j01/text/base","ho_j02/text/base","ho_j03/text/base","ho_j04/text/base","ho_j05/text/base","ho_j06/text/base","ho_j07/text/base","ho_j08/text/base","ho_l01/text/base","ho_l02/text/base","ho_i01/g256/base","ho_i01/g448/base","ho_i02/g256/base","ho_i02/g448/base","ho_i03/g256/base","ho_i03/g448/base","ho_i04/g256/base","ho_i04/g448/base","ho_i05/g256/base","ho_i05/g448/base","ho_i06/g256/base","ho_i06/g448/base","ho_i07/g256/base","ho_i07/g448/base","ho_i08/g256/base","ho_i08/g448/base","ho_p01/g256/base","ho_p01/g448/base","ho_p02/g256/base","ho_p02/g448/base"],"prompts":41,"pid":14970,"load_seconds":55.41733841699897,"load_split_seconds":{"model":40.10886199999368,"main":15.308476417005295},"function_names":["main"],"process_wall_seconds":150.77537112502614,"reset_check":{"run":["ho_t01","text","base"],"bit_equal":true,"hidden_max_abs_diff":0.0,"wall_seconds":1.6227208329946734},"contract_mismatch":[]}],"summary":{"runs":40,"expected_runs":40,"missing_runs":[],"questions":188,"argmax_equal":187,"questions_non_near_tie":186,"argmax_equal_non_near_tie":186,"near_tie_questions":2,"argmax_equal_near_tie":1,"max_abs_dp":0.017072200775146484,"max_abs_dp_non_near_tie":0.017072200775146484,"mean_of_run_mean_abs_dp":0.0006324522342802831,"mean_question_max_abs_dp":0.0011359780823732627,"max_abs_dlogit":0.13446760177612305,"min_pos_cos":0.8917932115996107,"positions":32093,"positions_below":{"0.99":14,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":5,"runs_with_positions_below_0.99":["ho_t03/text","ho_t10/text","ho_j06/text","ho_l01/text","ho_i01/g448","ho_i07/g256","ho_i07/g448","ho_p01/g256","ho_p01/g448","ho_p02/g448"],"hidden_max_abs_diff":8.259757280349731,"hidden_max_rel_diff":0.17094980713682184,"worst_run":{"id":"ho_t08","arm":"text","max_abs_dp":0.017072200775146484},"finite_all":true,"tokens":32093,"calls":521,"ms_per_call_median":134.40287500270642,"ms_per_call_warm_median":134.3717919953633},"by_source_arm":{"heldout_image/g256":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":31,"argmax_equal":31,"questions_non_near_tie":31,"argmax_equal_non_near_tie":31,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.005230963230133057,"max_abs_dp_non_near_tie":0.005230963230133057,"mean_of_run_mean_abs_dp":0.00046352947188134125,"mean_question_max_abs_dp":0.0008976325963533693,"max_abs_dlogit":0.0676572322845459,"min_pos_cos":0.8917932115996107,"positions":4522,"positions_below":{"0.99":2,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":2,"runs_with_positions_below_0.99":["ho_i07/g256"],"hidden_max_abs_diff":7.5727128982543945,"hidden_max_rel_diff":0.15672995132321985,"worst_run":{"id":"ho_i04","arm":"g256","max_abs_dp":0.005230963230133057},"finite_all":true,"tokens":4522,"calls":74,"ms_per_call_median":134.41916700685397,"ms_per_call_warm_median":134.41916700685397},"heldout_image/g448":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":31,"argmax_equal":31,"questions_non_near_tie":31,"argmax_equal_non_near_tie":31,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.003522336483001709,"max_abs_dp_non_near_tie":0.003522336483001709,"mean_of_run_mean_abs_dp":0.00039062325763813304,"mean_question_max_abs_dp":0.0007387449469176993,"max_abs_dlogit":0.057102084159851074,"min_pos_cos":0.9785905534628597,"positions":5578,"positions_below":{"0.99":3,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1,"runs_with_positions_below_0.99":["ho_i01/g448","ho_i07/g448"],"hidden_max_abs_diff":3.775679588317871,"hidden_max_rel_diff":0.07814414825105435,"worst_run":{"id":"ho_i04","arm":"g448","max_abs_dp":0.003522336483001709},"finite_all":true,"tokens":5578,"calls":91,"ms_per_call_median":134.4553749950137,"ms_per_call_warm_median":134.4553749950137},"heldout_json/text":{"runs":8,"expected_runs":8,"missing_runs":[],"questions":36,"argmax_equal":36,"questions_non_near_tie":35,"argmax_equal_non_near_tie":35,"near_tie_questions":1,"argmax_equal_near_tie":1,"max_abs_dp":0.010159045457839966,"max_abs_dp_non_near_tie":0.010159045457839966,"mean_of_run_mean_abs_dp":0.0009221034762440151,"mean_question_max_abs_dp":0.0014620315826808412,"max_abs_dlogit":0.05870091915130615,"min_pos_cos":0.979391940972539,"positions":5621,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_j06/text"],"hidden_max_abs_diff":2.8887195587158203,"hidden_max_rel_diff":0.05978699308872733,"worst_run":{"id":"ho_j05","arm":"text","max_abs_dp":0.010159045457839966},"finite_all":true,"tokens":5621,"calls":93,"ms_per_call_median":134.16275000781752,"ms_per_call_warm_median":134.16275000781752},"heldout_photo/g256":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":10,"argmax_equal":10,"questions_non_near_tie":10,"argmax_equal_non_near_tie":10,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.0017777085304260254,"max_abs_dp_non_near_tie":0.0017777085304260254,"mean_of_run_mean_abs_dp":0.0003734177955444537,"mean_question_max_abs_dp":0.0005927182734012604,"max_abs_dlogit":0.02589467167854309,"min_pos_cos":0.9698986244816614,"positions":1254,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_p01/g256"],"hidden_max_abs_diff":6.875188827514648,"hidden_max_rel_diff":0.14229353599246866,"worst_run":{"id":"ho_p01","arm":"g256","max_abs_dp":0.0017777085304260254},"finite_all":true,"tokens":1254,"calls":20,"ms_per_call_median":134.45789548859466,"ms_per_call_warm_median":134.45789548859466},"heldout_photo/g448":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":10,"argmax_equal":10,"questions_non_near_tie":10,"argmax_equal_non_near_tie":10,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.002624690532684326,"max_abs_dp_non_near_tie":0.002624690532684326,"mean_of_run_mean_abs_dp":0.00040381282029271714,"mean_question_max_abs_dp":0.00062791146337986,"max_abs_dlogit":0.03529810905456543,"min_pos_cos":0.9471616175950758,"positions":1518,"positions_below":{"0.99":3,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":2,"runs_with_positions_below_0.99":["ho_p01/g448","ho_p02/g448"],"hidden_max_abs_diff":8.259757280349731,"hidden_max_rel_diff":0.17094980713682184,"worst_run":{"id":"ho_p01","arm":"g448","max_abs_dp":0.002624690532684326},"finite_all":true,"tokens":1518,"calls":24,"ms_per_call_median":134.39204149472062,"ms_per_call_warm_median":134.39204149472062},"heldout_text/text":{"runs":10,"expected_runs":10,"missing_runs":[],"questions":56,"argmax_equal":55,"questions_non_near_tie":55,"argmax_equal_non_near_tie":55,"near_tie_questions":1,"argmax_equal_near_tie":0,"max_abs_dp":0.017072200775146484,"max_abs_dp_non_near_tie":0.017072200775146484,"mean_of_run_mean_abs_dp":0.0008670288450812639,"mean_question_max_abs_dp":0.0015657394625512616,"max_abs_dlogit":0.13446760177612305,"min_pos_cos":0.9753598818329652,"positions":8019,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_t03/text","ho_t10/text"],"hidden_max_abs_diff":5.995674133300781,"hidden_max_rel_diff":0.12346000971982567,"worst_run":{"id":"ho_t08","arm":"text","max_abs_dp":0.017072200775146484},"finite_all":true,"tokens":8019,"calls":131,"ms_per_call_median":134.33470801101066,"ms_per_call_warm_median":134.32291700155474},"heldout_text_long/text":{"runs":2,"expected_runs":2,"missing_runs":[],"questions":14,"argmax_equal":14,"questions_non_near_tie":14,"argmax_equal_non_near_tie":14,"near_tie_questions":0,"argmax_equal_near_tie":0,"max_abs_dp":0.004114687442779541,"max_abs_dp_non_near_tie":0.004114687442779541,"mean_of_run_mean_abs_dp":0.00043164502130821347,"mean_question_max_abs_dp":0.0007368093855412943,"max_abs_dlogit":0.05040156841278076,"min_pos_cos":0.9800416625965211,"positions":5581,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0,"runs_with_positions_below_0.99":["ho_l01/text"],"hidden_max_abs_diff":3.463749885559082,"hidden_max_rel_diff":0.06805122106431213,"worst_run":{"id":"ho_l01","arm":"text","max_abs_dp":0.004114687442779541},"finite_all":true,"tokens":5581,"calls":88,"ms_per_call_median":135.00618749822024,"ms_per_call_warm_median":135.00618749822024}},"near_ties":[{"id":"ho_t07","arm":"text","question_id":"questions_asked","oracle_top2_margin":0.011684805154800415,"argmax_equal":false,"max_abs_dp":0.01224970817565918,"probs":[0.010005100630223751,0.3979370594024658,0.40590396523475647,0.18615387380123138],"probs_oracle":[0.009856987744569778,0.410186767578125,0.3985019624233246,0.18145422637462616]},{"id":"ho_j01","arm":"text","question_id":"late_start","oracle_top2_margin":0.0010575354099273682,"argmax_equal":true,"max_abs_dp":0.001629173755645752,"probs":[0.5021579265594482,0.49784207344055176],"probs_oracle":[0.5005287528038025,0.4994712173938751]}],"checks":{"all_runs":true,"argmax_non_near_tie":true,"max_abs_dp":true,"mean_of_run_mean_abs_dp":true,"finite":true,"reset_bit_equal_all_processes":true},"result":"PASS","red_arm":null,"timing":{"contended":true,"S":64,"load_seconds":{"first_process":55.41733841699897,"median":55.41733841699897,"all":[55.41733841699897]},"ms_per_call_warm_runs":{"n":509,"median":134.3717919953633,"p10":133.77579218358733,"p90":135.2056248113513},"ms_per_call_all_runs":{"n":521,"median":134.40287500270642,"p10":133.78391601145267,"p90":135.24420899921097},"first_call_of_process_ms":[21194.895499997074],"ms_per_token_warm_runs":{"n":39,"median":2.186575867142295,"p10":2.110669146602898,"p90":2.266466214412303},"run_decode_seconds_warm_runs":{"n":39,"median":1.6102373330213595,"p10":1.2099818253831471,"p90":1.8821196497941854},"run_wall_seconds_warm_runs":{"n":39,"median":1.6230265000194777,"p10":1.2250821085937786,"p90":1.8941294336109422},"by_source_arm_warm":{"heldout_image/g256":{"runs":8,"median_tokens":555.5,"median_calls":9.0,"ms_per_call_median":134.41916700685397,"run_decode_seconds_median":1.2127869165124139},"heldout_image/g448":{"runs":8,"median_tokens":687.5,"median_calls":11.0,"ms_per_call_median":134.4553749950137,"run_decode_seconds_median":1.4810275829804596},"heldout_json/text":{"runs":8,"median_tokens":772.0,"median_calls":13.0,"ms_per_call_median":134.16275000781752,"run_decode_seconds_median":1.7436351454962278},"heldout_photo/g256":{"runs":2,"median_tokens":627.0,"median_calls":10.0,"ms_per_call_median":134.45789548859466,"run_decode_seconds_median":1.3439299375022529},"heldout_photo/g448":{"runs":2,"median_tokens":759.0,"median_calls":12.0,"ms_per_call_median":134.39204149472062,"run_decode_seconds_median":1.6127453960070852},"heldout_text/text":{"runs":9,"median_tokens":800.0,"median_calls":13.0,"ms_per_call_median":134.32291700155474,"run_decode_seconds_median":1.7462713749846444},"heldout_text_long/text":{"runs":2,"median_tokens":2790.5,"median_calls":44.0,"ms_per_call_median":135.00618749822024,"run_decode_seconds_median":5.944000207993668}}},"seconds":{"gpu_processes":150.8066514999955,"scoring":19.040083415980916,"total":170.00592204200802},"runs":[{"id":"ho_t01","arm":"text","variant":"base","tokens":708,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":22.68046995799523,"wall_seconds":22.69623520798632,"finite":true,"index":0,"shard":"shard_00","first_in_process":true,"source":"heldout_text","hidden":{"max_abs_diff":2.2350502014160156,"ref_absmax":50.3668098449707,"rel_max_abs_diff":0.04437545693871642,"min_pos_cos":0.9965460082535906,"min_pos_cos_index":486,"mean_pos_cos":0.9998297091730876,"positions":708,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"leave_days","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00036579370498657227,"mean_abs_dp":0.00018289638683199883,"max_abs_dlogit":0.007332324981689453,"oracle_top2_margin":0.9403126761317253,"near_tie":false,"probs":[0.012090323492884636,0.014469588175415993,0.9566135406494141,0.016826540231704712]},{"question_id":"cover_arranged","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00019949674606323242,"mean_abs_dp":0.00019947439432144165,"max_abs_dlogit":0.00966954231262207,"oracle_top2_margin":0.8687373250722885,"near_tie":false,"probs":[0.9345681667327881,0.06543189287185669]},{"question_id":"days_left_after","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0039015263319015503,"mean_abs_dp":0.0019507892429828644,"max_abs_dlogit":0.06364107131958008,"oracle_top2_margin":0.6971522346138954,"near_tie":false,"probs":[0.056382667273283005,0.06409162282943726,0.08762792497873306,0.7918977737426758]},{"question_id":"reason","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.0444393157958984e-05,"mean_abs_dp":1.256866380572319e-05,"max_abs_dlogit":0.01162862777709961,"oracle_top2_margin":0.9823572798632085,"near_tie":false,"probs":[0.9869483709335327,0.0045608594082295895,0.003948370926082134,0.004542484879493713]},{"question_id":"ease_of_approval","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0014308691024780273,"mean_abs_dp":0.0009539326032002767,"max_abs_dlogit":0.00616908073425293,"oracle_top2_margin":0.680185854434967,"near_tie":false,"probs":[0.07311240583658218,0.12442474812269211,0.8024628758430481]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0039015263319015503,"mean_abs_dp":0.0006968096336897681,"max_abs_dlogit":0.06364107131958008},{"id":"ho_t02","arm":"text","variant":"base","tokens":746,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6133108749927487,"wall_seconds":1.627663583989488,"finite":true,"index":1,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.5486106872558594,"ref_absmax":56.83818817138672,"rel_max_abs_diff":0.02724595447318385,"min_pos_cos":0.9911401034894088,"min_pos_cos_index":470,"mean_pos_cos":0.9998227013773627,"positions":746,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"in_warranty","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.003685206174850464,"mean_abs_dp":0.00368519127368927,"max_abs_dlogit":0.009039640426635742,"oracle_top2_margin":0.1861058473587036,"near_tie":false,"probs":[0.40326187014579773,0.5967381000518799]},{"question_id":"faulty_part","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00010001659393310547,"mean_abs_dp":4.000086337327957e-05,"max_abs_dlogit":0.0023016929626464844,"oracle_top2_margin":0.9428951684385538,"near_tie":false,"probs":[0.9560220837593079,0.009598182514309883,0.013006331399083138,0.009141584858298302,0.012231831438839436]},{"question_id":"possible_damage","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0003573298454284668,"mean_abs_dp":0.00035730935633182526,"max_abs_dlogit":0.00542140007019043,"oracle_top2_margin":0.9125267155468464,"near_tie":false,"probs":[0.9559060335159302,0.044093936681747437]},{"question_id":"remedy","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0001442432403564453,"mean_abs_dp":7.212464697659016e-05,"max_abs_dlogit":0.006173968315124512,"oracle_top2_margin":0.9695126404985785,"near_tie":false,"probs":[0.006703763268887997,0.9787361025810242,0.005137871019542217,0.009422309696674347]},{"question_id":"claim_strength","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0015513747930526733,"mean_abs_dp":0.0010342498620351155,"max_abs_dlogit":0.017431020736694336,"oracle_top2_margin":0.4374786615371704,"near_tie":false,"probs":[0.2052273005247116,0.6412692666053772,0.1535034477710724]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.003685206174850464,"mean_abs_dp":0.0007297658594325185,"max_abs_dlogit":0.017431020736694336},{"id":"ho_t03","arm":"text","variant":"base","tokens":842,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.8801108750049025,"wall_seconds":1.891070125013357,"finite":true,"index":2,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":5.995674133300781,"ref_absmax":48.56369400024414,"rel_max_abs_diff":0.12346000971982567,"min_pos_cos":0.9812210310313069,"min_pos_cos_index":359,"mean_pos_cos":0.999769204515598,"positions":842,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"asks_about_pet","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0005077123641967773,"mean_abs_dp":0.0005076825618743896,"max_abs_dlogit":0.024872541427612305,"oracle_top2_margin":0.9131554178893566,"near_tie":false,"probs":[0.9560700058937073,0.04392995312809944]},{"question_id":"bedrooms","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00032335519790649414,"mean_abs_dp":0.0001616673544049263,"max_abs_dlogit":0.014631152153015137,"oracle_top2_margin":0.9704894721508026,"near_tie":false,"probs":[0.0087018683552742,0.9787455797195435,0.006463830824941397,0.0060886843129992485]},{"question_id":"needs_parking","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.2860938012599945e-05,"mean_abs_dp":5.2805524319410324e-05,"max_abs_dlogit":0.004441022872924805,"oracle_top2_margin":0.9770000539720058,"near_tie":false,"probs":[0.011447143740952015,0.9885528087615967]},{"question_id":"viewing_time","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008708834648132324,"mean_abs_dp":0.00043540773913264275,"max_abs_dlogit":0.022212892770767212,"oracle_top2_margin":0.8937774673104286,"near_tie":false,"probs":[0.028677647933363914,0.023284098133444786,0.9239193201065063,0.02411898784339428]},{"question_id":"other_question","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00019949674606323242,"mean_abs_dp":9.976045112125576e-05,"max_abs_dlogit":0.05530405044555664,"oracle_top2_margin":0.9933792083756998,"near_tie":false,"probs":[0.0018660404020920396,0.0012422521831467748,0.0018845153972506523,0.9950071573257446]},{"question_id":"match","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.001867055892944336,"mean_abs_dp":0.001070437952876091,"max_abs_dlogit":0.012884557247161865,"oracle_top2_margin":0.204206645488739,"near_tie":false,"probs":[0.08816158771514893,0.31525781750679016,0.5231499671936035,0.0734306201338768]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.001867055892944336,"mean_abs_dp":0.00040950350812636317,"max_abs_dlogit":0.05530405044555664},{"id":"ho_t04","arm":"text","variant":"base","tokens":658,"calls":11,"padded_tokens":704,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.477503208006965,"wall_seconds":1.488060540985316,"finite":true,"index":3,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.947977066040039,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.04031671784578115,"min_pos_cos":0.996725298864791,"min_pos_cos_index":269,"mean_pos_cos":0.9998609835411352,"positions":658,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_problem","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0001241154968738556,"mean_abs_dp":7.751584053039551e-05,"max_abs_dlogit":0.01104736328125,"oracle_top2_margin":0.9411561395972967,"near_tie":false,"probs":[0.012393337674438953,0.9576665163040161,0.016285469755530357,0.006641518324613571,0.007013136055320501]},{"question_id":"would_return","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00018680095672607422,"mean_abs_dp":0.00018678326159715652,"max_abs_dlogit":0.12230825424194336,"oracle_top2_margin":0.9438176397234201,"near_tie":false,"probs":[0.9717220067977905,0.028277933597564697]},{"question_id":"food_left","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.720183253288269e-06,"mean_abs_dp":5.691312253475189e-06,"max_abs_dlogit":0.13446760177612305,"oracle_top2_margin":0.9789652321487665,"near_tie":false,"probs":[0.010511688888072968,0.9894883036613464]},{"question_id":"tone","type":"score","n_options":7,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0026790499687194824,"mean_abs_dp":0.0007654396550995963,"max_abs_dlogit":0.02800893783569336,"oracle_top2_margin":0.6996366679668427,"near_tie":false,"probs":[0.02181822992861271,0.017610834911465645,0.019554322585463524,0.8029600381851196,0.09833379834890366,0.020409753546118736,0.019312992691993713]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0026790499687194824,"mean_abs_dp":0.00038316287100315094,"max_abs_dlogit":0.13446760177612305},{"id":"ho_t05","arm":"text","variant":"base","tokens":727,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6121679170173593,"wall_seconds":1.6230265000194777,"finite":true,"index":4,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.6176681518554688,"ref_absmax":51.32048797607422,"rel_max_abs_diff":0.03152090355433937,"min_pos_cos":0.9936225961298197,"min_pos_cos_index":273,"mean_pos_cos":0.9998635139047376,"positions":727,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"defect_kind","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0004647970199584961,"mean_abs_dp":0.00023238570429384708,"max_abs_dlogit":0.009520351886749268,"oracle_top2_margin":0.8769160024821758,"near_tie":false,"probs":[0.9172247052192688,0.029014887288212776,0.04094349965453148,0.012816832400858402]},{"question_id":"regression","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00018844380974769592,"mean_abs_dp":0.00018839724361896515,"max_abs_dlogit":0.006299018859863281,"oracle_top2_margin":0.8895891383290291,"near_tie":false,"probs":[0.9449829459190369,0.05501701310276985]},{"question_id":"loses_work","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00011547282338142395,"mean_abs_dp":0.0001154635101556778,"max_abs_dlogit":0.0016918182373046875,"oracle_top2_margin":0.9126163460314274,"near_tie":false,"probs":[0.9561927318572998,0.04380731284618378]},{"question_id":"small_notebooks","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.7983059883117676e-05,"mean_abs_dp":3.797560930252075e-05,"max_abs_dlogit":0.0051000118255615234,"oracle_top2_margin":0.9837353955954313,"near_tie":false,"probs":[0.008094342425465584,0.991905689239502]},{"question_id":"priority","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0003857612609863281,"mean_abs_dp":0.00027765892446041107,"max_abs_dlogit":0.005248665809631348,"oracle_top2_margin":0.47177982330322266,"near_tie":false,"probs":[0.06422583758831024,0.14560064673423767,0.6309352517127991,0.15923818945884705]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0004647970199584961,"mean_abs_dp":0.00019456080294081142,"max_abs_dlogit":0.009520351886749268},{"id":"ho_t06","arm":"text","variant":"base","tokens":721,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6122537920018658,"wall_seconds":1.6231282080116216,"finite":true,"index":5,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.332087516784668,"ref_absmax":49.36119079589844,"rel_max_abs_diff":0.04724536582651583,"min_pos_cos":0.9960499189981269,"min_pos_cos_index":390,"mean_pos_cos":0.9998494701441291,"positions":721,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"problem","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0003814101219177246,"mean_abs_dp":0.00015255156904459,"max_abs_dlogit":0.006544262170791626,"oracle_top2_margin":0.9249128699302673,"near_tie":false,"probs":[0.0114413071423769,0.017438553273677826,0.9429286122322083,0.014659893698990345,0.0135316401720047]},{"question_id":"was_home","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0007501617074012756,"mean_abs_dp":0.0007501430809497833,"max_abs_dlogit":0.005485057830810547,"oracle_top2_margin":0.7574861496686935,"near_tie":false,"probs":[0.8779929280281067,0.12200706452131271]},{"question_id":"outcome_wanted","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00039970874786376953,"mean_abs_dp":0.00019984738901257515,"max_abs_dlogit":0.008756637573242188,"oracle_top2_margin":0.9236043281853199,"near_tie":false,"probs":[0.0173967108130455,0.018157850950956345,0.020675869658589363,0.9437695741653442]},{"question_id":"time_sensitive","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.284480094909668e-05,"mean_abs_dp":1.2829899787902832e-05,"max_abs_dlogit":0.00022649765014648438,"oracle_top2_margin":0.8349910154938698,"near_tie":false,"probs":[0.9175083041191101,0.08249162882566452]},{"question_id":"mood","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008625984191894531,"mean_abs_dp":0.0007888469845056534,"max_abs_dlogit":0.040053606033325195,"oracle_top2_margin":0.3556882441043854,"near_tie":false,"probs":[0.06607967615127563,0.19698737561702728,0.5525281429290771,0.18440479040145874]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008625984191894531,"mean_abs_dp":0.0003672636059277198,"max_abs_dlogit":0.040053606033325195},{"id":"ho_t07","arm":"text","variant":"base","tokens":1086,"calls":17,"padded_tokens":1088,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":2.283700207975926,"wall_seconds":2.2941403339791577,"finite":true,"index":6,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.2400989532470703,"ref_absmax":50.011722564697266,"rel_max_abs_diff":0.0447914776450498,"min_pos_cos":0.9938092224574597,"min_pos_cos_index":1018,"mean_pos_cos":0.9998149450285576,"positions":1086,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"factor","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00536954402923584,"mean_abs_dp":0.0021478530019521714,"max_abs_dlogit":0.04826474189758301,"oracle_top2_margin":0.7617465928196907,"near_tie":false,"probs":[0.05526839941740036,0.07414910197257996,0.8462758660316467,0.014023618772625923,0.0102829085662961]},{"question_id":"oats_total","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00011815689504146576,"mean_abs_dp":6.501306779682636e-05,"max_abs_dlogit":0.028052330017089844,"oracle_top2_margin":0.848384864628315,"near_tie":false,"probs":[0.06384405493736267,0.0181131549179554,0.005921449512243271,0.9121214151382446]},{"question_id":"syrup_total","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0019794106483459473,"mean_abs_dp":0.0009896840201690793,"max_abs_dlogit":0.04405730962753296,"oracle_top2_margin":0.9178646951913834,"near_tie":false,"probs":[0.9496403336524963,0.03519235551357269,0.005554842762649059,0.00961248204112053]},{"question_id":"oats_one_weighing","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.004387855529785156,"mean_abs_dp":0.004387810826301575,"max_abs_dlogit":0.02466517686843872,"oracle_top2_margin":0.4905860126018524,"near_tie":false,"probs":[0.2503191828727722,0.7496808171272278]},{"question_id":"asks_swap","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.027387619018555e-05,"mean_abs_dp":7.025990635156631e-05,"max_abs_dlogit":0.014817476272583008,"oracle_top2_margin":0.9517021421343088,"near_tie":false,"probs":[0.9757807850837708,0.02421916276216507]},{"question_id":"oven_temperature","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001754164695739746,"mean_abs_dp":8.907378651201725e-05,"max_abs_dlogit":0.012944459915161133,"oracle_top2_margin":0.9683727771043777,"near_tie":false,"probs":[0.007921160198748112,0.007696329150348902,0.976291298866272,0.008091207593679428]},{"question_id":"questions_asked","type":"choice","n_options":4,"argmax":2,"argmax_oracle":1,"argmax_equal":false,"max_abs_dp":0.01224970817565918,"mean_abs_dp":0.006124867824837565,"max_abs_dlogit":0.033066049218177795,"oracle_top2_margin":0.011684805154800415,"near_tie":true,"probs":[0.010005100630223751,0.3979370594024658,0.40590396523475647,0.18615387380123138]},{"question_id":"difficulty","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0012232735753059387,"mean_abs_dp":0.0006116572767496109,"max_abs_dlogit":0.013055682182312012,"oracle_top2_margin":0.6591825112700462,"near_tie":false,"probs":[0.038488347083330154,0.08373619616031647,0.7675687074661255,0.11020664870738983]}],"argmax_all_equal":false,"argmax_equal_non_near_tie":true,"max_abs_dp":0.01224970817565918,"mean_abs_dp":0.0017647100130802598,"max_abs_dlogit":0.04826474189758301},{"id":"ho_t08","arm":"text","variant":"base","tokens":864,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.8820425829908345,"wall_seconds":1.8928625420085154,"finite":true,"index":7,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.3855628967285156,"ref_absmax":54.1589241027832,"rel_max_abs_diff":0.044047457298102465,"min_pos_cos":0.9945570453391324,"min_pos_cos_index":719,"mean_pos_cos":0.9997686054853682,"positions":864,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"policy_section","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00011511705815792084,"mean_abs_dp":6.532920524477958e-05,"max_abs_dlogit":0.024513959884643555,"oracle_top2_margin":0.9393905811011791,"near_tie":false,"probs":[0.016583262011408806,0.013893086463212967,0.9561628699302673,0.007561480160802603,0.005799334961920977]},{"question_id":"costs_total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00014835596084594727,"mean_abs_dp":7.4133335147053e-05,"max_abs_dlogit":0.015368223190307617,"oracle_top2_margin":0.9614982744678855,"near_tie":false,"probs":[0.006002620328217745,0.9760760068893433,0.01483217254281044,0.0030891329515725374]},{"question_id":"payout","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.017072200775146484,"mean_abs_dp":0.008536095730960369,"max_abs_dlogit":0.047601282596588135,"oracle_top2_margin":0.5377697646617889,"near_tie":false,"probs":[0.6742246747016907,0.162287175655365,0.09851274639368057,0.06497541815042496]},{"question_id":"airline_paid","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.000574648380279541,"mean_abs_dp":0.000574648380279541,"max_abs_dlogit":0.011066436767578125,"oracle_top2_margin":0.901993878185749,"near_tie":false,"probs":[0.048428408801555634,0.9515715837478638]},{"question_id":"baggage_lost","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.048513412475586e-05,"mean_abs_dp":5.044834688305855e-05,"max_abs_dlogit":0.010363101959228516,"oracle_top2_margin":0.9824593458324671,"near_tie":false,"probs":[0.008719895966351032,0.9912801384925842]},{"question_id":"clarity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0022213011980056763,"mean_abs_dp":0.0014808525641759236,"max_abs_dlogit":0.014247357845306396,"oracle_top2_margin":0.43105728924274445,"near_tie":false,"probs":[0.199692964553833,0.6264912486076355,0.1738157868385315]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.017072200775146484,"mean_abs_dp":0.002023015671875328,"max_abs_dlogit":0.047601282596588135},{"id":"ho_t09","arm":"text","variant":"base","tokens":867,"calls":14,"padded_tokens":896,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.8803122079989407,"wall_seconds":1.8914602499862667,"finite":true,"index":8,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":2.0255537033081055,"ref_absmax":50.86228942871094,"rel_max_abs_diff":0.03982427307262958,"min_pos_cos":0.9903623078062876,"min_pos_cos_index":791,"mean_pos_cos":0.9998444143465564,"positions":867,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"bartholomew_chemistry","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0029706060886383057,"mean_abs_dp":0.0019804040590922036,"max_abs_dlogit":0.01642298698425293,"oracle_top2_margin":0.23581808805465698,"near_tie":false,"probs":[0.5062416195869446,0.27258434891700745,0.22117403149604797]},{"question_id":"ingrid_laptop","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00012862682342529297,"mean_abs_dp":0.00012860260903835297,"max_abs_dlogit":0.016570091247558594,"oracle_top2_margin":0.9835674799978733,"near_tie":false,"probs":[0.00834483653306961,0.9916551113128662]},{"question_id":"within_limit","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0014497637748718262,"mean_abs_dp":0.0014497339725494385,"max_abs_dlogit":0.007439374923706055,"oracle_top2_margin":0.5680407285690308,"near_tie":false,"probs":[0.7825706601142883,0.21742939949035645]},{"question_id":"moved_lesson","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":6.365776062011719e-05,"mean_abs_dp":3.366515738889575e-05,"max_abs_dlogit":0.012774944305419922,"oracle_top2_margin":0.9909667111933231,"near_tie":false,"probs":[0.0026688934303820133,0.9935756325721741,0.0013888144167140126,0.002366618951782584]},{"question_id":"maths_length","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0004831552505493164,"mean_abs_dp":0.0002624919870868325,"max_abs_dlogit":0.012436747550964355,"oracle_top2_margin":0.835874080657959,"near_tie":false,"probs":[0.04618818312883377,0.007255020085722208,0.05496133118867874,0.8915954828262329]},{"question_id":"planning_effort","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.007371366024017334,"mean_abs_dp":0.004914239048957825,"max_abs_dlogit":0.016313493251800537,"oracle_top2_margin":0.03400421142578125,"near_tie":false,"probs":[0.08248293399810791,0.46839165687561035,0.44912540912628174]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.007371366024017334,"mean_abs_dp":0.0013902906147349211,"max_abs_dlogit":0.016570091247558594},{"id":"ho_t10","arm":"text","variant":"base","tokens":800,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7462713749846444,"wall_seconds":1.7570916670083534,"finite":true,"index":9,"shard":"shard_00","first_in_process":false,"source":"heldout_text","hidden":{"max_abs_diff":1.7937278747558594,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.037124196608905195,"min_pos_cos":0.9753598818329652,"min_pos_cos_index":445,"mean_pos_cos":0.9998081940712459,"positions":800,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"most_wanted","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011485815048217773,"mean_abs_dp":4.8093730583786964e-05,"max_abs_dlogit":0.006087303161621094,"oracle_top2_margin":0.9730223435908556,"near_tie":false,"probs":[0.00504136411473155,0.9819965362548828,0.00877421349287033,0.0021047370973974466,0.0020831443835049868]},{"question_id":"mistake_reported","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0015687942504882812,"mean_abs_dp":0.0015687942504882812,"max_abs_dlogit":0.007968664169311523,"oracle_top2_margin":0.1922793686389923,"near_tie":false,"probs":[0.40542909502983093,0.5945708751678467]},{"question_id":"all_positive","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00043451786041259766,"mean_abs_dp":0.0004345132037997246,"max_abs_dlogit":0.01703011989593506,"oracle_top2_margin":0.955326372757554,"near_tie":false,"probs":[0.021902278065681458,0.978097677230835]},{"question_id":"digital_readers","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0011210441589355469,"mean_abs_dp":0.0005605011247098446,"max_abs_dlogit":0.01809835433959961,"oracle_top2_margin":0.8206045627593994,"near_tie":false,"probs":[0.020612021908164024,0.06495009362697601,0.8834918737411499,0.030945954844355583]},{"question_id":"reply_count","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00015485286712646484,"mean_abs_dp":9.31047834455967e-05,"max_abs_dlogit":0.010372400283813477,"oracle_top2_margin":0.3369687795639038,"near_tie":false,"probs":[0.02952699549496174,0.3024747371673584,0.6396453380584717,0.028353016823530197]},{"question_id":"reception","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.003891885280609131,"mean_abs_dp":0.0020184540189802647,"max_abs_dlogit":0.010213375091552734,"oracle_top2_margin":0.38388296961784363,"near_tie":false,"probs":[0.023408105596899986,0.6608657836914062,0.2845475971698761,0.03117852844297886]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.003891885280609131,"mean_abs_dp":0.0007112058700017986,"max_abs_dlogit":0.01809835433959961},{"id":"ho_j01","arm":"text","variant":"base","tokens":769,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7428471659950446,"wall_seconds":1.7531938749889378,"finite":true,"index":10,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.1149826049804688,"ref_absmax":51.30471420288086,"rel_max_abs_diff":0.041223942825544646,"min_pos_cos":0.9968378753246899,"min_pos_cos_index":447,"mean_pos_cos":0.9999055127960712,"positions":769,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"sick_days","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00014847517013549805,"mean_abs_dp":7.42604024708271e-05,"max_abs_dlogit":0.01214599609375,"oracle_top2_margin":0.9564286395907402,"near_tie":false,"probs":[0.012837243266403675,0.9690799713134766,0.010411175899207592,0.007671688217669725]},{"question_id":"longest_day","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0032848119735717773,"mean_abs_dp":0.0016423910856246948,"max_abs_dlogit":0.016978144645690918,"oracle_top2_margin":0.5593627244234085,"near_tie":false,"probs":[0.18196795880794525,0.05618702620267868,0.747383713722229,0.014461291022598743]},{"question_id":"approved","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.17692756652832e-05,"mean_abs_dp":3.173062577843666e-05,"max_abs_dlogit":0.0077626705169677734,"oracle_top2_margin":0.9902706993743777,"near_tie":false,"probs":[0.004896359518170357,0.9951035976409912]},{"question_id":"late_start","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.001629173755645752,"mean_abs_dp":0.001629158854484558,"max_abs_dlogit":0.007995843887329102,"oracle_top2_margin":0.0010575354099273682,"near_tie":true,"probs":[0.5021579265594482,0.49784207344055176]},{"question_id":"hours_concern","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0007923468947410583,"mean_abs_dp":0.0005282536149024963,"max_abs_dlogit":0.015117764472961426,"oracle_top2_margin":0.3177451193332672,"near_tie":false,"probs":[0.30243366956710815,0.6204555034637451,0.07711083441972733]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0032848119735717773,"mean_abs_dp":0.0007848763838410377,"max_abs_dlogit":0.016978144645690918},{"id":"ho_j02","arm":"text","variant":"base","tokens":335,"calls":6,"padded_tokens":384,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":0.8045391659834422,"wall_seconds":0.8150405419874005,"finite":true,"index":11,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":0.8936862945556641,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.018496331674772094,"min_pos_cos":0.9988773040154744,"min_pos_cos_index":174,"mean_pos_cos":0.9999363816535101,"positions":335,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"beyond_tolerance","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.004511356353759766,"mean_abs_dp":0.0015534923101464908,"max_abs_dlogit":0.01928997039794922,"oracle_top2_margin":0.10029461979866028,"near_tie":false,"probs":[0.024175232276320457,0.0638476014137268,0.36234235763549805,0.4706559181213379,0.05973788723349571,0.01924106292426586]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.004511356353759766,"mean_abs_dp":0.0015534923101464908,"max_abs_dlogit":0.01928997039794922},{"id":"ho_j03","arm":"text","variant":"base","tokens":775,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.744423124997411,"wall_seconds":1.7556397499865852,"finite":true,"index":12,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.2660903930664062,"ref_absmax":49.201229095458984,"rel_max_abs_diff":0.04605759723339014,"min_pos_cos":0.9928918713166864,"min_pos_cos_index":414,"mean_pos_cos":0.9998532057983123,"positions":775,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"excursion","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0013482794165611267,"mean_abs_dp":0.0013482384383678436,"max_abs_dlogit":0.014943122863769531,"oracle_top2_margin":0.8218574672937393,"near_tie":false,"probs":[0.9095805287361145,0.09041953831911087]},{"question_id":"first_excursion","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0002790093421936035,"mean_abs_dp":0.0001069920447965463,"max_abs_dlogit":0.013489723205566406,"oracle_top2_margin":0.9088389798998833,"near_tie":false,"probs":[0.011989203281700611,0.014739119447767735,0.926331102848053,0.014520259574055672,0.014376009814441204,0.01804438978433609]},{"question_id":"cause","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00020520668476819992,"mean_abs_dp":0.00010258890688419342,"max_abs_dlogit":0.01979827880859375,"oracle_top2_margin":0.9396330676972866,"near_tie":false,"probs":[0.015740489587187767,0.9553438425064087,0.014260424301028252,0.014655198901891708]},{"question_id":"recovered","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0011615380644798279,"mean_abs_dp":0.0011614970862865448,"max_abs_dlogit":0.012349605560302734,"oracle_top2_margin":0.8434159979224205,"near_tie":false,"probs":[0.9205465316772461,0.07945352792739868]},{"question_id":"risk","type":"score","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.001217484474182129,"mean_abs_dp":0.0005583532154560089,"max_abs_dlogit":0.005995750427246094,"oracle_top2_margin":0.36235082149505615,"near_tie":false,"probs":[0.058482710272073746,0.10404934734106064,0.5630780458450317,0.20297062397003174,0.07141924649477005]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0013482794165611267,"mean_abs_dp":0.0004665023696265723,"max_abs_dlogit":0.01979827880859375},{"id":"ho_j04","arm":"text","variant":"base","tokens":610,"calls":10,"padded_tokens":640,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.3416084170166869,"wall_seconds":1.3524329999927431,"finite":true,"index":13,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.5590405464172363,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.032266950065837685,"min_pos_cos":0.994461128252571,"min_pos_cos_index":333,"mean_pos_cos":0.9998333301081307,"positions":610,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"free_cancellation","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00184573233127594,"mean_abs_dp":0.0018456950783729553,"max_abs_dlogit":0.05870091915130615,"oracle_top2_margin":0.640770673751831,"near_tie":false,"probs":[0.17776893079280853,0.8222309947013855]},{"question_id":"party","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00046497583389282227,"mean_abs_dp":0.00023249280638992786,"max_abs_dlogit":0.015117526054382324,"oracle_top2_margin":0.9523126352578402,"near_tie":false,"probs":[0.011256342753767967,0.011928590014576912,0.9648825526237488,0.011932481080293655]},{"question_id":"kitchen_need","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0002267155796289444,"mean_abs_dp":0.00011338666081428528,"max_abs_dlogit":0.02116250991821289,"oracle_top2_margin":0.9280225206166506,"near_tie":false,"probs":[0.02614637464284897,0.9537919759750366,0.010379081591963768,0.009682503528892994]},{"question_id":"special_occasion","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0005485936999320984,"mean_abs_dp":0.0005485378205776215,"max_abs_dlogit":0.020288944244384766,"oracle_top2_margin":0.8431091755628586,"near_tie":false,"probs":[0.9210060834884644,0.07899398356676102]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00184573233127594,"mean_abs_dp":0.0005143319722265005,"max_abs_dlogit":0.05870091915130615},{"id":"ho_j05","arm":"text","variant":"base","tokens":775,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7444770000001881,"wall_seconds":1.7551291669951752,"finite":true,"index":14,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.2389907836914062,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.04633965677240751,"min_pos_cos":0.9927251779308535,"min_pos_cos_index":169,"mean_pos_cos":0.9998802707130456,"positions":775,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"missed_month","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00022035837173461914,"mean_abs_dp":8.813058957457542e-05,"max_abs_dlogit":0.016236543655395508,"oracle_top2_margin":0.9646178442053497,"near_tie":false,"probs":[0.007264615036547184,0.007543036714196205,0.9724490642547607,0.006033718585968018,0.006709613371640444]},{"question_id":"july_late","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00010694563388824463,"mean_abs_dp":0.00010690838098526001,"max_abs_dlogit":0.013639897108078003,"oracle_top2_margin":0.8146922215819359,"near_tie":false,"probs":[0.9074530005455017,0.0925469622015953]},{"question_id":"double_payment","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0003097653388977051,"mean_abs_dp":0.0003097653388977051,"max_abs_dlogit":0.01129060983657837,"oracle_top2_margin":0.33765122294425964,"near_tie":false,"probs":[0.6685158610343933,0.3314841687679291]},{"question_id":"owed","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.010159045457839966,"mean_abs_dp":0.005079545080661774,"max_abs_dlogit":0.037596821784973145,"oracle_top2_margin":0.13490816950798035,"near_tie":false,"probs":[0.09452196955680847,0.1520211398601532,0.30055975914001465,0.4528971314430237]},{"question_id":"reliability","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00045058131217956543,"mean_abs_dp":0.00022526830434799194,"max_abs_dlogit":0.005281567573547363,"oracle_top2_margin":0.2541508674621582,"near_tie":false,"probs":[0.1679958552122116,0.5043200254440308,0.2504727244377136,0.0772114172577858]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.010159045457839966,"mean_abs_dp":0.001323132583981051,"max_abs_dlogit":0.037596821784973145},{"id":"ho_j06","arm":"text","variant":"base","tokens":733,"calls":12,"padded_tokens":768,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.6090899999835528,"wall_seconds":1.62011300001177,"finite":true,"index":15,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":2.8887195587158203,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.05978699308872733,"min_pos_cos":0.979391940972539,"min_pos_cos_index":497,"mean_pos_cos":0.9997855899728935,"positions":733,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"hides_endpoints","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.806586265563965e-05,"mean_abs_dp":8.80509614944458e-05,"max_abs_dlogit":0.010135054588317871,"oracle_top2_margin":0.9664727952331305,"near_tie":false,"probs":[0.016675511375069618,0.983324408531189]},{"question_id":"route_audience","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.8848648071289062e-05,"mean_abs_dp":1.924748842914899e-05,"max_abs_dlogit":0.020158052444458008,"oracle_top2_margin":0.9730440806597471,"near_tie":false,"probs":[0.01082122977823019,0.9838082194328308,0.005370569881051779]},{"question_id":"marketing","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.264689490199089e-05,"mean_abs_dp":1.2641539797186852e-05,"max_abs_dlogit":0.004467487335205078,"oracle_top2_margin":0.9910305761732161,"near_tie":false,"probs":[0.004472063854336739,0.9955279231071472]},{"question_id":"zone_boundaries","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.006981611251831055,"mean_abs_dp":0.0034908298403024673,"max_abs_dlogit":0.02388852834701538,"oracle_top2_margin":0.3823526203632355,"near_tie":false,"probs":[0.6366093158721924,0.2668871581554413,0.05555683746933937,0.04094671458005905]},{"question_id":"unit_mismatch","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0014006495475769043,"mean_abs_dp":0.0014006495475769043,"max_abs_dlogit":0.011277258396148682,"oracle_top2_margin":0.3890989124774933,"near_tie":false,"probs":[0.30404987931251526,0.6959500908851624]},{"question_id":"privacy_level","type":"score","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.001165628433227539,"mean_abs_dp":0.0004662730731070042,"max_abs_dlogit":0.012508630752563477,"oracle_top2_margin":0.1093018651008606,"near_tie":false,"probs":[0.14661727845668793,0.465727835893631,0.35782313346862793,0.024518754333257675,0.0053128874860703945]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.006981611251831055,"mean_abs_dp":0.001075283960542745,"max_abs_dlogit":0.02388852834701538},{"id":"ho_j07","arm":"text","variant":"base","tokens":826,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7445145829988178,"wall_seconds":1.754775458975928,"finite":true,"index":16,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.9900398254394531,"ref_absmax":50.477657318115234,"rel_max_abs_diff":0.03942417162702349,"min_pos_cos":0.9969489384344329,"min_pos_cos_index":391,"mean_pos_cos":0.9998601747796279,"positions":826,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"unassigned","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0002415180206298828,"mean_abs_dp":0.000128265842795372,"max_abs_dlogit":0.013881683349609375,"oracle_top2_margin":0.928688820451498,"near_tie":false,"probs":[0.9448867440223694,0.013328743167221546,0.01344634871929884,0.011857859790325165,0.01648038998246193]},{"question_id":"open_high","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00016736984252929688,"mean_abs_dp":7.512494921684265e-05,"max_abs_dlogit":0.012830495834350586,"oracle_top2_margin":0.9467990919947624,"near_tie":false,"probs":[0.007680831011384726,0.016482984647154808,0.9629503488540649,0.008142894133925438,0.004742932040244341]},{"question_id":"shared_theme","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.295608520507812e-05,"mean_abs_dp":3.647487028501928e-05,"max_abs_dlogit":0.02138376235961914,"oracle_top2_margin":0.9929788664449006,"near_tie":false,"probs":[0.0014029360609129071,0.9950826168060303,0.00200496893376112,0.0015094587579369545]},{"question_id":"all_closed","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.0682811737060547e-05,"mean_abs_dp":2.0665349438786507e-05,"max_abs_dlogit":0.0029927492141723633,"oracle_top2_margin":0.9899770021438599,"near_tie":false,"probs":[0.004990851040929556,0.995009183883667]},{"question_id":"wider_incident","type":"score","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0009705126285552979,"mean_abs_dp":0.000970497727394104,"max_abs_dlogit":0.011175215244293213,"oracle_top2_margin":0.13046076893806458,"near_tie":false,"probs":[0.4357401132583618,0.5642598867416382]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0009705126285552979,"mean_abs_dp":0.00017473219971482953,"max_abs_dlogit":0.02138376235961914},{"id":"ho_j08","arm":"text","variant":"base","tokens":798,"calls":13,"padded_tokens":832,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":1.7448875419795513,"wall_seconds":1.7554695420258213,"finite":true,"index":17,"shard":"shard_00","first_in_process":false,"source":"heldout_json","hidden":{"max_abs_diff":1.230051040649414,"ref_absmax":52.01266860961914,"rel_max_abs_diff":0.02364906615120168,"min_pos_cos":0.9961650558300945,"min_pos_cos_index":224,"mean_pos_cos":0.9999142976482218,"positions":798,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"all_pass","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0011567175388336182,"mean_abs_dp":0.0011566728353500366,"max_abs_dlogit":0.003506183624267578,"oracle_top2_margin":0.521710604429245,"near_tie":false,"probs":[0.23798799514770508,0.7620119452476501]},{"question_id":"failing_dimension","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0072307586669921875,"mean_abs_dp":0.003057970106601715,"max_abs_dlogit":0.029584288597106934,"oracle_top2_margin":0.3418823480606079,"near_tie":false,"probs":[0.531194269657135,0.08215468376874924,0.17554400861263275,0.17596453428268433,0.03514253348112106]},{"question_id":"failing_sample","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.002221137285232544,"mean_abs_dp":0.0013134442269802094,"max_abs_dlogit":0.021771252155303955,"oracle_top2_margin":0.039329737424850464,"near_tie":false,"probs":[0.37393683195114136,0.12745387852191925,0.1681187003850937,0.3304905891418457]},{"question_id":"disposition","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0008356794714927673,"mean_abs_dp":0.0005571221311887106,"max_abs_dlogit":0.009411871433258057,"oracle_top2_margin":0.41586053371429443,"near_tie":false,"probs":[0.2529325783252716,0.6684622168540955,0.07860518246889114]},{"question_id":"capability","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.001096189022064209,"mean_abs_dp":0.0005480572581291199,"max_abs_dlogit":0.012767672538757324,"oracle_top2_margin":0.03322687745094299,"near_tie":false,"probs":[0.05988406762480736,0.3106493651866913,0.28576117753982544,0.34370532631874084]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0072307586669921875,"mean_abs_dp":0.0014844760298728943,"max_abs_dlogit":0.029584288597106934},{"id":"ho_l01","arm":"text","variant":"base","tokens":2850,"calls":45,"padded_tokens":2880,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":6.0792398329940625,"wall_seconds":6.089700041018659,"finite":true,"index":18,"shard":"shard_00","first_in_process":false,"source":"heldout_text_long","hidden":{"max_abs_diff":3.463749885559082,"ref_absmax":50.8991584777832,"rel_max_abs_diff":0.06805122106431213,"min_pos_cos":0.9800416625965211,"min_pos_cos_index":1103,"mean_pos_cos":0.9997480416747426,"positions":2850,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"root_cause","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0003427267074584961,"mean_abs_dp":0.00013711154460906982,"max_abs_dlogit":0.011502265930175781,"oracle_top2_margin":0.9651294462382793,"near_tie":false,"probs":[0.9757819175720215,0.0110629266127944,0.004292289726436138,0.0043523251079022884,0.0045105647295713425]},{"question_id":"peak_temperature","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00017726421356201172,"mean_abs_dp":8.861086098477244e-05,"max_abs_dlogit":0.013463020324707031,"oracle_top2_margin":0.9787671356461942,"near_tie":false,"probs":[0.986185610294342,0.007652732543647289,0.0033841035328805447,0.002777489135041833]},{"question_id":"first_text_reached","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.38690185546875e-05,"mean_abs_dp":4.3861567974090576e-05,"max_abs_dlogit":0.007699728012084961,"oracle_top2_margin":0.9262152798473835,"near_tie":false,"probs":[0.03693621978163719,0.9630637764930725]},{"question_id":"dough_outcome","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.256570875644684e-05,"mean_abs_dp":2.628419315442443e-05,"max_abs_dlogit":0.008045196533203125,"oracle_top2_margin":0.9787769615650177,"near_tie":false,"probs":[0.004077042452991009,0.9855549335479736,0.006852412596344948,0.003515625838190317]},{"question_id":"standby_ready","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0001183822751045227,"mean_abs_dp":0.00011834874749183655,"max_abs_dlogit":0.018794894218444824,"oracle_top2_margin":0.9688542913645506,"near_tie":false,"probs":[0.015454480424523354,0.9845454692840576]},{"question_id":"above_limit","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.004114687442779541,"mean_abs_dp":0.0022343965247273445,"max_abs_dlogit":0.05040156841278076,"oracle_top2_margin":0.15288570523262024,"near_tie":false,"probs":[0.17057952284812927,0.4693034589290619,0.32017838954925537,0.039938610047101974]},{"question_id":"seriousness","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0028735995292663574,"mean_abs_dp":0.0014368155971169472,"max_abs_dlogit":0.01566857099533081,"oracle_top2_margin":0.03678572177886963,"near_tie":false,"probs":[0.03751992806792259,0.4672025144100189,0.4358515739440918,0.059425968676805496]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.004114687442779541,"mean_abs_dp":0.0006461762823164463,"max_abs_dlogit":0.05040156841278076},{"id":"ho_l02","arm":"text","variant":"base","tokens":2731,"calls":43,"padded_tokens":2752,"grid":null,"n_image_rows":0,"start":1073741824,"amount":0,"image_tokens":0,"decode_seconds":5.808760582993273,"wall_seconds":5.819191375019727,"finite":true,"index":19,"shard":"shard_00","first_in_process":false,"source":"heldout_text_long","hidden":{"max_abs_diff":1.4593229293823242,"ref_absmax":56.4263916015625,"rel_max_abs_diff":0.025862418062931995,"min_pos_cos":0.9948945202344053,"min_pos_cos_index":2718,"mean_pos_cos":0.9998600081287541,"positions":2731,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"break_notice","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.000695347785949707,"mean_abs_dp":0.00028216652572155,"max_abs_dlogit":0.014304637908935547,"oracle_top2_margin":0.9312390778213739,"near_tie":false,"probs":[0.022861422970891,0.005199036560952663,0.9553045034408569,0.010960283689200878,0.005674825049936771]},{"question_id":"review_direction","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00013244152069091797,"mean_abs_dp":6.624707020819187e-05,"max_abs_dlogit":0.012965917587280273,"oracle_top2_margin":0.9693590383976698,"near_tie":false,"probs":[0.007123055402189493,0.004154789727181196,0.9789495468139648,0.009772652760148048]},{"question_id":"sublet_part","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011238548904657364,"mean_abs_dp":0.00011234031990170479,"max_abs_dlogit":0.010183453559875488,"oracle_top2_margin":0.9782952684909105,"near_tie":false,"probs":[0.010740009136497974,0.9892599582672119]},{"question_id":"roof","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":6.0021644458174706e-05,"mean_abs_dp":3.157748142257333e-05,"max_abs_dlogit":0.020382404327392578,"oracle_top2_margin":0.985643103485927,"near_tie":false,"probs":[0.0039450377225875854,0.9894744157791138,0.0033014402724802494,0.00327917723916471]},{"question_id":"deposit","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0008147954940795898,"mean_abs_dp":0.0004073876771144569,"max_abs_dlogit":0.009550511837005615,"oracle_top2_margin":0.8725584335625172,"near_tie":false,"probs":[0.05570084974169731,0.9298556447029114,0.010719523765146732,0.0037239831872284412]},{"question_id":"kilns","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.181529998779297e-05,"mean_abs_dp":2.180691808462143e-05,"max_abs_dlogit":0.019030094146728516,"oracle_top2_margin":0.9571442604064941,"near_tie":false,"probs":[0.9785503149032593,0.02144966833293438]},{"question_id":"balance","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0007554292678833008,"mean_abs_dp":0.0005035847425460815,"max_abs_dlogit":0.011129498481750488,"oracle_top2_margin":0.31222064793109894,"near_tie":false,"probs":[0.5563672184944153,0.2435609996318817,0.2000717967748642]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008147954940795898,"mean_abs_dp":0.00021711376029998064,"max_abs_dlogit":0.020382404327392578},{"id":"ho_i01","arm":"g256","variant":"base","tokens":529,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2091101249970961,"wall_seconds":1.2219031660060864,"finite":true,"index":20,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.991244316101074,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.06190880591555485,"min_pos_cos":0.9914004517537414,"min_pos_cos_index":126,"mean_pos_cos":0.9998820275373076,"positions":529,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_slice","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00027358531951904297,"mean_abs_dp":0.00014290062244981527,"max_abs_dlogit":0.00886678695678711,"oracle_top2_margin":0.9263720586895943,"near_tie":false,"probs":[0.03160638362169266,0.0056717307306826115,0.9584693312644958,0.0042525832541286945]},{"question_id":"over_half","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0015985369682312012,"mean_abs_dp":0.0015985220670700073,"max_abs_dlogit":0.010611951351165771,"oracle_top2_margin":0.4712936580181122,"near_tie":false,"probs":[0.2659516930580139,0.7340483069419861]},{"question_id":"smallest_label","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00024586915969848633,"mean_abs_dp":0.00012490490917116404,"max_abs_dlogit":0.008318424224853516,"oracle_top2_margin":0.9675928661599755,"near_tie":false,"probs":[0.0045936573296785355,0.9777417778968811,0.009775660000741482,0.00788880605250597]},{"question_id":"blue_share","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00027520954608917236,"mean_abs_dp":0.0001834773768981298,"max_abs_dlogit":0.009051084518432617,"oracle_top2_margin":0.9218912087380886,"near_tie":false,"probs":[0.027026016265153885,0.9484136700630188,0.024560267105698586]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0015985369682312012,"mean_abs_dp":0.00037066910702448624,"max_abs_dlogit":0.010611951351165771},{"id":"ho_i01","arm":"g448","variant":"base","tokens":661,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4767697920033243,"wall_seconds":1.4932952499948442,"finite":true,"index":21,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.0264772176742554,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.0626381235071223,"min_pos_cos":0.9879556442030867,"min_pos_cos_index":154,"mean_pos_cos":0.9998361790958292,"positions":661,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"biggest_slice","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0003287792205810547,"mean_abs_dp":0.00016440078616142273,"max_abs_dlogit":0.008395776152610779,"oracle_top2_margin":0.9189082495868206,"near_tie":false,"probs":[0.035333212465047836,0.005201953463256359,0.9548182487487793,0.004646554123610258]},{"question_id":"over_half","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0005277693271636963,"mean_abs_dp":0.0005277544260025024,"max_abs_dlogit":0.0057411789894104,"oracle_top2_margin":0.3951268196105957,"near_tie":false,"probs":[0.30190882086753845,0.6980911493301392]},{"question_id":"smallest_label","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011932849884033203,"mean_abs_dp":5.966587923467159e-05,"max_abs_dlogit":0.012779712677001953,"oracle_top2_margin":0.9696091758087277,"near_tie":false,"probs":[0.003704702714458108,0.9806337952613831,0.010828902013599873,0.004832667764276266]},{"question_id":"blue_share","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0001781061291694641,"mean_abs_dp":0.00011870451271533966,"max_abs_dlogit":0.005744218826293945,"oracle_top2_margin":0.9033529125154018,"near_tie":false,"probs":[0.028332524001598358,0.9376459121704102,0.03402155265212059]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0005277693271636963,"mean_abs_dp":0.00017752992705656932,"max_abs_dlogit":0.012779712677001953},{"id":"ho_i02","arm":"g256","variant":"base","tokens":570,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2091332910058554,"wall_seconds":1.2221448750060517,"finite":true,"index":22,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.9170246124267578,"ref_absmax":51.21390151977539,"rel_max_abs_diff":0.037431723722250114,"min_pos_cos":0.995968618755459,"min_pos_cos_index":100,"mean_pos_cos":0.9998595960216379,"positions":570,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"harlowick_platform","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00047016143798828125,"mean_abs_dp":0.00023507821606472135,"max_abs_dlogit":0.03828072547912598,"oracle_top2_margin":0.9853119025938213,"near_tie":false,"probs":[0.9888073205947876,0.003800919745117426,0.004118706565350294,0.0032731404062360525]},{"question_id":"earliest","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.247420191764832e-05,"mean_abs_dp":4.677078686654568e-05,"max_abs_dlogit":0.007880210876464844,"oracle_top2_margin":0.9508627159520984,"near_tie":false,"probs":[0.010176203213632107,0.9648943543434143,0.013960250653326511,0.01096916850656271]},{"question_id":"platform_3","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0001761317253112793,"mean_abs_dp":9.107380174100399e-05,"max_abs_dlogit":0.005095243453979492,"oracle_top2_margin":0.9140067957341671,"near_tie":false,"probs":[0.014026835560798645,0.02744407020509243,0.9417558312416077,0.016773264855146408]},{"question_id":"after_nine","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00020074471831321716,"mean_abs_dp":0.0002007167786359787,"max_abs_dlogit":0.013600945472717285,"oracle_top2_margin":0.943212391808629,"near_tie":false,"probs":[0.9714055061340332,0.028594547882676125]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00047016143798828125,"mean_abs_dp":0.00013522319828293154,"max_abs_dlogit":0.03828072547912598},{"id":"ho_i02","arm":"g448","variant":"base","tokens":702,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4767850839998573,"wall_seconds":1.4937551250040997,"finite":true,"index":23,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.5673141479492188,"ref_absmax":51.250701904296875,"rel_max_abs_diff":0.06960517642491319,"min_pos_cos":0.9964668337071142,"min_pos_cos_index":227,"mean_pos_cos":0.9998542433863519,"positions":702,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"harlowick_platform","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":6.753206253051758e-05,"mean_abs_dp":3.379839472472668e-05,"max_abs_dlogit":0.015048861503601074,"oracle_top2_margin":0.9784979093819857,"near_tie":false,"probs":[0.9842703938484192,0.005678046494722366,0.00566966412588954,0.004381807986646891]},{"question_id":"earliest","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0003216862678527832,"mean_abs_dp":0.00016945553943514824,"max_abs_dlogit":0.01688385009765625,"oracle_top2_margin":0.9492904739454389,"near_tie":false,"probs":[0.010599231347441673,0.9633229970932007,0.013393419794738293,0.012684294022619724]},{"question_id":"platform_3","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009492039680480957,"mean_abs_dp":0.00047462061047554016,"max_abs_dlogit":0.02438068389892578,"oracle_top2_margin":0.8928603697568178,"near_tie":false,"probs":[0.018376212567090988,0.02976970002055168,0.9238580465316772,0.0279960036277771]},{"question_id":"after_nine","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.440918564796448e-05,"mean_abs_dp":4.4407323002815247e-05,"max_abs_dlogit":0.00992777943611145,"oracle_top2_margin":0.9381010439246893,"near_tie":false,"probs":[0.9690949320793152,0.030905073508620262]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0009492039680480957,"mean_abs_dp":0.00020002234461052076,"max_abs_dlogit":0.02438068389892578},{"id":"ho_i03","arm":"g256","variant":"base","tokens":541,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2128325420198962,"wall_seconds":1.2258164169907104,"finite":true,"index":24,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.1933550834655762,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.024698524961443945,"min_pos_cos":0.9981662733225831,"min_pos_cos_index":90,"mean_pos_cos":0.999865128127041,"positions":541,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"limit","type":"choice","n_options":10,"argmax":6,"argmax_oracle":6,"argmax_equal":true,"max_abs_dp":8.83936882019043e-05,"mean_abs_dp":2.0934874191880227e-05,"max_abs_dlogit":0.0038590431213378906,"oracle_top2_margin":0.9458053046837449,"near_tie":false,"probs":[0.005401438567787409,0.004845323972404003,0.004600202664732933,0.004840880632400513,0.004772335756570101,0.006062936969101429,0.9525426626205444,0.006809440441429615,0.005208855960518122,0.0049158548936247826]},{"question_id":"outline","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00014853477478027344,"mean_abs_dp":7.424317300319672e-05,"max_abs_dlogit":0.005409598350524902,"oracle_top2_margin":0.9205611497163773,"near_tie":false,"probs":[0.021570825949311256,0.9423277378082275,0.01802004873752594,0.018081478774547577]},{"question_id":"seventy_ok","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00035412609577178955,"mean_abs_dp":0.0003541186451911926,"max_abs_dlogit":0.023581862449645996,"oracle_top2_margin":0.955428808927536,"near_tie":false,"probs":[0.0219314843416214,0.9780685305595398]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00035412609577178955,"mean_abs_dp":7.59099202696234e-05,"max_abs_dlogit":0.023581862449645996},{"id":"ho_i03","arm":"g448","variant":"base","tokens":673,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4797394159832038,"wall_seconds":1.4964345830085222,"finite":true,"index":25,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.0039443969726562,"ref_absmax":48.58998107910156,"rel_max_abs_diff":0.061822300199755494,"min_pos_cos":0.9929333592527406,"min_pos_cos_index":190,"mean_pos_cos":0.9998322147223698,"positions":673,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"limit","type":"choice","n_options":10,"argmax":6,"argmax_oracle":6,"argmax_equal":true,"max_abs_dp":8.767843246459961e-05,"mean_abs_dp":1.7537502571940422e-05,"max_abs_dlogit":0.0035429000854492188,"oracle_top2_margin":0.947433871217072,"near_tie":false,"probs":[0.005209476221352816,0.004792790859937668,0.004543950315564871,0.004663475789129734,0.004614856094121933,0.00554837379604578,0.9541544318199158,0.006592470221221447,0.005149712320417166,0.004730416461825371]},{"question_id":"outline","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.0005892515182495e-05,"mean_abs_dp":2.026068978011608e-05,"max_abs_dlogit":0.0019702911376953125,"oracle_top2_margin":0.9329899717122316,"near_tie":false,"probs":[0.01811048574745655,0.9511579275131226,0.015004596672952175,0.01572703756392002]},{"question_id":"seventy_ok","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00018078088760375977,"mean_abs_dp":0.0001807771623134613,"max_abs_dlogit":0.004722952842712402,"oracle_top2_margin":0.9579436834901571,"near_tie":false,"probs":[0.02084738202393055,0.9791526198387146]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.00018078088760375977,"mean_abs_dp":3.862325684167445e-05,"max_abs_dlogit":0.004722952842712402},{"id":"ho_i04","arm":"g256","variant":"base","tokens":437,"calls":7,"padded_tokens":448,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":0.940095250000013,"wall_seconds":0.9529728329798672,"finite":true,"index":26,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.8389453887939453,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.038060120761340584,"min_pos_cos":0.9981880094808036,"min_pos_cos_index":319,"mean_pos_cos":0.999888943235276,"positions":437,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"reading","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00022975727915763855,"mean_abs_dp":0.00014001433737576008,"max_abs_dlogit":0.00775456428527832,"oracle_top2_margin":0.8822150155901909,"near_tie":false,"probs":[0.011424925178289413,0.04304029792547226,0.9251869320869446,0.020347805693745613]},{"question_id":"below_zero","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.072146162390709e-05,"mean_abs_dp":2.0702136680483818e-05,"max_abs_dlogit":0.014079570770263672,"oracle_top2_margin":0.9933206476271152,"near_tie":false,"probs":[0.0033189826644957066,0.9966810345649719]},{"question_id":"warmth","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.005230963230133057,"mean_abs_dp":0.002665412612259388,"max_abs_dlogit":0.0261685848236084,"oracle_top2_margin":0.25455665588378906,"near_tie":false,"probs":[0.013993262313306332,0.21585802733898163,0.5077261328697205,0.2624226212501526]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.005230963230133057,"mean_abs_dp":0.001126311207190156,"max_abs_dlogit":0.0261685848236084},{"id":"ho_i04","arm":"g448","variant":"base","tokens":569,"calls":9,"padded_tokens":576,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.2101939589774702,"wall_seconds":1.2272267499938607,"finite":true,"index":27,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.3646316528320312,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.06963668158140256,"min_pos_cos":0.9938547279457663,"min_pos_cos_index":161,"mean_pos_cos":0.9998491585012471,"positions":569,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"reading","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0002956315875053406,"mean_abs_dp":0.0002081508282572031,"max_abs_dlogit":0.032214224338531494,"oracle_top2_margin":0.8055376335978508,"near_tie":false,"probs":[0.015648094937205315,0.08466126024723053,0.8903738260269165,0.009316778741776943]},{"question_id":"below_zero","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.2968666851520538e-07,"mean_abs_dp":1.2444797903299332e-07,"max_abs_dlogit":0.004334926605224609,"oracle_top2_margin":0.9941885098814964,"near_tie":false,"probs":[0.002905893372371793,0.9970941543579102]},{"question_id":"warmth","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.003522336483001709,"mean_abs_dp":0.001761163119226694,"max_abs_dlogit":0.028028011322021484,"oracle_top2_margin":0.4384543299674988,"near_tie":false,"probs":[0.010850201360881329,0.20486365258693695,0.6410807967185974,0.14320531487464905]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.003522336483001709,"mean_abs_dp":0.0007877504685893655,"max_abs_dlogit":0.032214224338531494},{"id":"ho_i05","arm":"g256","variant":"base","tokens":701,"calls":11,"padded_tokens":704,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.4757833339972422,"wall_seconds":1.4877730830048677,"finite":true,"index":28,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.9326934814453125,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.040000397916496594,"min_pos_cos":0.9938776132473593,"min_pos_cos_index":613,"mean_pos_cos":0.9998538108133501,"positions":701,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"first_weekday","type":"choice","n_options":7,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0007224678993225098,"mean_abs_dp":0.0002064276486635208,"max_abs_dlogit":0.042061567306518555,"oracle_top2_margin":0.686411127448082,"near_tie":false,"probs":[0.031204575672745705,0.7590996623039246,0.02437175251543522,0.027982983738183975,0.034389130771160126,0.07190273702144623,0.05104915052652359]},{"question_id":"circled","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00046390295028686523,"mean_abs_dp":0.00023197510745376348,"max_abs_dlogit":0.02098703384399414,"oracle_top2_margin":0.9759097397327423,"near_tie":false,"probs":[0.9824351668357849,0.00714831193909049,0.004043999128043652,0.006372533272951841]},{"question_id":"month_length","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0008336007595062256,"mean_abs_dp":0.0004400215111672878,"max_abs_dlogit":0.0676572322845459,"oracle_top2_margin":0.9243375789374113,"near_tie":false,"probs":[0.9490615725517273,0.02633764035999775,0.012883739545941353,0.011717087589204311]},{"question_id":"shaded_weekend","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.004549950361251831,"mean_abs_dp":0.004549935460090637,"max_abs_dlogit":0.0455244779586792,"oracle_top2_margin":0.7392615079879761,"near_tie":false,"probs":[0.12581929564476013,0.8741806745529175]},{"question_id":"shaded_count","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.004110932350158691,"mean_abs_dp":0.0020824549719691277,"max_abs_dlogit":0.010619521141052246,"oracle_top2_margin":0.181378573179245,"near_tie":false,"probs":[0.03534220531582832,0.032953742891550064,0.3712076246738434,0.5604963302612305]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.004549950361251831,"mean_abs_dp":0.0010267938487231731,"max_abs_dlogit":0.0676572322845459},{"id":"ho_i05","arm":"g448","variant":"base","tokens":833,"calls":14,"padded_tokens":896,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.8824279170075897,"wall_seconds":1.8991970000206493,"finite":true,"index":29,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.2186832427978516,"ref_absmax":48.98490905761719,"rel_max_abs_diff":0.04529319918075554,"min_pos_cos":0.9945869747030425,"min_pos_cos_index":151,"mean_pos_cos":0.9998519203473628,"positions":833,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"first_weekday","type":"choice","n_options":7,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0003262758255004883,"mean_abs_dp":0.00011991070849554879,"max_abs_dlogit":0.037109375,"oracle_top2_margin":0.8434231374412775,"near_tie":false,"probs":[0.020049454644322395,0.8732885122299194,0.015171071514487267,0.02118503302335739,0.01836591586470604,0.030236659571528435,0.021703308448195457]},{"question_id":"circled","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00039958953857421875,"mean_abs_dp":0.00019978429190814495,"max_abs_dlogit":0.011942863464355469,"oracle_top2_margin":0.9703278709203005,"near_tie":false,"probs":[0.9786578416824341,0.00889353733509779,0.005764411296695471,0.0066841659136116505]},{"question_id":"month_length","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00014977063983678818,"mean_abs_dp":8.730788249522448e-05,"max_abs_dlogit":0.04236960411071777,"oracle_top2_margin":0.9584462745115161,"near_tie":false,"probs":[0.9710932970046997,0.012929949909448624,0.008528189733624458,0.007448598742485046]},{"question_id":"shaded_weekend","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0006469152867794037,"mean_abs_dp":0.0006469022482633591,"max_abs_dlogit":0.012161731719970703,"oracle_top2_margin":0.904242217540741,"near_tie":false,"probs":[0.0472320057451725,0.9527680277824402]},{"question_id":"shaded_count","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0015345215797424316,"mean_abs_dp":0.0007891911081969738,"max_abs_dlogit":0.012902498245239258,"oracle_top2_margin":0.703109622001648,"near_tie":false,"probs":[0.03487464413046837,0.02004997991025448,0.8226555585861206,0.12241988629102707]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0015345215797424316,"mean_abs_dp":0.0003065863136379492,"max_abs_dlogit":0.04236960411071777},{"id":"ho_i06","arm":"g256","variant":"base","tokens":628,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.343901249987539,"wall_seconds":1.3571691249962896,"finite":true,"index":30,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.359935760498047,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.06953961850861212,"min_pos_cos":0.9947335724579381,"min_pos_cos_index":397,"mean_pos_cos":0.9998702893087782,"positions":628,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"third_digit","type":"choice","n_options":10,"argmax":9,"argmax_oracle":9,"argmax_equal":true,"max_abs_dp":0.0023425817489624023,"mean_abs_dp":0.00047487332485616205,"max_abs_dlogit":0.055295467376708984,"oracle_top2_margin":0.821707833558321,"near_tie":false,"probs":[0.01517884898930788,0.006117936689406633,0.03725938871502876,0.030700139701366425,0.0218223724514246,0.007072991691529751,0.005704856012016535,0.005838204175233841,0.01507444866001606,0.8552307486534119]},{"question_id":"digit_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0015915632247924805,"mean_abs_dp":0.0006409034132957459,"max_abs_dlogit":0.016437649726867676,"oracle_top2_margin":0.27862802147865295,"near_tie":false,"probs":[0.03819863498210907,0.20209763944149017,0.47897547483444214,0.19953158497810364,0.08119665086269379]},{"question_id":"has_seven","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00019700918346643448,"mean_abs_dp":0.00019697146490216255,"max_abs_dlogit":0.014748096466064453,"oracle_top2_margin":0.9801594745367765,"near_tie":false,"probs":[0.00972327683120966,0.9902766942977905]},{"question_id":"red_last","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.993511199951172e-06,"mean_abs_dp":3.929249942302704e-06,"max_abs_dlogit":0.008844256401062012,"oracle_top2_margin":0.9475219752639532,"near_tie":false,"probs":[0.973764955997467,0.026235122233629227]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0023425817489624023,"mean_abs_dp":0.0004397395655120674,"max_abs_dlogit":0.055295467376708984},{"id":"ho_i06","arm":"g448","variant":"base","tokens":760,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6146877499995753,"wall_seconds":1.6315944169764407,"finite":true,"index":31,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.759695053100586,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.036419830478419464,"min_pos_cos":0.9922552787100797,"min_pos_cos_index":189,"mean_pos_cos":0.9998515894788665,"positions":760,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"third_digit","type":"choice","n_options":10,"argmax":9,"argmax_oracle":9,"argmax_equal":true,"max_abs_dp":0.0033112764358520508,"mean_abs_dp":0.000664506759494543,"max_abs_dlogit":0.017783403396606445,"oracle_top2_margin":0.41684913635253906,"near_tie":false,"probs":[0.008105896413326263,0.004539093468338251,0.2635660767555237,0.010200797580182552,0.005654067266732454,0.004251508042216301,0.002541228663176298,0.0033635145518928766,0.011062359437346458,0.6867154240608215]},{"question_id":"digit_count","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.002385258674621582,"mean_abs_dp":0.0010839171707630157,"max_abs_dlogit":0.020868659019470215,"oracle_top2_margin":0.577194482088089,"near_tie":false,"probs":[0.10573200136423111,0.6938474178314209,0.12082872539758682,0.04665210843086243,0.032939713448286057]},{"question_id":"has_seven","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011285301297903061,"mean_abs_dp":0.00011281250044703484,"max_abs_dlogit":0.008392810821533203,"oracle_top2_margin":0.9850624282844365,"near_tie":false,"probs":[0.007355958689004183,0.9926440119743347]},{"question_id":"red_last","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001410301774740219,"mean_abs_dp":0.00014099758118391037,"max_abs_dlogit":0.01125335693359375,"oracle_top2_margin":0.9580337218940258,"near_tie":false,"probs":[0.9788758754730225,0.02112414874136448]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0033112764358520508,"mean_abs_dp":0.0006616986111590737,"max_abs_dlogit":0.020868659019470215},{"id":"ho_i07","arm":"g256","variant":"base","tokens":535,"calls":9,"padded_tokens":576,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.2127412910049316,"wall_seconds":1.2260994579992257,"finite":true,"index":32,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":7.5727128982543945,"ref_absmax":48.31694793701172,"rel_max_abs_diff":0.15672995132321985,"min_pos_cos":0.8917932115996107,"min_pos_cos_index":94,"mean_pos_cos":0.9995902193484186,"positions":535,"positions_below":{"0.99":2,"0.9":1,"0.5":0},"positions_below_0.99_image_rows":2},"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011388398706912994,"mean_abs_dp":5.697988672181964e-05,"max_abs_dlogit":0.015163183212280273,"oracle_top2_margin":0.9713920271024108,"near_tie":false,"probs":[0.009263979271054268,0.9807456731796265,0.0072436160407960415,0.0027467745821923018]},{"question_id":"paid_by","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.535664528608322e-05,"mean_abs_dp":5.6887976825237274e-05,"max_abs_dlogit":0.011904239654541016,"oracle_top2_margin":0.9541426971554756,"near_tie":false,"probs":[0.9745169878005981,0.020457426086068153,0.0050256079994142056]},{"question_id":"priciest","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0014279037714004517,"mean_abs_dp":0.0007150668534450233,"max_abs_dlogit":0.009349822998046875,"oracle_top2_margin":0.7043779790401459,"near_tie":false,"probs":[0.14090989530086517,0.0034551587887108326,0.8425172567367554,0.013117624446749687]},{"question_id":"change_shown","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.500150680541992e-05,"mean_abs_dp":4.499219357967377e-05,"max_abs_dlogit":0.0007497072219848633,"oracle_top2_margin":0.9286240413784981,"near_tie":false,"probs":[0.03573295846581459,0.9642670154571533]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0014279037714004517,"mean_abs_dp":0.0002576027137155716,"max_abs_dlogit":0.015163183212280273},{"id":"ho_i07","arm":"g448","variant":"base","tokens":667,"calls":11,"padded_tokens":704,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.4823157499777153,"wall_seconds":1.4999795829935465,"finite":true,"index":33,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":3.775679588317871,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.07814414825105435,"min_pos_cos":0.9785905534628597,"min_pos_cos_index":489,"mean_pos_cos":0.9997758201244764,"positions":667,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0003699660301208496,"mean_abs_dp":0.00019009370589628816,"max_abs_dlogit":0.04134011268615723,"oracle_top2_margin":0.9620198309421539,"near_tie":false,"probs":[0.012504691258072853,0.9751191735267639,0.008814778178930283,0.003561366582289338]},{"question_id":"paid_by","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.93924643099308e-05,"mean_abs_dp":1.2968511631091436e-05,"max_abs_dlogit":0.007063865661621094,"oracle_top2_margin":0.9557577352970839,"near_tie":false,"probs":[0.9749624133110046,0.0192094836384058,0.005828037858009338]},{"question_id":"priciest","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008881092071533203,"mean_abs_dp":0.0004463231307454407,"max_abs_dlogit":0.009410381317138672,"oracle_top2_margin":0.7944317236542702,"near_tie":false,"probs":[0.09330081194639206,0.003258356126025319,0.8859661221504211,0.017474651336669922]},{"question_id":"change_shown","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0005336403846740723,"mean_abs_dp":0.00053362175822258,"max_abs_dlogit":0.007668733596801758,"oracle_top2_margin":0.9117454215884209,"near_tie":false,"probs":[0.04466089606285095,0.9553390741348267]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0008881092071533203,"mean_abs_dp":0.0002809089536850269,"max_abs_dlogit":0.04134011268615723},{"id":"ho_i08","arm":"g256","variant":"base","tokens":581,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3432994169997983,"wall_seconds":1.3554765410081018,"finite":true,"index":34,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":1.5175628662109375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.0314085596575518,"min_pos_cos":0.9973578828647585,"min_pos_cos_index":174,"mean_pos_cos":0.9999051285266352,"positions":581,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_space","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00026743486523628235,"mean_abs_dp":0.00014792364090681075,"max_abs_dlogit":0.0074193477630615234,"oracle_top2_margin":0.76869260892272,"near_tie":false,"probs":[0.06251239031553268,0.04880332946777344,0.8309842944145203,0.043412208557128906,0.014287720434367657]},{"question_id":"entrance","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.4261494874954224e-05,"mean_abs_dp":2.0464509725570677e-05,"max_abs_dlogit":0.011136054992675781,"oracle_top2_margin":0.9482293706387281,"near_tie":false,"probs":[0.95976322889328,0.008873446844518185,0.011502591893076897,0.008704466745257378,0.011156310327351093]},{"question_id":"wc_store_wall","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0001087188720703125,"mean_abs_dp":0.00010871142148971558,"max_abs_dlogit":0.007025301456451416,"oracle_top2_margin":0.8891508653759956,"near_tie":false,"probs":[0.9446841478347778,0.055315859615802765]},{"question_id":"space_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0015328377485275269,"mean_abs_dp":0.0007264804095029831,"max_abs_dlogit":0.025713682174682617,"oracle_top2_margin":0.6871495172381401,"near_tie":false,"probs":[0.01853092759847641,0.10076291114091873,0.7882384061813354,0.07182487100362778,0.020642897114157677]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0015328377485275269,"mean_abs_dp":0.0002759862143327208,"max_abs_dlogit":0.025713682174682617},{"id":"ho_i08","arm":"g448","variant":"base","tokens":713,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6102373330213595,"wall_seconds":1.627214250009274,"finite":true,"index":35,"shard":"shard_00","first_in_process":false,"source":"heldout_image","hidden":{"max_abs_diff":2.0508594512939453,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.04244604481266107,"min_pos_cos":0.9960541894252992,"min_pos_cos_index":191,"mean_pos_cos":0.9998625697711806,"positions":713,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"biggest_space","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0017526745796203613,"mean_abs_dp":0.0007010672241449356,"max_abs_dlogit":0.03768801689147949,"oracle_top2_margin":0.7769975811243057,"near_tie":false,"probs":[0.06061577424407005,0.04876085743308067,0.8353481888771057,0.04061400145292282,0.01466115191578865]},{"question_id":"entrance","type":"choice","n_options":5,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0006411075592041016,"mean_abs_dp":0.00025645680725574496,"max_abs_dlogit":0.01356363296508789,"oracle_top2_margin":0.946435839869082,"near_tie":false,"probs":[0.957703709602356,0.0092157032340765,0.01200847513973713,0.008963084779679775,0.012109066359698772]},{"question_id":"wc_store_wall","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0008079223334789276,"mean_abs_dp":0.0008078422397375107,"max_abs_dlogit":0.057102084159851074,"oracle_top2_margin":0.9023991413414478,"near_tie":false,"probs":[0.9503917694091797,0.04960831254720688]},{"question_id":"space_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.002217411994934082,"mean_abs_dp":0.0010036841034889222,"max_abs_dlogit":0.02747631072998047,"oracle_top2_margin":0.7507094368338585,"near_tie":false,"probs":[0.017303112894296646,0.0715048611164093,0.8202886581420898,0.07194340229034424,0.01895998977124691]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.002217411994934082,"mean_abs_dp":0.0006718661855248844,"max_abs_dlogit":0.057102084159851074},{"id":"ho_p01","arm":"g256","variant":"base","tokens":634,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3444151250005234,"wall_seconds":1.3573604170233011,"finite":true,"index":36,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":6.875188827514648,"ref_absmax":48.31694412231445,"rel_max_abs_diff":0.14229353599246866,"min_pos_cos":0.9698986244816614,"min_pos_cos_index":135,"mean_pos_cos":0.9998393216101966,"positions":634,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"lamp_on","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0004308559000492096,"mean_abs_dp":0.0004308391362428665,"max_abs_dlogit":0.007059574127197266,"oracle_top2_margin":0.8864576034247875,"near_tie":false,"probs":[0.9427979588508606,0.05720203369855881]},{"question_id":"time_of_day","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.0004126988351345062,"mean_abs_dp":0.00020636466797441244,"max_abs_dlogit":0.012990951538085938,"oracle_top2_margin":0.9142597317695618,"near_tie":false,"probs":[0.006494088564068079,0.007571396417915821,0.035500649362802505,0.9504338502883911]},{"question_id":"sea_behind","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00017219781875610352,"mean_abs_dp":0.00017216801643371582,"max_abs_dlogit":0.0026221275329589844,"oracle_top2_margin":0.8596098273992538,"near_tie":false,"probs":[0.9296327233314514,0.070367231965065]},{"question_id":"tower_colour","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.00036400556564331055,"mean_abs_dp":0.00018196925520896912,"max_abs_dlogit":0.009798288345336914,"oracle_top2_margin":0.9164357371628284,"near_tie":false,"probs":[0.013715354725718498,0.028241464868187904,0.9442859888076782,0.013757149688899517]},{"question_id":"remoteness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0017777085304260254,"mean_abs_dp":0.0011851427455743153,"max_abs_dlogit":0.01725602149963379,"oracle_top2_margin":0.7929675057530403,"near_tie":false,"probs":[0.04248281195759773,0.08373820781707764,0.8737789988517761]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0017777085304260254,"mean_abs_dp":0.0004209852156539758,"max_abs_dlogit":0.01725602149963379},{"id":"ho_p01","arm":"g448","variant":"base","tokens":766,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.6126852500019595,"wall_seconds":1.629917790996842,"finite":true,"index":37,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":3.574766159057617,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.07398590112375092,"min_pos_cos":0.9885712588859206,"min_pos_cos_index":123,"mean_pos_cos":0.9998466713879639,"positions":766,"positions_below":{"0.99":1,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"lamp_on","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00039978697896003723,"mean_abs_dp":0.0003997776657342911,"max_abs_dlogit":0.007999062538146973,"oracle_top2_margin":0.8949422650039196,"near_tie":false,"probs":[0.947071373462677,0.05292866379022598]},{"question_id":"time_of_day","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00021288730204105377,"mean_abs_dp":0.0001064579701051116,"max_abs_dlogit":0.011775016784667969,"oracle_top2_margin":0.9366869516670704,"near_tie":false,"probs":[0.006314266938716173,0.007121984381228685,0.02478945441544056,0.9617742896080017]},{"question_id":"sea_behind","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00019600242376327515,"mean_abs_dp":0.00019596144556999207,"max_abs_dlogit":0.0036466121673583984,"oracle_top2_margin":0.8574466854333878,"near_tie":false,"probs":[0.9285274147987366,0.0714726522564888]},{"question_id":"tower_colour","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009598135948181152,"mean_abs_dp":0.00047990214079618454,"max_abs_dlogit":0.021100759506225586,"oracle_top2_margin":0.924212034791708,"near_tie":false,"probs":[0.01369630079716444,0.024745820090174675,0.947737991809845,0.013819864951074123]},{"question_id":"remoteness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.002624690532684326,"mean_abs_dp":0.0017497700949509938,"max_abs_dlogit":0.03529810905456543,"oracle_top2_margin":0.8101317957043648,"near_tie":false,"probs":[0.040591295808553696,0.07669833302497864,0.882710337638855]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.002624690532684326,"mean_abs_dp":0.0005857485967377822,"max_abs_dlogit":0.03529810905456543},{"id":"ho_p02","arm":"g256","variant":"base","tokens":620,"calls":10,"padded_tokens":640,"grid":[8,8],"n_image_rows":64,"start":101,"amount":56,"image_tokens":64,"decode_seconds":1.3434447500039823,"wall_seconds":1.3567330410005525,"finite":true,"index":38,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":1.7807307243347168,"ref_absmax":51.6200065612793,"rel_max_abs_diff":0.03449691007343772,"min_pos_cos":0.9963305370160053,"min_pos_cos_index":189,"mean_pos_cos":0.9999046204786579,"positions":620,"positions_below":{"0.99":0,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":0},"questions":[{"question_id":"petal_colour","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.001192629337310791,"mean_abs_dp":0.000596323749050498,"max_abs_dlogit":0.02589467167854309,"oracle_top2_margin":0.9628678802400827,"near_tie":false,"probs":[0.009779850952327251,0.009927202947437763,0.009107585065066814,0.9711853861808777]},{"question_id":"insect","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00011045485734939575,"mean_abs_dp":0.00011045113205909729,"max_abs_dlogit":0.021697282791137695,"oracle_top2_margin":0.937197607010603,"near_tie":false,"probs":[0.03151163086295128,0.968488335609436]},{"question_id":"centre_colour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.00046455860137939453,"mean_abs_dp":0.00023228605277836323,"max_abs_dlogit":0.010606348514556885,"oracle_top2_margin":0.9540995731949806,"near_tie":false,"probs":[0.008488254621624947,0.9672139883041382,0.013750395737588406,0.010547359474003315]},{"question_id":"setting","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00011616945266723633,"mean_abs_dp":5.808798596262932e-05,"max_abs_dlogit":0.00853586196899414,"oracle_top2_margin":0.9648665515705943,"near_tie":false,"probs":[0.9748742580413818,0.0074460469186306,0.007519132923334837,0.010160481557250023]},{"question_id":"density","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0008859038352966309,"mean_abs_dp":0.0005905876557032267,"max_abs_dlogit":0.004082083702087402,"oracle_top2_margin":0.37225431203842163,"near_tie":false,"probs":[0.06722158938646317,0.28103524446487427,0.6517431139945984]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.001192629337310791,"mean_abs_dp":0.0003258503754349316,"max_abs_dlogit":0.02589467167854309},{"id":"ho_p02","arm":"g448","variant":"base","tokens":752,"calls":12,"padded_tokens":768,"grid":[14,14],"n_image_rows":196,"start":233,"amount":182,"image_tokens":196,"decode_seconds":1.612805542012211,"wall_seconds":1.6298113340162672,"finite":true,"index":39,"shard":"shard_00","first_in_process":false,"source":"heldout_photo","hidden":{"max_abs_diff":8.259757280349731,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.17094980713682184,"min_pos_cos":0.9471616175950758,"min_pos_cos_index":324,"mean_pos_cos":0.9997736302539016,"positions":752,"positions_below":{"0.99":2,"0.9":0,"0.5":0},"positions_below_0.99_image_rows":1},"questions":[{"question_id":"petal_colour","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":0.00022739171981811523,"mean_abs_dp":0.00011367816478013992,"max_abs_dlogit":0.008183479309082031,"oracle_top2_margin":0.9763394580222666,"near_tie":false,"probs":[0.006078463047742844,0.006013424135744572,0.005800986662507057,0.9821071028709412]},{"question_id":"insect","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.764886736869812e-06,"mean_abs_dp":5.743466317653656e-06,"max_abs_dlogit":0.00031757354736328125,"oracle_top2_margin":0.9645282085984945,"near_tie":false,"probs":[0.01773012802004814,0.982269823551178]},{"question_id":"centre_colour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0003821849822998047,"mean_abs_dp":0.0001911052968353033,"max_abs_dlogit":0.013330578804016113,"oracle_top2_margin":0.9650700585916638,"near_tie":false,"probs":[0.006506036501377821,0.9751896858215332,0.010659362189471722,0.0076449294574558735]},{"question_id":"setting","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.00034695863723754883,"mean_abs_dp":0.00017351104179397225,"max_abs_dlogit":0.016878962516784668,"oracle_top2_margin":0.9797865096479654,"near_tie":false,"probs":[0.9854954481124878,0.004412717651575804,0.0039043896831572056,0.006187517661601305]},{"question_id":"density","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":0.0009236335754394531,"mean_abs_dp":0.0006157482663790385,"max_abs_dlogit":0.004041433334350586,"oracle_top2_margin":0.24045434594154358,"near_tie":false,"probs":[0.07862147688865662,0.3412059545516968,0.5801725387573242]}],"argmax_all_equal":true,"argmax_equal_non_near_tie":true,"max_abs_dp":0.0009236335754394531,"mean_abs_dp":0.00022187704384765203,"max_abs_dlogit":0.016878962516784668}],"compare_with":{"other":"<lane>/results/readout_heldout_fp16_pf64.json","other_chunk":64,"other_bundle":"clef_flash_decode_fp16_pf64","runs":40,"max_abs_dp":0.014504492282867432,"argmax_equal_runs":39,"hidden_max_abs_diff":5.21875,"vs_oracle_on_shared_runs":{"order":["other","this"],"max_abs_dp":[0.008473306894302368,0.017072200775146484],"min_pos_cos":[0.8900131817819378,0.8917932115996107],"positions_below_0.99":[15,14]},"timing_warm_common_runs":{"other":{"S":64,"runs":39,"ms_per_call_median":134.35833400581032,"ms_per_token_median":2.190111302479636,"run_decode_seconds_median":1.609269749984378,"calls_total":509},"this":{"S":64,"runs":39,"ms_per_call_median":134.3717919953633,"ms_per_token_median":2.186575867142295,"run_decode_seconds_median":1.6102373330213595,"calls_total":509}}},"note":"round 5 held-out: int8mix with layers 0-11 fp16 (chosen in round 4 outside the bisect rule, on fixture runs); these 30 records were never used to select it","generated_at":"2026-10-03T05:18:10+00:00"}}