{"what":"The decoder module (conversion/clef_flash/qwen3_5_clef_decoder.py) in fp32 torch against the author's fp32 oracle, read through the author's own JointSchemaHead: P1 every oracle run, P2 bit equality with the overlay's plain Qwen3.5 text decoder, P3 per layer, P4 red arms, P5 chunk widths 16 / 32 / 64, CPU vs MPS. Then the host spec (conversion/clef_flash/host.py) against every oracle run: ids, image offset, spans, option order, static inputs, M-RoPE planes, pixel_values.","sources":[{"file":"results/parity_decoder.json","bytes":568553,"sha256":"b048ce3f4a62abe0a922e891c3376b150df293415f46639ec51641c69e41fa9e"},{"file":"results/test_host.json","bytes":22505,"sha256":"048d57f092ca8ccf8ecc09dc09abf609877bbedd98179cbe56c9f446bf9759b3"}],"trimmed":{"removed":["p1.runs[].questions[].probs (the module's probabilities; each question keeps max / mean |dp|, max |dlogit|, the argmax pair and the oracle margin)","p1.runs[].layer_hashes (per chunk x layer hashes of 3 runs; p2 keeps chunk_layer_hashes_equal per run)","p1.runs[].npz (lane file names)"],"serialization":"compact json (no indentation)","full_copies":"the source files above, in the lane directory $ZOO_WORK_ROOT/_clefflash/ (not in the repository)"},"parity":{"schema":"clef-flash-decoder-torch-parity/1","purpose":"the hidden-output decoder module (fp32 torch, S-token chunks from fresh zero states, unrolled GDN step scan, in-graph M-RoPE, ids input with image rows on a static input) vs the author's fp32 oracle, read through the author's JointSchemaHead","module":{"path":"conversion/clef_flash/qwen3_5_clef_decoder.py","sha256":"6342008b9fde0a521aa4db5b7ac9e164c93086a7cae92ee19ba1c2e356f59b8f","class":"Qwen3_5ClefDecoder","parent":"conversion/decider_vision/qwen3_5_vl_pipelined.py","parent_sha256":"7f7fd56967f7ba4ca9e9ca12092205df638d6996470454cfc0845b495090211f","n_image_max":1024,"chunk":16,"load_report":{"checkpoint_text_keys":427,"checkpoint_other_keys_skipped":455,"module_tensors":426,"unread_checkpoint_keys":[],"module_tensors_not_in_checkpoint":[],"tied_lm_head":false,"intentionally_unread_keys":["lm_head.weight"],"lm_head_untied":true,"checkpoint_lm_head":{"file":"model-00001-of-00004.safetensors","dtype":"BF16","shape":[248320,4096]},"module_has_lm_head_weight":false}},"harness":{"path":"conversion/clef_flash/parity_decoder_torch.py","sha256":"ffc52299b66c85fe2030869b65261fdb350f40d34d5871755bb72eef5ffdb190","depthwise_conv":"CPU: one torch.bmm over the conv windows (no oneDNN in this torch build), checked against F.conv1d (torch.equal, all channels, all 24 layers) at the last chunk of every CPU run; MPS: F.conv1d","conv_check":{"processes":[{"pid":84441,"stages":["device"],"cpu_runs":5,"checked_runs":5,"conv":{"fast_calls":3384,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0}},{"pid":88754,"stages":["device"],"cpu_runs":0,"checked_runs":0,"conv":{"fast_calls":0,"checked_vs_F_conv1d":0,"mismatched":0,"max_abs_diff":0.0}},{"pid":90531,"stages":["p1","p4","p5"],"cpu_runs":31,"checked_runs":31,"conv":{"fast_calls":34056,"checked_vs_F_conv1d":744,"mismatched":0,"max_abs_diff":0.0}},{"pid":12921,"stages":["p1","p4","p5"],"cpu_runs":202,"checked_runs":38,"conv":{"fast_calls":92880,"checked_vs_F_conv1d":912,"mismatched":0,"max_abs_diff":0.0}},{"pid":null,"stages":["p2"],"cpu_runs":3,"checked_runs":3,"conv":{"fast_calls":1680,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0}}],"total":{"fast_calls":132000,"checked_vs_F_conv1d":1848,"mismatched":0,"checked_runs":77}},"invocations":[{"pid":84441,"started":"2026-10-03T09:53:05+09:00","stages":["device"],"threads":10,"load_seconds":7.827150083991},{"pid":88754,"started":"2026-10-03T09:56:48+09:00","stages":["device"],"threads":10,"load_seconds":4.43367841700092},{"pid":90531,"started":"2026-10-03T09:58:32+09:00","stages":["p1","p4","p5"],"threads":10,"load_seconds":1.480711624986725},{"pid":12921,"started":"2026-10-03T10:21:34+09:00","stages":["p1","p4","p5"],"threads":10,"load_seconds":7.111030500003835}]},"head":{"joint_schema_model_sha256":"0e304cf7c6500e8bb59bef7e2afd2c6373f82596dfb3b57d1aa93c175e2dc3a3","joint_head_safetensors_sha256":"19cdcec8c81dc9212be320fff47462ab342fbc1278be4368fb3da71241cf5ba0","lm_head_file":"model-00001-of-00004.safetensors","lm_head_shape":[248320,4096],"dtype":"float32 (bf16 checkpoint values cast up)","device":"cpu"},"oracle":{"path":"<lane>/oracle/records_oracle.json","sha256":"d8adaa746a5d575e654212ca78f5634af334aaa62eb53ef150d5d5925b1fa7c0","hf_id":"Cloudflare/clef-flash","revision":"17f0b0ad64efb65d273590632833508766b2aae6","versions":{"python":"3.12.11","torch":"2.9.0","torchvision":"0.24.0","transformers":"5.17.0","pillow":"12.3.0","numpy":"2.5.3","huggingface_hub":"1.33.0"},"image_embeds":"oracle npz image_embeds (HF fp32 tower merger output), zero-padded to the buffer"},"contract":{"input_names":["input_ids","position_ids","image_embeds","image_rc","rope_shift_start","rope_shift_amount"],"output":"hidden [1, S, 4096] (final norm, every position)","image":"<|image_pad|> -> V + k (row-major), image_rc[k] = (k // W, k % W), start = 37 + N, amount = N - max(H, W) (i0 = 36)","text":"image_embeds 0, image_rc 0, start 1 << 30, amount 0","chunks":"ceil(T / S) chunks from zero states, position_ids = 0..cursor+S-1, the last chunk padded with 248044, padded outputs discarded"},"bar":{"argmax_equal":"every question","max_abs_dp":0.0001,"min_pos_cos":0.9999},"versions":{"python":"3.11.13","torch":"2.9.0","transformers":"4.57.6","numpy":"2.3.5","platform":"macOS-27.0-arm64-arm-64bit","mkldnn":false,"coreai_torch":"0.4.1","overlay":{"path":"<coreai-models>","branch":"zoo-0.2","rev":"397b337","uncommitted_python":[" M python/src/coreai_models/model_registry.py"," M python/src/coreai_models/models/base.py"," M python/src/coreai_models/models/macos/lfm2.py"," M python/src/coreai_models/models/registry.py","?? python/src/coreai_models/models/ios/qwen3_5_mtp_ios.py","?? python/src/coreai_models/models/macos/bitcpm_ternary_gemm.py","?? python/src/coreai_models/models/macos/cohere_compass.py","?? python/src/coreai_models/models/macos/lfm2_vl.py","?? python/src/coreai_models/models/macos/muse_glimmer.py","?? python/src/coreai_models/models/macos/muse_glimmer_config_shim.py","?? python/src/coreai_models/models/macos/nanbeige.py","?? python/src/coreai_models/models/macos/qwen3_5_vision.py","?? python/src/coreai_models/models/macos/qwen3_8_mtp.py"],"sha256":{"models/macos/qwen3_5.py":"8e9da56ac2fad9c43017a5ca88316af49e9667960c4367cc3f00bca8038620b6","models/base.py":"5b47e8d362238d2ca242e7520386c9b879ce7653b554e2adb08de13c86754008","primitives/macos/cache.py":"c5a1fc439db3f6268f3be96974563b14ba80dcd802580ed09a37e3a08633cc01","primitives/_ops.py":"76680cc6a61d9f25d8ffed5754336f11bcf255cfc312a4d2a586513138dd5aa2"}}},"p1":{"result":"PASS","device_of_record":"cpu","summary":{"runs":214,"questions":405,"argmax_equal":405,"near_tie_questions":4,"max_abs_dp":9.59634780883789e-06,"worst_run":"semif_4aa83af95bb69e90bc2d","mean_run_mean_abs_dp":3.1519240184877734e-07,"mean_question_max_abs_dp":4.276041125441775e-07,"max_abs_dlogit":6.389617919921875e-05,"min_pos_cos":0.9999993875949071,"hidden_max_abs_diff":0.009288787841796875,"hidden_max_rel_diff":0.00018416874849204747,"rope_planes_equal_runs":214,"devices":["cpu"],"seconds_total":3623.798951959936,"expected_runs":214,"missing_runs":[],"head_floor_bit_equal_runs":214,"head_floor_max_abs_dlogit":0.0,"runs_by_device":{"cpu":214,"mps":0},"n_image_max_over_1024":["photo_01:native (1200)"]},"by_source_arm":{"cc0_photo/g256":{"runs":4,"questions":16,"argmax_equal":16,"near_tie_questions":1,"max_abs_dp":1.5497207641601562e-06,"worst_run":"photo_01","mean_run_mean_abs_dp":1.5270052244886756e-07,"mean_question_max_abs_dp":2.6711495593190193e-07,"max_abs_dlogit":5.125999450683594e-06,"min_pos_cos":0.9999999960609904,"hidden_max_abs_diff":0.0015048980712890625,"hidden_max_rel_diff":3.114644006059068e-05,"rope_planes_equal_runs":4,"devices":["cpu"],"seconds_total":86.98102920901147},"cc0_photo/g448":{"runs":4,"questions":16,"argmax_equal":16,"near_tie_questions":0,"max_abs_dp":9.834766387939453e-07,"worst_run":"photo_01","mean_run_mean_abs_dp":8.401865414224373e-08,"mean_question_max_abs_dp":1.61642674356699e-07,"max_abs_dlogit":3.337860107421875e-06,"min_pos_cos":0.9999999949185113,"hidden_max_abs_diff":0.0014996528625488281,"hidden_max_rel_diff":3.103782513133392e-05,"rope_planes_equal_runs":4,"devices":["cpu"],"seconds_total":107.97170754199033},"cc0_photo/native":{"runs":4,"questions":16,"argmax_equal":16,"near_tie_questions":0,"max_abs_dp":4.172325134277344e-07,"worst_run":"photo_01","mean_run_mean_abs_dp":5.048647066220068e-08,"mean_question_max_abs_dp":9.771611075848341e-08,"max_abs_dlogit":6.67572021484375e-06,"min_pos_cos":0.9999999942321458,"hidden_max_abs_diff":0.0024123191833496094,"hidden_max_rel_diff":4.9927072327797294e-05,"rope_planes_equal_runs":4,"devices":["cpu"],"seconds_total":176.43012991704745},"own_image/g256":{"runs":10,"questions":34,"argmax_equal":34,"near_tie_questions":1,"max_abs_dp":5.602836608886719e-06,"worst_run":"img_07","mean_run_mean_abs_dp":3.895637529537963e-07,"mean_question_max_abs_dp":6.080988575430478e-07,"max_abs_dlogit":2.5272369384765625e-05,"min_pos_cos":0.9999999884983474,"hidden_max_abs_diff":0.0021610260009765625,"hidden_max_rel_diff":4.4726048806106615e-05,"rope_planes_equal_runs":10,"devices":["cpu"],"seconds_total":260.92651600099634},"own_image/g448":{"runs":10,"questions":34,"argmax_equal":34,"near_tie_questions":1,"max_abs_dp":2.115964889526367e-06,"worst_run":"img_06","mean_run_mean_abs_dp":1.4544877225014383e-07,"mean_question_max_abs_dp":2.4408869007054495e-07,"max_abs_dlogit":9.5367431640625e-06,"min_pos_cos":0.9999999985911548,"hidden_max_abs_diff":0.0012140274047851562,"hidden_max_rel_diff":2.5126328389308794e-05,"rope_planes_equal_runs":10,"devices":["cpu"],"seconds_total":304.8883932089957},"own_image/native":{"runs":10,"questions":34,"argmax_equal":34,"near_tie_questions":1,"max_abs_dp":1.1324882507324219e-06,"worst_run":"img_08","mean_run_mean_abs_dp":1.389134411859994e-07,"mean_question_max_abs_dp":2.4669091491138233e-07,"max_abs_dlogit":1.0967254638671875e-05,"min_pos_cos":0.999999984147918,"hidden_max_abs_diff":0.0054264068603515625,"hidden_max_rel_diff":0.00011230877309553928,"rope_planes_equal_runs":10,"devices":["cpu"],"seconds_total":357.002038998995},"own_json/text":{"runs":12,"questions":45,"argmax_equal":45,"near_tie_questions":0,"max_abs_dp":3.4570693969726562e-06,"worst_run":"own_j12","mean_run_mean_abs_dp":4.066273031191869e-07,"mean_question_max_abs_dp":6.781580547491709e-07,"max_abs_dlogit":1.5020370483398438e-05,"min_pos_cos":0.9999999869770055,"hidden_max_abs_diff":0.00180816650390625,"hidden_max_rel_diff":3.7423099084207816e-05,"rope_planes_equal_runs":12,"devices":["cpu"],"seconds_total":488.04901433200575},"own_text/text":{"runs":13,"questions":50,"argmax_equal":50,"near_tie_questions":0,"max_abs_dp":1.3113021850585938e-06,"worst_run":"own_t07","mean_run_mean_abs_dp":1.5787659519229095e-07,"mean_question_max_abs_dp":2.3900531232357025e-07,"max_abs_dlogit":2.1249055862426758e-05,"min_pos_cos":0.9999999968941463,"hidden_max_abs_diff":0.0012989044189453125,"hidden_max_rel_diff":2.6883049025680932e-05,"rope_planes_equal_runs":13,"devices":["cpu"],"seconds_total":330.4681236690085},"own_text_long/text":{"runs":3,"questions":16,"argmax_equal":16,"near_tie_questions":0,"max_abs_dp":1.817941665649414e-06,"worst_run":"own_t15","mean_run_mean_abs_dp":1.4330572629390022e-07,"mean_question_max_abs_dp":2.847227733582258e-07,"max_abs_dlogit":3.9577484130859375e-05,"min_pos_cos":0.9999999948432337,"hidden_max_abs_diff":0.0016231536865234375,"hidden_max_rel_diff":3.0379555237599538e-05,"rope_planes_equal_runs":3,"devices":["cpu"],"seconds_total":377.99356137699215},"semif_authored144/text":{"runs":144,"questions":144,"argmax_equal":144,"near_tie_questions":0,"max_abs_dp":9.59634780883789e-06,"worst_run":"semif_4aa83af95bb69e90bc2d","mean_run_mean_abs_dp":3.625086888119026e-07,"mean_question_max_abs_dp":5.581338579456011e-07,"max_abs_dlogit":6.389617919921875e-05,"min_pos_cos":0.9999993875949071,"hidden_max_abs_diff":0.009288787841796875,"hidden_max_rel_diff":0.00018416874849204747,"rope_planes_equal_runs":144,"devices":["cpu"],"seconds_total":1133.0884377048933}},"runs":[{"kind":"p1","id":"own_t01","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":436,"padded_tokens":448,"chunks":28,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":22.52041866601212,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012989044189453125,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.6883049025680932e-05,"min_pos_cos":0.9999999983907859,"min_pos_cos_index":402,"mean_pos_cos":0.9999999999827827},"rope_planes_equal_oracle":true,"questions":[{"question_id":"department","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":4.190951585769653e-08,"mean_abs_dp":1.3387762010097504e-08,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.9682950582355261,"near_tie":false},{"question_id":"wants_refund","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_abs_dp":9.778887033462524e-09,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.9727394469082355,"near_tie":false},{"question_id":"urgency","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_abs_dp":6.016343832015991e-07,"max_abs_dlogit":6.273388862609863e-06,"oracle_top2_margin":0.07407951354980469,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_question_max_abs_dp":4.377216100692749e-07,"mean_abs_dp":2.4796463549137115e-07,"conv":{"fast_calls":672,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"p3":{"id":"own_t01","arm":"text","tokens":436,"stages":[{"stage":"layer-0 input (hs[0])","max_abs_diff":0.0,"ref_absmax":0.462890625,"min_pos_cos":0.9999999999999998,"mean_pos_cos":1.0},{"stage":"layer 0 output (hs[1])","max_abs_diff":3.4332275390625e-05,"ref_absmax":22.78901481628418,"min_pos_cos":0.9999999999996432,"mean_pos_cos":0.9999999999999802},{"stage":"layer 1 output (hs[2])","max_abs_diff":3.24249267578125e-05,"ref_absmax":31.73855209350586,"min_pos_cos":0.9999999999997372,"mean_pos_cos":0.9999999999999285},{"stage":"layer 2 output (hs[3])","max_abs_diff":5.14984130859375e-05,"ref_absmax":32.855560302734375,"min_pos_cos":0.9999999999988266,"mean_pos_cos":0.9999999999997875},{"stage":"layer 3 output (hs[4])","max_abs_diff":4.9591064453125e-05,"ref_absmax":30.969675064086914,"min_pos_cos":0.9999999999988156,"mean_pos_cos":0.9999999999997821},{"stage":"layer 4 output (hs[5])","max_abs_diff":4.57763671875e-05,"ref_absmax":28.546762466430664,"min_pos_cos":0.9999999999978126,"mean_pos_cos":0.9999999999996794},{"stage":"layer 5 output (hs[6])","max_abs_diff":3.814697265625e-05,"ref_absmax":28.99304962158203,"min_pos_cos":0.9999999999974135,"mean_pos_cos":0.9999999999995526},{"stage":"layer 6 output (hs[7])","max_abs_diff":3.814697265625e-05,"ref_absmax":63.745628356933594,"min_pos_cos":0.9999999999967402,"mean_pos_cos":0.9999999999993479},{"stage":"layer 7 output (hs[8])","max_abs_diff":3.62396240234375e-05,"ref_absmax":44.75798416137695,"min_pos_cos":0.9999999999963658,"mean_pos_cos":0.999999999999158},{"stage":"layer 8 output (hs[9])","max_abs_diff":2.6702880859375e-05,"ref_absmax":42.81202697753906,"min_pos_cos":0.9999999999960273,"mean_pos_cos":0.9999999999989657},{"stage":"layer 9 output (hs[10])","max_abs_diff":2.47955322265625e-05,"ref_absmax":43.58789825439453,"min_pos_cos":0.9999999999940886,"mean_pos_cos":0.9999999999988645},{"stage":"layer 10 output (hs[11])","max_abs_diff":2.6702880859375e-05,"ref_absmax":68.33386993408203,"min_pos_cos":0.9999999999921526,"mean_pos_cos":0.9999999999987395},{"stage":"layer 11 output (hs[12])","max_abs_diff":2.6702880859375e-05,"ref_absmax":67.96086883544922,"min_pos_cos":0.999999999986045,"mean_pos_cos":0.9999999999983245},{"stage":"layer 12 output (hs[13])","max_abs_diff":2.47955322265625e-05,"ref_absmax":67.7262191772461,"min_pos_cos":0.9999999999725822,"mean_pos_cos":0.9999999999979051},{"stage":"layer 13 output (hs[14])","max_abs_diff":3.62396240234375e-05,"ref_absmax":67.68138885498047,"min_pos_cos":0.9999999999596528,"mean_pos_cos":0.9999999999974938},{"stage":"layer 14 output (hs[15])","max_abs_diff":7.05718994140625e-05,"ref_absmax":74.43360137939453,"min_pos_cos":0.9999999999269855,"mean_pos_cos":0.9999999999969025},{"stage":"layer 15 output (hs[16])","max_abs_diff":0.0001068115234375,"ref_absmax":70.99459075927734,"min_pos_cos":0.9999999998343649,"mean_pos_cos":0.9999999999955997},{"stage":"layer 16 output (hs[17])","max_abs_diff":0.0001010894775390625,"ref_absmax":69.69348907470703,"min_pos_cos":0.999999999763129,"mean_pos_cos":0.9999999999941982},{"stage":"layer 17 output (hs[18])","max_abs_diff":0.000202178955078125,"ref_absmax":68.98184204101562,"min_pos_cos":0.9999999996825502,"mean_pos_cos":0.99999999999293},{"stage":"layer 18 output (hs[19])","max_abs_diff":9.72747802734375e-05,"ref_absmax":79.57122802734375,"min_pos_cos":0.9999999995659704,"mean_pos_cos":0.9999999999903293},{"stage":"layer 19 output (hs[20])","max_abs_diff":0.0001621246337890625,"ref_absmax":53.280818939208984,"min_pos_cos":0.9999999995877067,"mean_pos_cos":0.9999999999884926},{"stage":"layer 20 output (hs[21])","max_abs_diff":0.000213623046875,"ref_absmax":59.358970642089844,"min_pos_cos":0.9999999994859263,"mean_pos_cos":0.9999999999865072},{"stage":"layer 21 output (hs[22])","max_abs_diff":0.0002422332763671875,"ref_absmax":67.45379638671875,"min_pos_cos":0.999999999395574,"mean_pos_cos":0.999999999984558},{"stage":"layer 22 output (hs[23])","max_abs_diff":0.000537872314453125,"ref_absmax":157.20408630371094,"min_pos_cos":0.9999999990732287,"mean_pos_cos":0.9999999999804116},{"stage":"layer 23 output (hs[24])","max_abs_diff":0.0005130767822265625,"ref_absmax":120.97592163085938,"min_pos_cos":0.99999999883224,"mean_pos_cos":0.999999999978108},{"stage":"layer 24 output (hs[25])","max_abs_diff":0.00139617919921875,"ref_absmax":126.42041015625,"min_pos_cos":0.9999999986516616,"mean_pos_cos":0.9999999999758205},{"stage":"layer 25 output (hs[26])","max_abs_diff":0.00225067138671875,"ref_absmax":138.5946807861328,"min_pos_cos":0.9999999982792424,"mean_pos_cos":0.9999999999728995},{"stage":"layer 26 output (hs[27])","max_abs_diff":0.00191497802734375,"ref_absmax":294.0437927246094,"min_pos_cos":0.9999999977242038,"mean_pos_cos":0.9999999999679544},{"stage":"layer 27 output (hs[28])","max_abs_diff":0.00304412841796875,"ref_absmax":153.15187072753906,"min_pos_cos":0.9999999971634439,"mean_pos_cos":0.9999999999660923},{"stage":"layer 28 output (hs[29])","max_abs_diff":0.003292083740234375,"ref_absmax":159.9530792236328,"min_pos_cos":0.9999999972070555,"mean_pos_cos":0.9999999999652416},{"stage":"layer 29 output (hs[30])","max_abs_diff":0.00281524658203125,"ref_absmax":165.33294677734375,"min_pos_cos":0.9999999974374257,"mean_pos_cos":0.9999999999668803},{"stage":"layer 30 output (hs[31])","max_abs_diff":0.003017425537109375,"ref_absmax":162.252685546875,"min_pos_cos":0.9999999972160991,"mean_pos_cos":0.9999999999661379},{"stage":"layer 31 output (hs[32])","max_abs_diff":0.01593017578125,"ref_absmax":2268.877197265625,"min_pos_cos":0.9999999992441753,"mean_pos_cos":0.999999999988716},{"stage":"final norm (last_hidden)","max_abs_diff":0.0012989044189453125,"ref_absmax":48.316856384277344,"min_pos_cos":0.9999999983907859,"mean_pos_cos":0.9999999999827827}],"worst_max_abs_diff":{"value":0.01593017578125,"stage":"layer 31 output (hs[32])"},"worst_min_pos_cos":{"value":0.9999999971634439,"stage":"layer 27 output (hs[28])"}}},{"kind":"p1","id":"own_t02","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":599,"padded_tokens":608,"chunks":38,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":28.668268334004097,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006289482116699219,"ref_absmax":51.46957778930664,"rel_max_abs_diff":1.221980514867547e-05,"min_pos_cos":0.9999999968941463,"min_pos_cos_index":479,"mean_pos_cos":0.9999999999811368},"rope_planes_equal_oracle":true,"questions":[{"question_id":"severity","type":"score","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_abs_dp":3.3974647521972657e-07,"max_abs_dlogit":4.231929779052734e-06,"oracle_top2_margin":0.1987611949443817,"near_tie":false},{"question_id":"resolved","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.3527612686157227e-08,"mean_abs_dp":1.6763806343078613e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.9355993382632732,"near_tie":false},{"question_id":"root_cause","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.880130290985107e-08,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.9514973126351833,"near_tie":false},{"question_id":"data_loss","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":2.7939677238464355e-09,"max_abs_dlogit":7.152557373046875e-07,"oracle_top2_margin":0.9881136221811175,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_question_max_abs_dp":2.3329630494117737e-07,"mean_abs_dp":1.4156103134155273e-07,"conv":{"fast_calls":912,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t03","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":463,"padded_tokens":464,"chunks":29,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":22.767689917003736,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005130767822265625,"ref_absmax":56.47786331176758,"rel_max_abs_diff":9.084564325571063e-06,"min_pos_cos":0.9999999993475877,"min_pos_cos_index":421,"mean_pos_cos":0.9999999999714199},"rope_planes_equal_oracle":true,"questions":[{"question_id":"new_slot","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.07222318649292e-07,"mean_abs_dp":4.138176639874776e-07,"max_abs_dlogit":1.0967254638671875e-05,"oracle_top2_margin":0.8634682968258858,"near_tie":false},{"question_id":"thursday_still_on","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":5.960464477539063e-08,"max_abs_dlogit":4.112720489501953e-06,"oracle_top2_margin":0.8205529078841209,"near_tie":false},{"question_id":"reply_needed","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.8312206268310547e-07,"mean_abs_dp":2.60770320892334e-07,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.5112259685993195,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.07222318649292e-07,"mean_question_max_abs_dp":3.166496753692627e-07,"mean_abs_dp":2.6888613189969745e-07,"conv":{"fast_calls":1608,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t04","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":493,"padded_tokens":496,"chunks":31,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":27.405962916993303,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000469207763671875,"ref_absmax":48.97331619262695,"rel_max_abs_diff":9.580886085523351e-06,"min_pos_cos":0.9999999998247067,"min_pos_cos_index":234,"mean_pos_cos":0.9999999999852306},"rope_planes_equal_oracle":true,"questions":[{"question_id":"dark_mode","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.4458935260772705e-08,"mean_abs_dp":2.0644317070643108e-08,"max_abs_dlogit":3.635883331298828e-06,"oracle_top2_margin":0.9783774577081203,"near_tie":false},{"question_id":"release_note_owner","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":1.1175870895385742e-08,"max_abs_dlogit":2.5331974029541016e-06,"oracle_top2_margin":0.9677823921665549,"near_tie":false},{"question_id":"open_items","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7695128917694092e-08,"mean_abs_dp":8.071462313334147e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9687048075720668,"near_tie":false},{"question_id":"beta_label_decided","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.9185245037078857e-07,"max_abs_dlogit":4.351139068603516e-06,"oracle_top2_margin":0.9487586636096239,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":7.82310962677002e-08,"mean_abs_dp":4.5761804689060555e-08,"conv":{"fast_calls":2352,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t05","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":553,"padded_tokens":560,"chunks":35,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":27.22100562500418,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00040435791015625,"ref_absmax":48.316856384277344,"rel_max_abs_diff":8.368878698156179e-06,"min_pos_cos":0.9999999995155993,"min_pos_cos_index":303,"mean_pos_cos":0.9999999999891169},"rope_planes_equal_oracle":true,"questions":[{"question_id":"sentiment","type":"score","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.470348358154297e-07,"mean_abs_dp":1.4319084584712982e-07,"max_abs_dlogit":2.9802322387695312e-06,"oracle_top2_margin":0.28853923082351685,"near_tie":false},{"question_id":"safety_issue","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9395809434354305,"near_tie":false},{"question_id":"main_complaint","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.514570951461792e-08,"mean_abs_dp":1.280568540096283e-08,"max_abs_dlogit":2.562999725341797e-06,"oracle_top2_margin":0.9631505683064461,"near_tie":false},{"question_id":"recommend","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.725290298461914e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":8.344650268554688e-07,"oracle_top2_margin":0.9594815410673618,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.470348358154297e-07,"mean_question_max_abs_dp":1.227017492055893e-07,"mean_abs_dp":6.044641710244693e-08,"conv":{"fast_calls":3192,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t06","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":411,"padded_tokens":416,"chunks":26,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":20.21774079199531,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007658004760742188,"ref_absmax":51.968788146972656,"rel_max_abs_diff":1.4735777057345698e-05,"min_pos_cos":0.9999999982397317,"min_pos_cos_index":196,"mean_pos_cos":0.999999999977312},"rope_planes_equal_oracle":true,"questions":[{"question_id":"environment","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":1.1175870895385742e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9661621991544962,"near_tie":false},{"question_id":"fixed","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":2.1606683731079102e-07,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.8154464289546013,"near_tie":false},{"question_id":"urgency","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_abs_dp":5.066394805908203e-07,"max_abs_dlogit":4.649162292480469e-06,"oracle_top2_margin":0.5267177522182465,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_question_max_abs_dp":3.452102343241374e-07,"mean_abs_dp":2.2186173333062067e-07,"conv":{"fast_calls":3816,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t07","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":438,"padded_tokens":448,"chunks":28,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":21.770674208994024,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005464553833007812,"ref_absmax":51.89096450805664,"rel_max_abs_diff":1.053083881714972e-05,"min_pos_cos":0.9999999996173567,"min_pos_cos_index":100,"mean_pos_cos":0.9999999999810713},"rope_planes_equal_oracle":true,"questions":[{"question_id":"best_slot","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":5.960464477539062e-07,"mean_abs_dp":2.573554714520772e-07,"max_abs_dlogit":6.67572021484375e-06,"oracle_top2_margin":0.603644922375679,"near_tie":false},{"question_id":"length","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.259629011154175e-09,"mean_abs_dp":1.57160684466362e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9857203867286444,"near_tie":false},{"question_id":"needs_travel","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.3113021850585938e-06,"mean_abs_dp":1.296401023864746e-06,"max_abs_dlogit":8.58306884765625e-06,"oracle_top2_margin":0.7578679099678993,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.3113021850585938e-06,"mean_question_max_abs_dp":6.368694206078848e-07,"mean_abs_dp":3.4526844198505085e-07,"conv":{"fast_calls":4488,"checked_vs_F_conv1d":144,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t08","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":604,"padded_tokens":608,"chunks":38,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":27.00685491701006,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004665851593017578,"ref_absmax":50.80668640136719,"rel_max_abs_diff":9.183538473967516e-06,"min_pos_cos":0.9999999997134826,"min_pos_cos_index":571,"mean_pos_cos":0.9999999999895792},"rope_planes_equal_oracle":true,"questions":[{"question_id":"diet","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":5.432715018590291e-09,"max_abs_dlogit":8.940696716308594e-07,"oracle_top2_margin":0.9310548081994057,"near_tie":false},{"question_id":"needs_cooking","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.190951585769653e-09,"mean_abs_dp":2.0954757928848267e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9901977824047208,"near_tie":false},{"question_id":"effort","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.2782554626464844e-07,"mean_abs_dp":1.915420095125834e-07,"max_abs_dlogit":2.2649765014648438e-06,"oracle_top2_margin":0.7295103967189789,"near_tie":false},{"question_id":"servings","type":"choice","n_options":10,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.171153366565704e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.9073009714484215,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.2782554626464844e-07,"mean_question_max_abs_dp":1.1653173714876175e-07,"mean_abs_dp":5.067947010199229e-08,"conv":{"fast_calls":5400,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t09","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":582,"padded_tokens":592,"chunks":37,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":26.98823254101444,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0011413097381591797,"ref_absmax":50.3668098449707,"rel_max_abs_diff":2.2659956857941508e-05,"min_pos_cos":0.9999999992996386,"min_pos_cos_index":493,"mean_pos_cos":0.9999999999843698},"rope_planes_equal_oracle":true,"questions":[{"question_id":"department","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_abs_dp":1.2689270079135895e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9589813947677612,"near_tie":false},{"question_id":"duplicate_charge","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_abs_dp":9.313225746154785e-09,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.9166040010750294,"near_tie":false},{"question_id":"refund_amount","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.073364496231079e-08,"mean_abs_dp":1.924733320871989e-08,"max_abs_dlogit":3.2782554626464844e-06,"oracle_top2_margin":0.9625400472432375,"near_tie":false},{"question_id":"frustration","type":"score","n_options":7,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":5.662441253662109e-07,"mean_abs_dp":1.6391277313232422e-07,"max_abs_dlogit":4.5299530029296875e-06,"oracle_top2_margin":0.11354130506515503,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.662441253662109e-07,"mean_question_max_abs_dp":1.587904989719391e-07,"mean_abs_dp":7.965718396008015e-08,"conv":{"fast_calls":6288,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t10","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":396,"padded_tokens":400,"chunks":25,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":20.253834167000605,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005092620849609375,"ref_absmax":52.580326080322266,"rel_max_abs_diff":9.685411311123923e-06,"min_pos_cos":0.9999999998266753,"min_pos_cos_index":209,"mean_pos_cos":0.9999999999842691},"rope_planes_equal_oracle":true,"questions":[{"question_id":"meets_conditions","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.917383193969727e-07,"mean_abs_dp":4.246830940246582e-07,"max_abs_dlogit":2.1249055862426758e-05,"oracle_top2_margin":0.8212317898869514,"near_tie":false},{"question_id":"policy_issue","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.0046870708465576e-08,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.9324060156941414,"near_tie":false},{"question_id":"review","type":"score","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_abs_dp":2.682209014892578e-07,"max_abs_dlogit":6.198883056640625e-06,"oracle_top2_margin":0.8901551589369774,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.917383193969727e-07,"mean_question_max_abs_dp":2.8312206268310547e-07,"mean_abs_dp":1.932494342327118e-07,"conv":{"fast_calls":6888,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t11","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":497,"padded_tokens":512,"chunks":32,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":24.866463541984558,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004887580871582031,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.0115684747181702e-05,"min_pos_cos":0.9999999993780108,"min_pos_cos_index":207,"mean_pos_cos":0.9999999999856894},"rope_planes_equal_oracle":true,"questions":[{"question_id":"encrypted_at_rest","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.0803341865539551e-07,"max_abs_dlogit":6.794929504394531e-06,"oracle_top2_margin":0.9161899387836456,"near_tie":false},{"question_id":"pentest_current","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.4330954551696777e-07,"mean_abs_dp":4.302710294723511e-07,"max_abs_dlogit":1.71661376953125e-05,"oracle_top2_margin":0.8770000450313091,"near_tie":false},{"question_id":"risk","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.76837158203125e-07,"mean_abs_dp":2.146698534488678e-07,"max_abs_dlogit":6.67572021484375e-06,"oracle_top2_margin":0.6709898114204407,"near_tie":false},{"question_id":"next_step","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.767610708872478e-08,"max_abs_dlogit":8.463859558105469e-06,"oracle_top2_margin":0.9672962995246053,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.76837158203125e-07,"mean_question_max_abs_dp":2.896413207054138e-07,"mean_abs_dp":1.9439242102883078e-07,"conv":{"fast_calls":7656,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t12","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":967,"padded_tokens":976,"chunks":61,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":41.62137870900915,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0011663436889648438,"ref_absmax":49.259700775146484,"rel_max_abs_diff":2.367744161274547e-05,"min_pos_cos":0.9999999986489896,"min_pos_cos_index":813,"mean_pos_cos":0.9999999999763205},"rope_planes_equal_oracle":true,"questions":[{"question_id":"dairy_allergy","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_abs_dp":2.2351741790771484e-08,"max_abs_dlogit":5.0067901611328125e-06,"oracle_top2_margin":0.87893320992589,"near_tie":false},{"question_id":"guests","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5390257835388184e-08,"mean_abs_dp":1.800556977589925e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.8907695040106773,"near_tie":false},{"question_id":"split_bill","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.129243850708008e-07,"mean_abs_dp":2.4586915969848633e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.8779526241123676,"near_tie":false},{"question_id":"split_ways","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.122274160385132e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9427576903253794,"near_tie":false},{"question_id":"occasion","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_abs_dp":8.381903171539307e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8802992664277554,"near_tie":false},{"question_id":"kitchen_effort","type":"score","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":8.046627044677734e-07,"mean_abs_dp":4.917383193969727e-07,"max_abs_dlogit":7.152557373046875e-06,"oracle_top2_margin":0.27742141485214233,"near_tie":false},{"question_id":"dessert_planned","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":2.3096799850463867e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.6719503849744797,"near_tie":false},{"question_id":"child_meal","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.4435499906539917e-08,"mean_abs_dp":7.566995918750763e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9666099837049842,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":8.046627044677734e-07,"mean_question_max_abs_dp":1.9866274669766426e-07,"mean_abs_dp":1.3072236344732087e-07,"conv":{"fast_calls":9120,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t13","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text","tokens":370,"padded_tokens":384,"chunks":24,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":19.159599332982907,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009021759033203125,"ref_absmax":52.49782180786133,"rel_max_abs_diff":1.718501591594064e-05,"min_pos_cos":0.9999999997976327,"min_pos_cos_index":125,"mean_pos_cos":0.9999999999872566},"rope_planes_equal_oracle":true,"questions":[{"question_id":"car_park_open_june_4","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_abs_dp":3.725290298461914e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9685916639864445,"near_tie":false},{"question_id":"street_parking","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.270905335744222e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9700513584539294,"near_tie":false},{"question_id":"disruption","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.2665987014770508e-07,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.6014458537101746,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.0182460149129231e-07,"mean_abs_dp":7.194466888904572e-08,"conv":{"fast_calls":9696,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t14","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text_long","tokens":2389,"padded_tokens":2400,"chunks":150,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":93.57261433399981,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008497238159179688,"ref_absmax":48.31694412231445,"rel_max_abs_diff":1.7586456083954543e-05,"min_pos_cos":0.9999999989716162,"min_pos_cos_index":1035,"mean_pos_cos":0.9999999999876208},"rope_planes_equal_oracle":true,"questions":[{"question_id":"first_failing_component","type":"choice","n_options":5,"argmax":4,"argmax_oracle":4,"argmax_equal":true,"max_abs_dp":1.4435499906539917e-08,"mean_abs_dp":7.82310962677002e-09,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9446960557252169,"near_tie":false},{"question_id":"recovered","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.5273690223693848e-07,"mean_abs_dp":1.3597309589385986e-07,"max_abs_dlogit":3.516674041748047e-06,"oracle_top2_margin":0.9060924127697945,"near_tie":false},{"question_id":"severity","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":1.424923539161682e-07,"max_abs_dlogit":4.291534423828125e-06,"oracle_top2_margin":0.6201646775007248,"near_tie":false},{"question_id":"error_window","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.4016709327697754e-08,"max_abs_dlogit":5.602836608886719e-06,"oracle_top2_margin":0.6222178637981415,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":1.6100239008665085e-07,"mean_abs_dp":7.313986619313558e-08,"conv":{"fast_calls":13296,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t15","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text_long","tokens":2603,"padded_tokens":2608,"chunks":163,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":101.2419649589865,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0016231536865234375,"ref_absmax":53.42914581298828,"rel_max_abs_diff":3.0379555237599538e-05,"min_pos_cos":0.9999999948432337,"min_pos_cos_index":1300,"mean_pos_cos":0.9999999999832122},"rope_planes_equal_oracle":true,"questions":[{"question_id":"non_renewal_notice","type":"choice","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.210719347000122e-08,"mean_abs_dp":6.658956408500672e-09,"max_abs_dlogit":9.059906005859375e-06,"oracle_top2_margin":0.9844898809678853,"near_tie":false},{"question_id":"auto_renews","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.73225998878479e-07,"max_abs_dlogit":9.179115295410156e-06,"oracle_top2_margin":0.8997148089110851,"near_tie":false},{"question_id":"liability_cap","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.452924132347107e-08,"max_abs_dlogit":1.4901161193847656e-05,"oracle_top2_margin":0.9729512641206384,"near_tie":false},{"question_id":"assign_without_consent","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":9.12696123123169e-08,"mean_abs_dp":4.563480615615845e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9651009291410446,"near_tie":false},{"question_id":"client_risk","type":"score","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.817941665649414e-06,"mean_abs_dp":7.368624210357666e-07,"max_abs_dlogit":1.3589859008789062e-05,"oracle_top2_margin":0.11966484785079956,"near_tie":false},{"question_id":"uptime_target","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.8533471524715424e-08,"max_abs_dlogit":5.9604644775390625e-06,"oracle_top2_margin":0.9738647546619177,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.817941665649414e-06,"mean_question_max_abs_dp":4.0962671240170795e-07,"mean_abs_dp":2.1307178857651624e-07,"conv":{"fast_calls":17208,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_t16","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_text_long","tokens":2494,"padded_tokens":2496,"chunks":156,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":183.17898208400584,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008411407470703125,"ref_absmax":52.57929611206055,"rel_max_abs_diff":1.599756575815729e-05,"min_pos_cos":0.9999999959787249,"min_pos_cos_index":1582,"mean_pos_cos":0.9999999999839785},"rope_planes_equal_oracle":true,"questions":[{"question_id":"water_decision","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.789318725466728e-08,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.9877108454238623,"near_tie":false},{"question_id":"fees_increase","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.845220923423767e-08,"max_abs_dlogit":3.0994415283203125e-06,"oracle_top2_margin":0.9798774998635054,"near_tie":false},{"question_id":"new_chair","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.0489096641540527e-08,"mean_abs_dp":1.257285475730896e-08,"max_abs_dlogit":8.106231689453125e-06,"oracle_top2_margin":0.9809060669504106,"near_tie":false},{"question_id":"water_vote_consensus","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":9.5367431640625e-07,"mean_abs_dp":4.780013114213943e-07,"max_abs_dlogit":5.0067901611328125e-06,"oracle_top2_margin":0.7942192256450653,"near_tie":false},{"question_id":"waiting_list_order","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-09,"mean_abs_dp":1.5522042910257976e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.986916062887758,"near_tie":false},{"question_id":"bonfire_ban","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":2.3748725652694702e-07,"max_abs_dlogit":3.9577484130859375e-05,"oracle_top2_margin":0.9465006571263075,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.5367431640625e-07,"mean_question_max_abs_dp":2.42299089829127e-07,"mean_abs_dp":1.437055241120489e-07,"conv":{"fast_calls":20952,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j01","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":476,"padded_tokens":480,"chunks":30,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":22.376559665979585,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000514984130859375,"ref_absmax":49.30904769897461,"rel_max_abs_diff":1.0444008856210058e-05,"min_pos_cos":0.9999999999120478,"min_pos_cos_index":295,"mean_pos_cos":0.999999999991666},"rope_planes_equal_oracle":true,"questions":[{"question_id":"status","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.076879799365997e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.83003219217062,"near_tie":false},{"question_id":"large","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.3283064365386963e-09,"mean_abs_dp":1.1641532182693481e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9913223842158914,"near_tie":false},{"question_id":"priority","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_abs_dp":7.506459951400757e-07,"max_abs_dlogit":7.748603820800781e-06,"oracle_top2_margin":0.19878698885440826,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_question_max_abs_dp":4.577450454235077e-07,"mean_abs_dp":3.2479874789714813e-07,"conv":{"fast_calls":1392,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j02","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":475,"padded_tokens":480,"chunks":30,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":49.210452667000936,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006933212280273438,"ref_absmax":48.36116027832031,"rel_max_abs_diff":1.4336323281684181e-05,"min_pos_cos":0.9999999998611446,"min_pos_cos_index":146,"mean_pos_cos":0.9999999999908091},"rope_planes_equal_oracle":true,"questions":[{"question_id":"can_ship","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9560996349900961,"near_tie":false},{"question_id":"blocker","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.725290298461914e-08,"mean_abs_dp":1.0244548320770264e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9330523833632469,"near_tie":false},{"question_id":"gift","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.0803341865539551e-07,"max_abs_dlogit":1.2516975402832031e-06,"oracle_top2_margin":0.9137708581984043,"near_tie":false},{"question_id":"priority","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.450580596923828e-07,"mean_abs_dp":5.165735880533854e-07,"max_abs_dlogit":3.0994415283203125e-06,"oracle_top2_margin":0.18996360898017883,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.450580596923828e-07,"mean_question_max_abs_dp":2.2910535335540771e-07,"mean_abs_dp":1.6560608690435236e-07,"conv":{"fast_calls":21672,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j03","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":585,"padded_tokens":592,"chunks":37,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":54.53275345801376,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005512237548828125,"ref_absmax":51.966583251953125,"rel_max_abs_diff":1.060727337431974e-05,"min_pos_cos":0.9999999997565922,"min_pos_cos_index":147,"mean_pos_cos":0.9999999999885058},"rope_planes_equal_oracle":true,"questions":[{"question_id":"over_limit","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.0989606380462646e-07,"max_abs_dlogit":6.020069122314453e-06,"oracle_top2_margin":0.9472900442779064,"near_tie":false},{"question_id":"first_violation","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.091990947723389e-08,"max_abs_dlogit":2.562999725341797e-06,"oracle_top2_margin":0.9380843341350555,"near_tie":false},{"question_id":"trend","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.071462313334148e-08,"max_abs_dlogit":5.185604095458984e-06,"oracle_top2_margin":0.9589130245149136,"near_tie":false},{"question_id":"action","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-06,"mean_abs_dp":1.126900315284729e-06,"max_abs_dlogit":9.715557098388672e-06,"oracle_top2_margin":0.053304851055145264,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2351741790771484e-06,"mean_question_max_abs_dp":6.332993507385254e-07,"mean_abs_dp":3.6600977182388306e-07,"conv":{"fast_calls":22560,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j04","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":569,"padded_tokens":576,"chunks":36,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":47.76597037501051,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005497932434082031,"ref_absmax":48.31694412231445,"rel_max_abs_diff":1.137889105768776e-05,"min_pos_cos":0.9999999997839516,"min_pos_cos_index":329,"mean_pos_cos":0.9999999999904216},"rope_planes_equal_oracle":true,"questions":[{"question_id":"duration","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.9802322387695312e-06,"mean_abs_dp":1.5776604413986206e-06,"max_abs_dlogit":9.47713851928711e-06,"oracle_top2_margin":0.10314184427261353,"near_tie":false},{"question_id":"recurring","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.3504177331924438e-08,"mean_abs_dp":6.752088665962219e-09,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.9881622050888836,"near_tie":false},{"question_id":"accepted","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":1.4435499906539917e-07,"max_abs_dlogit":1.2159347534179688e-05,"oracle_top2_margin":0.8471281565725803,"near_tie":false},{"question_id":"attendance_risk","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.960464477539062e-07,"mean_abs_dp":3.973642985026042e-07,"max_abs_dlogit":4.589557647705078e-06,"oracle_top2_margin":0.3122881054878235,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.9802322387695312e-06,"mean_question_max_abs_dp":1.001753844320774e-06,"mean_abs_dp":5.884295595543725e-07,"conv":{"fast_calls":23424,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j05","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":539,"padded_tokens":544,"chunks":34,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":44.94062954100082,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012617111206054688,"ref_absmax":48.31694412231445,"rel_max_abs_diff":2.6113222670114324e-05,"min_pos_cos":0.9999999975974992,"min_pos_cos_index":375,"mean_pos_cos":0.9999999999843412},"rope_planes_equal_oracle":true,"questions":[{"question_id":"plan","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-09,"mean_abs_dp":6.984919309616089e-10,"max_abs_dlogit":3.814697265625e-06,"oracle_top2_margin":0.9829754745587707,"near_tie":false},{"question_id":"two_factor","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.3283064365386963e-09,"mean_abs_dp":1.1641532182693481e-09,"max_abs_dlogit":2.384185791015625e-07,"oracle_top2_margin":0.9906381368637085,"near_tie":false},{"question_id":"account_risk","type":"score","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.7418136596679688e-06,"mean_abs_dp":1.0862946510314942e-06,"max_abs_dlogit":6.318092346191406e-06,"oracle_top2_margin":0.15399393439292908,"near_tie":false},{"question_id":"action","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.0728836059570312e-06,"mean_abs_dp":5.445908755064011e-07,"max_abs_dlogit":1.5020370483398438e-05,"oracle_top2_margin":0.758894219994545,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7418136596679688e-06,"mean_question_max_abs_dp":9.549548849463463e-07,"mean_abs_dp":5.076639354228974e-07,"conv":{"fast_calls":24240,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j06","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":616,"padded_tokens":624,"chunks":39,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":49.85482312500244,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0010886192321777344,"ref_absmax":49.15265655517578,"rel_max_abs_diff":2.214771913611051e-05,"min_pos_cos":0.9999999984740479,"min_pos_cos_index":337,"mean_pos_cos":0.999999999988638},"rope_planes_equal_oracle":true,"questions":[{"question_id":"error_class","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":6.984919309616089e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9747133883647621,"near_tie":false},{"question_id":"retryable","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":5.21540641784668e-08,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.8738237470388412,"near_tie":false},{"question_id":"wait_seconds","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":2.0954757928848267e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.962949600070715,"near_tie":false},{"question_id":"client_at_fault","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.682209014892578e-07,"mean_abs_dp":2.5331974029541016e-07,"max_abs_dlogit":4.5299530029296875e-06,"oracle_top2_margin":0.6655509620904922,"near_tie":false},{"question_id":"severity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.0728836059570312e-06,"mean_abs_dp":7.624427477518717e-07,"max_abs_dlogit":6.198883056640625e-06,"oracle_top2_margin":0.3374139368534088,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.0728836059570312e-06,"mean_question_max_abs_dp":2.8405338525772095e-07,"mean_abs_dp":1.7344513360191795e-07,"conv":{"fast_calls":25176,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j07","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":595,"padded_tokens":608,"chunks":38,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":47.21907616700628,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006084442138671875,"ref_absmax":48.98674011230469,"rel_max_abs_diff":1.2420589989705316e-05,"min_pos_cos":0.9999999995018309,"min_pos_cos_index":459,"mean_pos_cos":0.9999999999922307},"rope_planes_equal_oracle":true,"questions":[{"question_id":"out_of_stock","type":"choice","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.222376108169556e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.969930759165436,"near_tie":false},{"question_id":"reorder_count","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":9.5367431640625e-07,"mean_abs_dp":3.825873136520386e-07,"max_abs_dlogit":5.0067901611328125e-06,"oracle_top2_margin":0.34384429454803467,"near_tie":false},{"question_id":"urgency","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_abs_dp":3.241002559661865e-07,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.48331697285175323,"near_tie":false},{"question_id":"any_below","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":0.0,"mean_abs_dp":0.0,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9102346263825893,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.5367431640625e-07,"mean_question_max_abs_dp":4.3213367462158203e-07,"mean_abs_dp":2.1065352484583855e-07,"conv":{"fast_calls":26088,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j08","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":473,"padded_tokens":480,"chunks":30,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":39.72456229099771,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00180816650390625,"ref_absmax":48.316856384277344,"rel_max_abs_diff":3.7423099084207816e-05,"min_pos_cos":0.9999999869770055,"min_pos_cos_index":323,"mean_pos_cos":0.9999999999601688},"rope_planes_equal_oracle":true,"questions":[{"question_id":"risky","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.364418029785156e-07,"mean_abs_dp":5.066394805908203e-07,"max_abs_dlogit":2.2649765014648438e-06,"oracle_top2_margin":0.47441771626472473,"near_tie":false},{"question_id":"riskiest_change","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-06,"mean_abs_dp":1.0937219485640526e-06,"max_abs_dlogit":1.195073127746582e-05,"oracle_top2_margin":0.1350460946559906,"near_tie":false},{"question_id":"review","type":"score","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1622905731201172e-06,"mean_abs_dp":6.034970283508301e-07,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.1977044939994812,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2351741790771484e-06,"mean_question_max_abs_dp":1.3113021850585938e-06,"mean_abs_dp":7.802154868841171e-07,"conv":{"fast_calls":26808,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j09","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":573,"padded_tokens":576,"chunks":36,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":45.90465741700609,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006580352783203125,"ref_absmax":50.20844650268555,"rel_max_abs_diff":1.3106067288600843e-05,"min_pos_cos":0.9999999988444772,"min_pos_cos_index":266,"mean_pos_cos":0.999999999985168},"rope_planes_equal_oracle":true,"questions":[{"question_id":"status","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.5331974029541015e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9055544193834066,"near_tie":false},{"question_id":"late","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_abs_dp":2.384185791015625e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.07446438074111938,"near_tie":false},{"question_id":"customer_action","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.76837158203125e-07,"mean_abs_dp":4.470348358154297e-07,"max_abs_dlogit":8.344650268554688e-06,"oracle_top2_margin":0.7763255760073662,"near_tie":false},{"question_id":"escalation","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1324882507324219e-06,"mean_abs_dp":7.947285970052084e-07,"max_abs_dlogit":3.6954879760742188e-06,"oracle_top2_margin":0.31227901577949524,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1324882507324219e-06,"mean_question_max_abs_dp":4.917383193969727e-07,"mean_abs_dp":3.234793742497762e-07,"conv":{"fast_calls":27672,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j10","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":413,"padded_tokens":416,"chunks":26,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":36.236546375002945,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00022792816162109375,"ref_absmax":49.30205535888672,"rel_max_abs_diff":4.6230965415523916e-06,"min_pos_cos":0.9999999998415064,"min_pos_cos_index":126,"mean_pos_cos":0.999999999994952},"rope_planes_equal_oracle":true,"questions":[{"question_id":"weakest","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.430511474609375e-06,"mean_abs_dp":7.296912372112274e-07,"max_abs_dlogit":1.0073184967041016e-05,"oracle_top2_margin":0.5314054191112518,"near_tie":false},{"question_id":"overall","type":"score","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.1008232831954956e-07,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.5522213131189346,"near_tie":false},{"question_id":"satisfied","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.1920928955078125e-07,"max_abs_dlogit":2.0265579223632812e-06,"oracle_top2_margin":0.702947661280632,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.430511474609375e-06,"mean_question_max_abs_dp":6.159146626790365e-07,"mean_abs_dp":3.3705410632220184e-07,"conv":{"fast_calls":28296,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j11","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":463,"padded_tokens":464,"chunks":29,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":23.147023874975275,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003495216369628906,"ref_absmax":48.316856384277344,"rel_max_abs_diff":7.233948214326036e-06,"min_pos_cos":0.9999999996703547,"min_pos_cos_index":241,"mean_pos_cos":0.9999999999911192},"rope_planes_equal_oracle":true,"questions":[{"question_id":"room","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.825392246246338e-07,"mean_abs_dp":1.1106021702289581e-07,"max_abs_dlogit":7.62939453125e-06,"oracle_top2_margin":0.8663652017712593,"near_tie":false},{"question_id":"slot_10","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.383036866784096e-08,"max_abs_dlogit":6.4373016357421875e-06,"oracle_top2_margin":0.9580284468829632,"near_tie":false},{"question_id":"feasible","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.955777406692505e-07,"max_abs_dlogit":9.059906005859375e-06,"oracle_top2_margin":0.9204762727022171,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":1.8005569775899252e-07,"mean_abs_dp":1.0107178241014481e-07,"conv":{"fast_calls":28992,"checked_vs_F_conv1d":600,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"own_j12","arm":"text","device":"cpu","S":16,"perturb":null,"source":"own_json","tokens":562,"padded_tokens":576,"chunks":36,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":27.135959375009406,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0014405250549316406,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.9814130362181383e-05,"min_pos_cos":0.9999999987367416,"min_pos_cos_index":291,"mean_pos_cos":0.999999999985177},"rope_planes_equal_oracle":true,"questions":[{"question_id":"coupon_valid","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":9.685754776000977e-08,"mean_abs_dp":7.82310962677002e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9615767225623131,"near_tie":false},{"question_id":"total_band","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.6689300537109375e-06,"mean_abs_dp":8.530914783477783e-07,"max_abs_dlogit":7.62939453125e-06,"oracle_top2_margin":0.5870811939239502,"near_tie":false},{"question_id":"dessert_items","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.4570693969726562e-06,"mean_abs_dp":1.7415732145309448e-06,"max_abs_dlogit":9.179115295410156e-06,"oracle_top2_margin":0.1630445122718811,"near_tie":false},{"question_id":"basket_size","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.2069940567016602e-06,"mean_abs_dp":8.263935645421346e-07,"max_abs_dlogit":7.3909759521484375e-06,"oracle_top2_margin":0.6478055715560913,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.4570693969726562e-06,"mean_question_max_abs_dp":1.607462763786316e-06,"mean_abs_dp":1.001100127513592e-06,"conv":{"fast_calls":29856,"checked_vs_F_conv1d":624,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_01","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":430,"padded_tokens":432,"chunks":27,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":20.93862583299051,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000446319580078125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.23732973903833e-06,"min_pos_cos":0.9999999997278849,"min_pos_cos_index":78,"mean_pos_cos":0.9999999999924615},"rope_planes_equal_oracle":true,"questions":[{"question_id":"circles","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.190951585769653e-08,"max_abs_dlogit":5.781650543212891e-06,"oracle_top2_margin":0.9245060188695788,"near_tie":false},{"question_id":"more_circles","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.1175870895385742e-07,"max_abs_dlogit":3.635883331298828e-06,"oracle_top2_margin":0.6116609573364258,"near_tie":false},{"question_id":"clutter","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.642673492431641e-07,"mean_abs_dp":5.414088567097982e-07,"max_abs_dlogit":2.9802322387695312e-06,"oracle_top2_margin":0.002184569835662842,"near_tie":true}],"argmax_all_equal":true,"max_abs_dp":8.642673492431641e-07,"mean_question_max_abs_dp":3.6756197611490887e-07,"mean_abs_dp":1.9083646210757169e-07,"conv":{"fast_calls":2040,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"p3":{"id":"img_01","arm":"g256","tokens":430,"stages":[{"stage":"layer-0 input (hs[0])","max_abs_diff":0.0,"ref_absmax":17.857091903686523,"min_pos_cos":0.9999999999999998,"mean_pos_cos":1.0},{"stage":"layer 0 output (hs[1])","max_abs_diff":3.4332275390625e-05,"ref_absmax":32.62185287475586,"min_pos_cos":0.9999999999996432,"mean_pos_cos":0.9999999999999819},{"stage":"layer 1 output (hs[2])","max_abs_diff":3.24249267578125e-05,"ref_absmax":38.02134323120117,"min_pos_cos":0.9999999999991824,"mean_pos_cos":0.9999999999999357},{"stage":"layer 2 output (hs[3])","max_abs_diff":5.14984130859375e-05,"ref_absmax":32.855560302734375,"min_pos_cos":0.999999999998462,"mean_pos_cos":0.9999999999998237},{"stage":"layer 3 output (hs[4])","max_abs_diff":4.9591064453125e-05,"ref_absmax":30.969675064086914,"min_pos_cos":0.9999999999978404,"mean_pos_cos":0.9999999999998096},{"stage":"layer 4 output (hs[5])","max_abs_diff":4.57763671875e-05,"ref_absmax":28.936403274536133,"min_pos_cos":0.9999999999970404,"mean_pos_cos":0.9999999999997319},{"stage":"layer 5 output (hs[6])","max_abs_diff":3.814697265625e-05,"ref_absmax":28.993045806884766,"min_pos_cos":0.9999999999965055,"mean_pos_cos":0.9999999999996391},{"stage":"layer 6 output (hs[7])","max_abs_diff":3.24249267578125e-05,"ref_absmax":63.745628356933594,"min_pos_cos":0.9999999999971534,"mean_pos_cos":0.9999999999994773},{"stage":"layer 7 output (hs[8])","max_abs_diff":3.0517578125e-05,"ref_absmax":44.75798416137695,"min_pos_cos":0.9999999999962327,"mean_pos_cos":0.999999999999417},{"stage":"layer 8 output (hs[9])","max_abs_diff":2.288818359375e-05,"ref_absmax":42.81202697753906,"min_pos_cos":0.9999999999961984,"mean_pos_cos":0.9999999999992938},{"stage":"layer 9 output (hs[10])","max_abs_diff":2.09808349609375e-05,"ref_absmax":43.587894439697266,"min_pos_cos":0.9999999999962185,"mean_pos_cos":0.9999999999992721},{"stage":"layer 10 output (hs[11])","max_abs_diff":3.3855438232421875e-05,"ref_absmax":68.33386993408203,"min_pos_cos":0.9999999999959743,"mean_pos_cos":0.9999999999991946},{"stage":"layer 11 output (hs[12])","max_abs_diff":3.3855438232421875e-05,"ref_absmax":67.96086883544922,"min_pos_cos":0.999999999988349,"mean_pos_cos":0.9999999999990505},{"stage":"layer 12 output (hs[13])","max_abs_diff":3.528594970703125e-05,"ref_absmax":67.7262191772461,"min_pos_cos":0.9999999999798523,"mean_pos_cos":0.9999999999988494},{"stage":"layer 13 output (hs[14])","max_abs_diff":3.62396240234375e-05,"ref_absmax":67.68138885498047,"min_pos_cos":0.9999999999633843,"mean_pos_cos":0.9999999999985885},{"stage":"layer 14 output (hs[15])","max_abs_diff":6.198883056640625e-05,"ref_absmax":74.43360137939453,"min_pos_cos":0.9999999999327538,"mean_pos_cos":0.9999999999982189},{"stage":"layer 15 output (hs[16])","max_abs_diff":0.0001773834228515625,"ref_absmax":70.99459075927734,"min_pos_cos":0.9999999997584652,"mean_pos_cos":0.9999999999971674},{"stage":"layer 16 output (hs[17])","max_abs_diff":0.000324249267578125,"ref_absmax":69.69348907470703,"min_pos_cos":0.9999999992514425,"mean_pos_cos":0.9999999999952288},{"stage":"layer 17 output (hs[18])","max_abs_diff":0.000247955322265625,"ref_absmax":68.98184204101562,"min_pos_cos":0.9999999990649938,"mean_pos_cos":0.999999999994109},{"stage":"layer 18 output (hs[19])","max_abs_diff":0.000148773193359375,"ref_absmax":79.57122802734375,"min_pos_cos":0.9999999986314215,"mean_pos_cos":0.9999999999916819},{"stage":"layer 19 output (hs[20])","max_abs_diff":0.00020599365234375,"ref_absmax":53.413963317871094,"min_pos_cos":0.9999999989547819,"mean_pos_cos":0.9999999999911849},{"stage":"layer 20 output (hs[21])","max_abs_diff":0.000308990478515625,"ref_absmax":61.58170700073242,"min_pos_cos":0.9999999989464332,"mean_pos_cos":0.9999999999901947},{"stage":"layer 21 output (hs[22])","max_abs_diff":0.000316619873046875,"ref_absmax":67.45378875732422,"min_pos_cos":0.9999999988594699,"mean_pos_cos":0.9999999999894328},{"stage":"layer 22 output (hs[23])","max_abs_diff":0.00086212158203125,"ref_absmax":157.20413208007812,"min_pos_cos":0.9999999986082141,"mean_pos_cos":0.9999999999883356},{"stage":"layer 23 output (hs[24])","max_abs_diff":0.00051116943359375,"ref_absmax":124.89531707763672,"min_pos_cos":0.9999999989908317,"mean_pos_cos":0.9999999999904693},{"stage":"layer 24 output (hs[25])","max_abs_diff":0.000396728515625,"ref_absmax":128.6216583251953,"min_pos_cos":0.9999999990595485,"mean_pos_cos":0.9999999999906934},{"stage":"layer 25 output (hs[26])","max_abs_diff":0.0003662109375,"ref_absmax":138.59469604492188,"min_pos_cos":0.9999999990560166,"mean_pos_cos":0.9999999999906266},{"stage":"layer 26 output (hs[27])","max_abs_diff":0.0013275146484375,"ref_absmax":294.0438232421875,"min_pos_cos":0.9999999989354892,"mean_pos_cos":0.9999999999890612},{"stage":"layer 27 output (hs[28])","max_abs_diff":0.000637054443359375,"ref_absmax":163.96519470214844,"min_pos_cos":0.9999999991511438,"mean_pos_cos":0.9999999999910592},{"stage":"layer 28 output (hs[29])","max_abs_diff":0.000728607177734375,"ref_absmax":170.21034240722656,"min_pos_cos":0.9999999993796341,"mean_pos_cos":0.9999999999915395},{"stage":"layer 29 output (hs[30])","max_abs_diff":0.0007781982421875,"ref_absmax":176.71734619140625,"min_pos_cos":0.9999999994045023,"mean_pos_cos":0.9999999999915758},{"stage":"layer 30 output (hs[31])","max_abs_diff":0.000911712646484375,"ref_absmax":172.61424255371094,"min_pos_cos":0.9999999993754266,"mean_pos_cos":0.9999999999899175},{"stage":"layer 31 output (hs[32])","max_abs_diff":0.005859375,"ref_absmax":2534.841064453125,"min_pos_cos":0.9999999998592494,"mean_pos_cos":0.9999999999956098},{"stage":"final norm (last_hidden)","max_abs_diff":0.000446319580078125,"ref_absmax":48.31694793701172,"min_pos_cos":0.9999999997278849,"mean_pos_cos":0.9999999999924615}],"worst_max_abs_diff":{"value":0.005859375,"stage":"layer 31 output (hs[32])"},"worst_min_pos_cos":{"value":0.9999999986082141,"stage":"layer 22 output (hs[23])"}}},{"kind":"p1","id":"img_01","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":562,"padded_tokens":576,"chunks":36,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":37.273597249994054,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000537872314453125,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.113218770226435e-05,"min_pos_cos":0.9999999997353887,"min_pos_cos_index":363,"mean_pos_cos":0.9999999999923223},"rope_planes_equal_oracle":true,"questions":[{"question_id":"circles","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":5.898376305898031e-08,"max_abs_dlogit":7.271766662597656e-06,"oracle_top2_margin":0.9202973758801818,"near_tie":false},{"question_id":"more_circles","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_abs_dp":2.7567148208618164e-07,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.5916850119829178,"near_tie":false},{"question_id":"clutter","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-07,"mean_abs_dp":1.0306636492411296e-07,"max_abs_dlogit":1.3262033462524414e-06,"oracle_top2_margin":0.28941109776496887,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_question_max_abs_dp":2.086162567138672e-07,"mean_abs_dp":1.1040405793623491e-07,"conv":{"fast_calls":30720,"checked_vs_F_conv1d":648,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_01","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":430,"padded_tokens":432,"chunks":27,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":30.280464249983197,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000446319580078125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.23732973903833e-06,"min_pos_cos":0.9999999997278849,"min_pos_cos_index":78,"mean_pos_cos":0.9999999999924615},"rope_planes_equal_oracle":true,"questions":[{"question_id":"circles","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.190951585769653e-08,"max_abs_dlogit":5.781650543212891e-06,"oracle_top2_margin":0.9245060188695788,"near_tie":false},{"question_id":"more_circles","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.1175870895385742e-07,"max_abs_dlogit":3.635883331298828e-06,"oracle_top2_margin":0.6116609573364258,"near_tie":false},{"question_id":"clutter","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.642673492431641e-07,"mean_abs_dp":5.414088567097982e-07,"max_abs_dlogit":2.9802322387695312e-06,"oracle_top2_margin":0.002184569835662842,"near_tie":true}],"argmax_all_equal":true,"max_abs_dp":8.642673492431641e-07,"mean_question_max_abs_dp":3.6756197611490887e-07,"mean_abs_dp":1.9083646210757169e-07,"conv":{"fast_calls":31368,"checked_vs_F_conv1d":672,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_02","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":466,"padded_tokens":480,"chunks":30,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":32.89317458297592,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004940032958984375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.0224243315200237e-05,"min_pos_cos":0.9999999997434058,"min_pos_cos_index":152,"mean_pos_cos":0.9999999999934236},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tallest","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_abs_dp":1.0989606380462647e-08,"max_abs_dlogit":1.2516975402832031e-06,"oracle_top2_margin":0.9406779743731022,"near_tie":false},{"question_id":"green_over_orange","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.842877388000488e-08,"mean_abs_dp":2.421438694000244e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.8968241214752197,"near_tie":false},{"question_id":"spread","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.1455267667770386e-07,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.29390622675418854,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":1.0182460149129231e-07,"mean_abs_dp":5.1053410226648506e-08,"conv":{"fast_calls":32088,"checked_vs_F_conv1d":696,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_02","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":598,"padded_tokens":608,"chunks":38,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":41.89351141700172,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003695487976074219,"ref_absmax":48.316856384277344,"rel_max_abs_diff":7.648444564942261e-06,"min_pos_cos":0.9999999997505042,"min_pos_cos_index":78,"mean_pos_cos":0.9999999999888222},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tallest","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":6.332993507385254e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9293951261788607,"near_tie":false},{"question_id":"green_over_orange","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.043081283569336e-07,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.8826808221638203,"near_tie":false},{"question_id":"spread","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":8.344650268554688e-07,"mean_abs_dp":4.33996319770813e-07,"max_abs_dlogit":6.198883056640625e-06,"oracle_top2_margin":0.3051571547985077,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":8.344650268554688e-07,"mean_question_max_abs_dp":3.228584925333659e-07,"mean_abs_dp":1.7966059121218594e-07,"conv":{"fast_calls":33000,"checked_vs_F_conv1d":720,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_02","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":702,"padded_tokens":704,"chunks":44,"n_image_tokens":300,"merged_hw":[15,20],"n_image_max_used":1024,"seconds":44.24498283298453,"host":{"start":337,"amount":280,"image_tokens":300},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0011196136474609375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.3172319791592815e-05,"min_pos_cos":0.9999999976385702,"min_pos_cos_index":327,"mean_pos_cos":0.9999999999818632},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tallest","type":"choice","n_options":5,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":2.7380883693695067e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9413488861173391,"near_tie":false},{"question_id":"green_over_orange","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.636845111846924e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.8825415037572384,"near_tie":false},{"question_id":"spread","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.5331974029541016e-07,"mean_abs_dp":1.601874828338623e-07,"max_abs_dlogit":5.602836608886719e-06,"oracle_top2_margin":0.30996087193489075,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.5331974029541016e-07,"mean_question_max_abs_dp":1.6391277313232422e-07,"mean_abs_dp":8.458102291280573e-08,"conv":{"fast_calls":34056,"checked_vs_F_conv1d":744,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_03","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":495,"padded_tokens":496,"chunks":31,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":34.82791620900389,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004911422729492188,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.0165010289753717e-05,"min_pos_cos":0.9999999995907369,"min_pos_cos_index":93,"mean_pos_cos":0.9999999999908318},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tens_digit","type":"choice","n_options":10,"argmax":4,"argmax_oracle":4,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":4.116445779800415e-08,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.9320426946505904,"near_tie":false},{"question_id":"even","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":3.427267074584961e-07,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.09057280421257019,"near_tie":false},{"question_id":"readability","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.718701044718424e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.540686309337616,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":1.986821492513021e-07,"mean_abs_dp":8.257726828257243e-08,"conv":{"fast_calls":744,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"img_03","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":627,"padded_tokens":640,"chunks":40,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":23.29658374999417,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009713172912597656,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.010307300488695e-05,"min_pos_cos":0.9999999997566154,"min_pos_cos_index":175,"mean_pos_cos":0.9999999999872713},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tens_digit","type":"choice","n_options":10,"argmax":4,"argmax_oracle":4,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.634471118450165e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9521092572249472,"near_tie":false},{"question_id":"even","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":5.587935447692871e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9024685919284821,"near_tie":false},{"question_id":"readability","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.384185791015625e-07,"max_abs_dlogit":7.748603820800781e-06,"oracle_top2_margin":0.7032996490597725,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":1.6267100969950357e-07,"mean_abs_dp":5.9325248003005984e-08,"conv":{"fast_calls":1704,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_03","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":501,"padded_tokens":512,"chunks":32,"n_image_tokens":70,"merged_hw":[7,10],"n_image_max_used":1024,"seconds":18.96470708301058,"host":{"start":107,"amount":60,"image_tokens":70},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005333423614501953,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.1038432575339252e-05,"min_pos_cos":0.9999999998580654,"min_pos_cos_index":229,"mean_pos_cos":0.9999999999917932},"rope_planes_equal_oracle":true,"questions":[{"question_id":"tens_digit","type":"choice","n_options":10,"argmax":4,"argmax_oracle":4,"argmax_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_abs_dp":4.051253199577331e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9546868926845491,"near_tie":false},{"question_id":"even","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.862645149230957e-07,"mean_abs_dp":1.825392246246338e-07,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.84402646869421,"near_tie":false},{"question_id":"readability","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":2.7318795522054035e-07,"max_abs_dlogit":6.318092346191406e-06,"oracle_top2_margin":0.6572192162275314,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_question_max_abs_dp":2.0364920298258463e-07,"mean_abs_dp":8.167698979377747e-08,"conv":{"fast_calls":2472,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_04","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":558,"padded_tokens":560,"chunks":35,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":26.238221959007205,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005159378051757812,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.0678215508732293e-05,"min_pos_cos":0.9999999993455437,"min_pos_cos_index":477,"mean_pos_cos":0.9999999999877031},"rope_planes_equal_oracle":true,"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.51808625459671e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9446959812194109,"near_tie":false},{"question_id":"paid_cash","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_abs_dp":2.2351741790771484e-08,"max_abs_dlogit":2.086162567138672e-06,"oracle_top2_margin":0.9258542135357857,"near_tie":false},{"question_id":"items","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":2.9103830456733704e-08,"max_abs_dlogit":3.0994415283203125e-06,"oracle_top2_margin":0.9453170308843255,"near_tie":false},{"question_id":"legibility","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":7.152557373046875e-07,"mean_abs_dp":4.824250936508179e-07,"max_abs_dlogit":2.1636486053466797e-05,"oracle_top2_margin":0.8597390353679657,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.152557373046875e-07,"mean_question_max_abs_dp":2.4959444999694824e-07,"mean_abs_dp":1.2582167983055114e-07,"conv":{"fast_calls":3312,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_04","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":690,"padded_tokens":704,"chunks":44,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":30.112736875016708,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0011761188507080078,"ref_absmax":48.31694793701172,"rel_max_abs_diff":2.434174551425004e-05,"min_pos_cos":0.9999999990149874,"min_pos_cos_index":208,"mean_pos_cos":0.9999999999827952},"rope_planes_equal_oracle":true,"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.7497451305389404e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9414124302566051,"near_tie":false},{"question_id":"paid_cash","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.450580596923828e-08,"max_abs_dlogit":1.6093254089355469e-06,"oracle_top2_margin":0.9361573942005634,"near_tie":false},{"question_id":"items","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":2.646508316198985e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9564370606094599,"near_tie":false},{"question_id":"legibility","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.5704503059387207e-07,"max_abs_dlogit":2.562999725341797e-06,"oracle_top2_margin":0.7817753255367279,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.459513386090596e-08,"conv":{"fast_calls":3096,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"img_04","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":1262,"padded_tokens":1264,"chunks":79,"n_image_tokens":768,"merged_hw":[24,32],"n_image_max_used":1024,"seconds":60.499346083001,"host":{"start":805,"amount":736,"image_tokens":768},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0025835037231445312,"ref_absmax":48.316856384277344,"rel_max_abs_diff":5.347002922949313e-05,"min_pos_cos":0.9999999916858646,"min_pos_cos_index":365,"mean_pos_cos":0.9999999999496413},"rope_planes_equal_oracle":true,"questions":[{"question_id":"total","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.4924596548080444e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9573134584352374,"near_tie":false},{"question_id":"paid_cash","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.195638656616211e-08,"max_abs_dlogit":2.7418136596679688e-06,"oracle_top2_margin":0.9134139567613602,"near_tie":false},{"question_id":"items","type":"choice","n_options":6,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":5.5025642116864525e-08,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.9510575840249658,"near_tie":false},{"question_id":"legibility","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":2.893308798472087e-07,"max_abs_dlogit":9.655952453613281e-06,"oracle_top2_margin":0.819927416741848,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_question_max_abs_dp":2.086162567138672e-07,"mean_abs_dp":1.0011717677116394e-07,"conv":{"fast_calls":5208,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_05","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":464,"padded_tokens":464,"chunks":29,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":21.32729320798535,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0021610260009765625,"ref_absmax":48.31694412231445,"rel_max_abs_diff":4.4726048806106615e-05,"min_pos_cos":0.9999999884983474,"min_pos_cos_index":69,"mean_pos_cos":0.9999999999618189},"rope_planes_equal_oracle":true,"questions":[{"question_id":"action","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.5324365297953286e-08,"max_abs_dlogit":2.4437904357910156e-06,"oracle_top2_margin":0.944669371470809,"near_tie":false},{"question_id":"lit_color","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.370626846949259e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9751403219997883,"near_tie":false},{"question_id":"may_proceed","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.043081283569336e-07,"mean_abs_dp":5.21540641784668e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.8753064572811127,"near_tie":false},{"question_id":"caution","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":2.384185791015625e-07,"oracle_top2_margin":0.7880274653434753,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":7.217749953269958e-08,"mean_abs_dp":3.699890591881492e-08,"conv":{"fast_calls":5904,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_05","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":596,"padded_tokens":608,"chunks":38,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":25.253983875009,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008960962295532227,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.8546244449893865e-05,"min_pos_cos":0.9999999997379783,"min_pos_cos_index":164,"mean_pos_cos":0.9999999999845003},"rope_planes_equal_oracle":true,"questions":[{"question_id":"action","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_abs_dp":1.0244548320770264e-08,"max_abs_dlogit":1.1324882507324219e-06,"oracle_top2_margin":0.9480947684496641,"near_tie":false},{"question_id":"lit_color","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.210719347000122e-08,"mean_abs_dp":6.829698880513509e-09,"max_abs_dlogit":1.3113021850585938e-06,"oracle_top2_margin":0.9755466179922223,"near_tie":false},{"question_id":"may_proceed","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.470348358154297e-08,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.8733854070305824,"near_tie":false},{"question_id":"caution","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.5324365297953286e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8226336687803268,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":3.748573362827301e-08,"mean_abs_dp":2.514570951461792e-08,"conv":{"fast_calls":6816,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_05","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":464,"padded_tokens":464,"chunks":29,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":23.275258542009396,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0021610260009765625,"ref_absmax":48.31694412231445,"rel_max_abs_diff":4.4726048806106615e-05,"min_pos_cos":0.9999999884983474,"min_pos_cos_index":69,"mean_pos_cos":0.9999999999618189},"rope_planes_equal_oracle":true,"questions":[{"question_id":"action","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.5324365297953286e-08,"max_abs_dlogit":2.4437904357910156e-06,"oracle_top2_margin":0.944669371470809,"near_tie":false},{"question_id":"lit_color","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.370626846949259e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9751403219997883,"near_tie":false},{"question_id":"may_proceed","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.043081283569336e-07,"mean_abs_dp":5.21540641784668e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.8753064572811127,"near_tie":false},{"question_id":"caution","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":2.384185791015625e-07,"oracle_top2_margin":0.7880274653434753,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":7.217749953269958e-08,"mean_abs_dp":3.699890591881492e-08,"conv":{"fast_calls":7512,"checked_vs_F_conv1d":24,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_06","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":465,"padded_tokens":480,"chunks":30,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":32.921041958994465,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006694793701171875,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.3855994608557493e-05,"min_pos_cos":0.9999999987143509,"min_pos_cos_index":127,"mean_pos_cos":0.9999999999852331},"rope_planes_equal_oracle":true,"questions":[{"question_id":"in_red_zone","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.0132789611816406e-06,"mean_abs_dp":9.685754776000977e-07,"max_abs_dlogit":6.496906280517578e-06,"oracle_top2_margin":0.1707080602645874,"near_tie":false},{"question_id":"reading","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.9669532775878906e-06,"mean_abs_dp":9.75094735622406e-07,"max_abs_dlogit":9.59634780883789e-06,"oracle_top2_margin":0.06736165285110474,"near_tie":false},{"question_id":"alarm","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":7.152557373046875e-07,"mean_abs_dp":4.600733518600464e-07,"max_abs_dlogit":1.1086463928222656e-05,"oracle_top2_margin":0.32940465211868286,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.9669532775878906e-06,"mean_question_max_abs_dp":1.231829325358073e-06,"mean_abs_dp":7.677823305130005e-07,"conv":{"fast_calls":8232,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"img_06","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":597,"padded_tokens":608,"chunks":38,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":30.422949083993444,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000713348388671875,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.4763965250520805e-05,"min_pos_cos":0.999999998949302,"min_pos_cos_index":180,"mean_pos_cos":0.9999999999839944},"rope_planes_equal_oracle":true,"questions":[{"question_id":"in_red_zone","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.1920928955078125e-07,"max_abs_dlogit":2.4437904357910156e-06,"oracle_top2_margin":0.017518073320388794,"near_tie":true},{"question_id":"reading","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.115964889526367e-06,"mean_abs_dp":1.0402873158454895e-06,"max_abs_dlogit":6.854534149169922e-06,"oracle_top2_margin":0.05165323615074158,"near_tie":false},{"question_id":"alarm","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.0132789611816406e-06,"mean_abs_dp":4.991888999938965e-07,"max_abs_dlogit":9.5367431640625e-06,"oracle_top2_margin":0.3209385871887207,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.115964889526367e-06,"mean_question_max_abs_dp":1.0828177134195964e-06,"mean_abs_dp":6.396323442459106e-07,"conv":{"fast_calls":9144,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_06","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":471,"padded_tokens":480,"chunks":30,"n_image_tokens":70,"merged_hw":[7,10],"n_image_max_used":1024,"seconds":24.082585416006623,"host":{"start":107,"amount":60,"image_tokens":70},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000415802001953125,"ref_absmax":48.79163360595703,"rel_max_abs_diff":8.521993858847949e-06,"min_pos_cos":0.9999999997066795,"min_pos_cos_index":126,"mean_pos_cos":0.9999999999914216},"rope_planes_equal_oracle":true,"questions":[{"question_id":"in_red_zone","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_abs_dp":7.599592208862305e-07,"max_abs_dlogit":2.0265579223632812e-06,"oracle_top2_margin":0.29172810912132263,"near_tie":false},{"question_id":"reading","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.0728836059570312e-06,"mean_abs_dp":5.499459803104401e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.30915674567222595,"near_tie":false},{"question_id":"alarm","type":"score","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":9.499490261077881e-08,"max_abs_dlogit":5.9604644775390625e-06,"oracle_top2_margin":0.39862820506095886,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.0728836059570312e-06,"mean_question_max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.0996819734573363e-07,"conv":{"fast_calls":9864,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_07","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":426,"padded_tokens":432,"chunks":27,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":22.062903625017498,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009365081787109375,"ref_absmax":52.16621398925781,"rel_max_abs_diff":1.7952389239974084e-05,"min_pos_cos":0.9999999995982428,"min_pos_cos_index":308,"mean_pos_cos":0.9999999999818201},"rope_planes_equal_oracle":true,"questions":[{"question_id":"odd_direction","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.814697265625e-06,"mean_abs_dp":1.9241124391555786e-06,"max_abs_dlogit":2.5272369384765625e-05,"oracle_top2_margin":0.7762424126267433,"near_tie":false},{"question_id":"odd_position","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.602836608886719e-06,"mean_abs_dp":2.780929207801819e-06,"max_abs_dlogit":2.3126602172851562e-05,"oracle_top2_margin":0.2659342586994171,"near_tie":false},{"question_id":"all_same","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-06,"mean_abs_dp":1.7881393432617188e-06,"max_abs_dlogit":1.5616416931152344e-05,"oracle_top2_margin":0.7499157190322876,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.602836608886719e-06,"mean_question_max_abs_dp":3.735224405924479e-06,"mean_abs_dp":2.239644527435303e-06,"conv":{"fast_calls":10512,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_07","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":558,"padded_tokens":560,"chunks":35,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":27.68919674999779,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007581710815429688,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.5691647559042835e-05,"min_pos_cos":0.9999999993961705,"min_pos_cos_index":169,"mean_pos_cos":0.9999999999830134},"rope_planes_equal_oracle":true,"questions":[{"question_id":"odd_direction","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.0046870708465576e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9483929080888629,"near_tie":false},{"question_id":"odd_position","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.9371509552001953e-07,"mean_abs_dp":1.4994293451309204e-07,"max_abs_dlogit":3.814697265625e-06,"oracle_top2_margin":0.7735076621174812,"near_tie":false},{"question_id":"all_same","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.847564458847046e-08,"max_abs_dlogit":4.5299530029296875e-06,"oracle_top2_margin":0.9536473024636507,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.9371509552001953e-07,"mean_question_max_abs_dp":1.44044558207194e-07,"mean_abs_dp":9.369105100631713e-08,"conv":{"fast_calls":11352,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_07","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":662,"padded_tokens":672,"chunks":42,"n_image_tokens":300,"merged_hw":[15,20],"n_image_max_used":1024,"seconds":33.42534237500513,"host":{"start":337,"amount":280,"image_tokens":300},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0054264068603515625,"ref_absmax":48.316856384277344,"rel_max_abs_diff":0.00011230877309553928,"min_pos_cos":0.999999984147918,"min_pos_cos_index":85,"mean_pos_cos":0.9999999999508905},"rope_planes_equal_oracle":true,"questions":[{"question_id":"odd_direction","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":1.1641532182693481e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9635526714846492,"near_tie":false},{"question_id":"odd_position","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.122274160385132e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.8184187300503254,"near_tie":false},{"question_id":"all_same","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.984919309616089e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9710730528458953,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":4.594524701436361e-08,"mean_abs_dp":2.2351741790771484e-08,"conv":{"fast_calls":12360,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_08","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":522,"padded_tokens":528,"chunks":33,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":26.245990167022683,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00038886070251464844,"ref_absmax":48.316856384277344,"rel_max_abs_diff":8.048137474465051e-06,"min_pos_cos":0.9999999997907831,"min_pos_cos_index":139,"mean_pos_cos":0.9999999999938657},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest_room","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.493631422519684e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.8920186869800091,"near_tie":false},{"question_id":"rooms","type":"choice","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.682209014892578e-07,"mean_abs_dp":1.4472752809524537e-07,"max_abs_dlogit":8.58306884765625e-06,"oracle_top2_margin":0.5571556091308594,"near_tie":false},{"question_id":"bath_next_to_bed","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":9.238719940185547e-07,"mean_abs_dp":9.08970832824707e-07,"max_abs_dlogit":2.2649765014648438e-06,"oracle_top2_margin":0.39446547627449036,"near_tie":false},{"question_id":"complexity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.2782554626464844e-07,"mean_abs_dp":1.9371509552001953e-07,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.5075182020664215,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.238719940185547e-07,"mean_question_max_abs_dp":4.0978193283081055e-07,"mean_abs_dp":2.358907035418919e-07,"conv":{"fast_calls":13152,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_08","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":654,"padded_tokens":656,"chunks":41,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":32.74815920801484,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012140274047851562,"ref_absmax":48.31694412231445,"rel_max_abs_diff":2.5126328389308794e-05,"min_pos_cos":0.9999999985911548,"min_pos_cos_index":322,"mean_pos_cos":0.9999999999868165},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest_room","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.0104849934577942e-07,"max_abs_dlogit":2.8014183044433594e-06,"oracle_top2_margin":0.8663040027022362,"near_tie":false},{"question_id":"rooms","type":"choice","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.5331974029541016e-07,"mean_abs_dp":1.0654330253601075e-07,"max_abs_dlogit":6.198883056640625e-06,"oracle_top2_margin":0.723192386329174,"near_tie":false},{"question_id":"bath_next_to_bed","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":3.427267074584961e-07,"max_abs_dlogit":3.2782554626464844e-06,"oracle_top2_margin":0.4448655843734741,"near_tie":false},{"question_id":"complexity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.066394805908203e-07,"mean_abs_dp":3.3403436342875165e-07,"max_abs_dlogit":4.827976226806641e-06,"oracle_top2_margin":0.5081702172756195,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.066394805908203e-07,"mean_question_max_abs_dp":3.241002559661865e-07,"mean_abs_dp":1.8746192966188703e-07,"conv":{"fast_calls":14136,"checked_vs_F_conv1d":48,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_08","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":1226,"padded_tokens":1232,"chunks":77,"n_image_tokens":768,"merged_hw":[24,32],"n_image_max_used":1024,"seconds":66.55886979197385,"host":{"start":805,"amount":736,"image_tokens":768},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012178421020507812,"ref_absmax":50.38888931274414,"rel_max_abs_diff":2.4168861799912106e-05,"min_pos_cos":0.9999999974216528,"min_pos_cos_index":592,"mean_pos_cos":0.9999999999537009},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest_room","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":3.3527612686157227e-08,"mean_abs_dp":1.7695128917694092e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8977233991026878,"near_tie":false},{"question_id":"rooms","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1324882507324219e-06,"mean_abs_dp":4.5336782932281495e-07,"max_abs_dlogit":4.291534423828125e-06,"oracle_top2_margin":0.14287957549095154,"near_tie":false},{"question_id":"bath_next_to_bed","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":2.384185791015625e-07,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.384887158870697,"near_tie":false},{"question_id":"complexity","type":"score","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_abs_dp":5.35200039545695e-07,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.4244216978549957,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1324882507324219e-06,"mean_question_max_abs_dp":5.448237061500549e-07,"mean_abs_dp":3.157183527946472e-07,"conv":{"fast_calls":15984,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"img_09","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":460,"padded_tokens":464,"chunks":29,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":22.984625207987847,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0014500617980957031,"ref_absmax":48.31694793701172,"rel_max_abs_diff":3.0011452709845683e-05,"min_pos_cos":0.9999999973276588,"min_pos_cos_index":180,"mean_pos_cos":0.9999999999835104},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.3527612686157227e-08,"mean_abs_dp":2.1730860074361164e-08,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.9686121270060539,"near_tie":false},{"question_id":"smallest","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.51925802230835e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9492924120277166,"near_tie":false},{"question_id":"same_size","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.1420419216156006e-08,"mean_abs_dp":1.0710209608078003e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9763501016423106,"near_tie":false},{"question_id":"spread","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.2103389104207358e-07,"max_abs_dlogit":1.1920928955078125e-06,"oracle_top2_margin":0.4161626994609833,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":1.0663643479347229e-07,"mean_abs_dp":6.99338587847623e-08,"conv":{"fast_calls":16680,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_09","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":592,"padded_tokens":592,"chunks":37,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":29.285472166986438,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000804901123046875,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.6658805710480693e-05,"min_pos_cos":0.999999999256012,"min_pos_cos_index":142,"mean_pos_cos":0.9999999999863673},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":3.725290298461914e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9668994825333357,"near_tie":false},{"question_id":"smallest","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.1420419216156006e-08,"mean_abs_dp":1.2728075186411539e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9433693159371614,"near_tie":false},{"question_id":"same_size","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":6.51925802230835e-09,"mean_abs_dp":3.259629011154175e-09,"max_abs_dlogit":1.3113021850585938e-06,"oracle_top2_margin":0.9738637022674084,"near_tie":false},{"question_id":"spread","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.443991343180339e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.4367528706789017,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":3.818422555923462e-08,"mean_abs_dp":2.810900861566717e-08,"conv":{"fast_calls":17568,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_09","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":696,"padded_tokens":704,"chunks":44,"n_image_tokens":300,"merged_hw":[15,20],"n_image_max_used":1024,"seconds":35.02442991701537,"host":{"start":337,"amount":280,"image_tokens":300},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00099945068359375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.068534168789546e-05,"min_pos_cos":0.9999999990400318,"min_pos_cos_index":246,"mean_pos_cos":0.9999999999818874},"rope_planes_equal_oracle":true,"questions":[{"question_id":"largest","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.0244548320770264e-08,"mean_abs_dp":3.4148494402567544e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.96602054964751,"near_tie":false},{"question_id":"smallest","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.078051567077637e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9375971518456936,"near_tie":false},{"question_id":"same_size","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_abs_dp":9.778887033462524e-09,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9730345876887441,"near_tie":false},{"question_id":"spread","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.086162567138672e-07,"mean_abs_dp":1.2169281641642252e-07,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.3400270342826843,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.086162567138672e-07,"mean_question_max_abs_dp":8.940696716308594e-08,"mean_abs_dp":5.5202028968117454e-08,"conv":{"fast_calls":18624,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_10","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":405,"padded_tokens":416,"chunks":26,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":20.486723250010982,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005335807800292969,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.104334613032467e-05,"min_pos_cos":0.9999999997249391,"min_pos_cos_index":89,"mean_pos_cos":0.9999999999897703},"rope_planes_equal_oracle":true,"questions":[{"question_id":"hour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.537854015827179e-08,"max_abs_dlogit":6.67572021484375e-06,"oracle_top2_margin":0.9501125644892454,"near_tie":false},{"question_id":"minute_hand_up","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.695007085800171e-07,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.8967790380120277,"near_tie":false},{"question_id":"confidence","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.3845662275950113e-07,"max_abs_dlogit":6.496906280517578e-06,"oracle_top2_margin":0.7686230093240738,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.3907750447591147e-07,"mean_abs_dp":9.50983828968472e-08,"conv":{"fast_calls":19248,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_10","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":537,"padded_tokens":544,"chunks":34,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":26.912202832987532,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007872581481933594,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.629365416350926e-05,"min_pos_cos":0.999999999167261,"min_pos_cos_index":191,"mean_pos_cos":0.9999999999828699},"rope_planes_equal_oracle":true,"questions":[{"question_id":"hour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.1874362826347351e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.949541799724102,"near_tie":false},{"question_id":"minute_hand_up","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.450580596923828e-08,"mean_abs_dp":6.705522537231445e-08,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.9062174409627914,"near_tie":false},{"question_id":"confidence","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.885197798411052e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.7931901961565018,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":7.388492425282796e-08,"mean_abs_dp":4.6462648444705536e-08,"conv":{"fast_calls":20064,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"img_10","arm":"native","device":"cpu","S":16,"perturb":null,"source":"own_image","tokens":405,"padded_tokens":416,"chunks":26,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":20.6460527080053,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00043570995330810547,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.01774577889559e-06,"min_pos_cos":0.9999999998650566,"min_pos_cos_index":92,"mean_pos_cos":0.9999999999896061},"rope_planes_equal_oracle":true,"questions":[{"question_id":"hour","type":"choice","n_options":4,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.774199962615967e-08,"max_abs_dlogit":9.179115295410156e-06,"oracle_top2_margin":0.9501407779753208,"near_tie":false},{"question_id":"minute_hand_up","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.0617077350616455e-07,"max_abs_dlogit":1.0013580322265625e-05,"oracle_top2_margin":0.8968178927898407,"near_tie":false},{"question_id":"confidence","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.272807518641154e-07,"max_abs_dlogit":1.0967254638671875e-05,"oracle_top2_margin":0.7686530649662018,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.3907750447591147e-07,"mean_abs_dp":9.168353345659044e-08,"conv":{"fast_calls":20688,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_01","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":556,"padded_tokens":560,"chunks":35,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":27.795687917008763,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007143020629882812,"ref_absmax":49.0733528137207,"rel_max_abs_diff":1.4555803140244484e-05,"min_pos_cos":0.9999999998597846,"min_pos_cos_index":303,"mean_pos_cos":0.999999999994029},"rope_planes_equal_oracle":true,"questions":[{"question_id":"limes","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.5497207641601562e-06,"mean_abs_dp":5.530503888924917e-07,"max_abs_dlogit":4.410743713378906e-06,"oracle_top2_margin":0.2646184265613556,"near_tie":false},{"question_id":"has_banana","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.190951585769653e-09,"mean_abs_dp":2.0954757928848267e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9909189967438579,"near_tie":false},{"question_id":"kinds","type":"choice","n_options":5,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.2814998626708984e-06,"mean_abs_dp":6.604939699172974e-07,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.00908571481704712,"near_tie":true},{"question_id":"freshness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.9371509552001953e-07,"mean_abs_dp":1.0679165522257487e-07,"max_abs_dlogit":6.556510925292969e-07,"oracle_top2_margin":0.5698036402463913,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.5497207641601562e-06,"mean_question_max_abs_dp":7.57281668484211e-07,"mean_abs_dp":4.340836312621832e-07,"conv":{"fast_calls":21528,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_01","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":688,"padded_tokens":688,"chunks":43,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":41.16009049999411,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00042629241943359375,"ref_absmax":48.66167068481445,"rel_max_abs_diff":8.760332586908574e-06,"min_pos_cos":0.999999999851705,"min_pos_cos_index":179,"mean_pos_cos":0.999999999992025},"rope_planes_equal_oracle":true,"questions":[{"question_id":"limes","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":9.834766387939453e-07,"mean_abs_dp":3.3853575587272644e-07,"max_abs_dlogit":2.4437904357910156e-06,"oracle_top2_margin":0.056382983922958374,"near_tie":false},{"question_id":"has_banana","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.725290298461914e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.992401025025174,"near_tie":false},{"question_id":"kinds","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":1.7434358596801757e-07,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.23297813534736633,"near_tie":false},{"question_id":"freshness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.4280279477437338e-07,"max_abs_dlogit":1.0132789611816406e-06,"oracle_top2_margin":0.6520294547080994,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.834766387939453e-07,"mean_question_max_abs_dp":4.10713255405426e-07,"mean_abs_dp":2.0844163373112679e-07,"conv":{"fast_calls":22560,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"photo_01","arm":"native","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":1692,"padded_tokens":1696,"chunks":106,"n_image_tokens":1200,"merged_hw":[40,30],"n_image_max_used":1200,"seconds":60.21052725001937,"host":{"start":1237,"amount":1160,"image_tokens":1200},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0024123191833496094,"ref_absmax":48.316856384277344,"rel_max_abs_diff":4.9927072327797294e-05,"min_pos_cos":0.9999999942321458,"min_pos_cos_index":1051,"mean_pos_cos":0.9999999999707782},"rope_planes_equal_oracle":true,"questions":[{"question_id":"limes","type":"choice","n_options":6,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":1.3442089160283407e-07,"max_abs_dlogit":1.2516975402832031e-06,"oracle_top2_margin":0.15403717756271362,"near_tie":false},{"question_id":"has_banana","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":8.614733815193176e-09,"mean_abs_dp":4.307366907596588e-09,"max_abs_dlogit":1.1920928955078125e-06,"oracle_top2_margin":0.9928691615350544,"near_tie":false},{"question_id":"kinds","type":"choice","n_options":5,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":9.573996067047119e-08,"max_abs_dlogit":6.67572021484375e-06,"oracle_top2_margin":0.4216667264699936,"near_tie":false},{"question_id":"freshness","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.086162567138672e-07,"mean_abs_dp":1.310060421625773e-07,"max_abs_dlogit":2.9802322387695312e-06,"oracle_top2_margin":0.7046183198690414,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_question_max_abs_dp":2.0331935957074165e-07,"mean_abs_dp":1.0542862582951784e-07,"conv":{"fast_calls":25104,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_02","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":499,"padded_tokens":512,"chunks":32,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":17.779716750024818,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004024505615234375,"ref_absmax":49.5633544921875,"rel_max_abs_diff":8.11992177783032e-06,"min_pos_cos":0.9999999998535939,"min_pos_cos_index":99,"mean_pos_cos":0.9999999999936853},"rope_planes_equal_oracle":true,"questions":[{"question_id":"animal","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":1.1641532182693481e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9301810264587402,"near_tie":false},{"question_id":"eye_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.146728992462158e-08,"max_abs_dlogit":5.125999450683594e-06,"oracle_top2_margin":0.94605927541852,"near_tie":false},{"question_id":"background","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.271666208902995e-07,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.5364251285791397,"near_tie":false},{"question_id":"striped","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.725290298461914e-08,"mean_abs_dp":1.862645149230957e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8525290191173553,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_question_max_abs_dp":2.0419247448444366e-07,"mean_abs_dp":1.2071373370977548e-07,"conv":{"fast_calls":25872,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_02","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":631,"padded_tokens":640,"chunks":40,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":22.247595124994405,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00048542022705078125,"ref_absmax":49.467185974121094,"rel_max_abs_diff":9.8129743483838e-06,"min_pos_cos":0.9999999997848018,"min_pos_cos_index":132,"mean_pos_cos":0.999999999990089},"rope_planes_equal_oracle":true,"questions":[{"question_id":"animal","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.377216100692749e-08,"max_abs_dlogit":1.3709068298339844e-06,"oracle_top2_margin":0.9437561091035604,"near_tie":false},{"question_id":"eye_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":5.029141902923584e-08,"mean_abs_dp":2.3515895009040833e-08,"max_abs_dlogit":2.8014183044433594e-06,"oracle_top2_margin":0.9384196568280458,"near_tie":false},{"question_id":"background","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-07,"mean_abs_dp":1.0927518208821614e-07,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.5623989552259445,"near_tie":false},{"question_id":"striped","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.1920928955078125e-07,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8637556880712509,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-07,"mean_question_max_abs_dp":1.0943040251731873e-07,"mean_abs_dp":6.426125764846802e-08,"conv":{"fast_calls":26832,"checked_vs_F_conv1d":96,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_02","arm":"native","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":1035,"padded_tokens":1040,"chunks":65,"n_image_tokens":600,"merged_hw":[20,30],"n_image_max_used":1024,"seconds":42.35778200000641,"host":{"start":637,"amount":570,"image_tokens":600},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00079345703125,"ref_absmax":48.347782135009766,"rel_max_abs_diff":1.6411446321038978e-05,"min_pos_cos":0.9999999985934064,"min_pos_cos_index":485,"mean_pos_cos":0.9999999999841952},"rope_planes_equal_oracle":true,"questions":[{"question_id":"animal","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.210719347000122e-08,"mean_abs_dp":6.05359673500061e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.949346523731947,"near_tie":false},{"question_id":"eye_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_abs_dp":9.66247171163559e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9311072155833244,"near_tie":false},{"question_id":"background","type":"score","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.940696716308594e-08,"mean_abs_dp":5.0912300745646157e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.6139450818300247,"near_tie":false},{"question_id":"striped","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.568167686462402e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.8699666932225227,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":5.98374754190445e-08,"mean_abs_dp":2.976650228867164e-08,"conv":{"fast_calls":28392,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"photo_03","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":505,"padded_tokens":512,"chunks":32,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":17.823991041979752,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00092315673828125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.9106271767925434e-05,"min_pos_cos":0.9999999960609904,"min_pos_cos_index":116,"mean_pos_cos":0.9999999999844932},"rope_planes_equal_oracle":true,"questions":[{"question_id":"snow_visible","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.470348358154297e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.8396650552749634,"near_tie":false},{"question_id":"reflection","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.705522537231445e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.903554379940033,"near_tie":false},{"question_id":"sky","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.0046870708465576e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.905474217608571,"near_tie":false},{"question_id":"wildness","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.7264620661735535e-08,"max_abs_dlogit":1.1920928955078125e-06,"oracle_top2_margin":0.8537349253892899,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":8.940696716308594e-08,"mean_abs_dp":4.7730281949043274e-08,"conv":{"fast_calls":29160,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_03","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":637,"padded_tokens":640,"chunks":40,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":22.336862209020182,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0002613067626953125,"ref_absmax":48.316856384277344,"rel_max_abs_diff":5.4081904794688506e-06,"min_pos_cos":0.9999999982837947,"min_pos_cos_index":247,"mean_pos_cos":0.9999999999885267},"rope_planes_equal_oracle":true,"questions":[{"question_id":"snow_visible","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.470348358154297e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8430746719241142,"near_tie":false},{"question_id":"reflection","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":1.043081283569336e-07,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9003455862402916,"near_tie":false},{"question_id":"sky","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":3.771856427192688e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.912934297695756,"near_tie":false},{"question_id":"wildness","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.005858838558197e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.8666355535387993,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.043081283569336e-07,"mean_abs_dp":5.4094319542249046e-08,"conv":{"fast_calls":30120,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_03","arm":"native","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":1041,"padded_tokens":1056,"chunks":66,"n_image_tokens":600,"merged_hw":[20,30],"n_image_max_used":1024,"seconds":37.00216870900476,"host":{"start":637,"amount":570,"image_tokens":600},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009577274322509766,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.9821807624111656e-05,"min_pos_cos":0.9999999990122198,"min_pos_cos_index":718,"mean_pos_cos":0.9999999999836136},"rope_planes_equal_oracle":true,"questions":[{"question_id":"snow_visible","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":1.1175870895385742e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8728517517447472,"near_tie":false},{"question_id":"reflection","type":"noul","n_options":2,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.450580596923828e-08,"mean_abs_dp":3.725290298461914e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.911050520837307,"near_tie":false},{"question_id":"sky","type":"choice","n_options":4,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.119930624961853e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9153185542672873,"near_tie":false},{"question_id":"wildness","type":"score","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.086162567138672e-07,"mean_abs_dp":1.0174699127674103e-07,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8406233042478561,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.086162567138672e-07,"mean_question_max_abs_dp":9.12696123123169e-08,"mean_abs_dp":5.2386894822120667e-08,"conv":{"fast_calls":31704,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_04","arm":"g256","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":502,"padded_tokens":512,"chunks":32,"n_image_tokens":64,"merged_hw":[8,8],"n_image_max_used":1024,"seconds":23.581633499998134,"host":{"start":101,"amount":56,"image_tokens":64},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0015048980712890625,"ref_absmax":48.316856384277344,"rel_max_abs_diff":3.114644006059068e-05,"min_pos_cos":0.9999999975213769,"min_pos_cos_index":387,"mean_pos_cos":0.9999999999810807},"rope_planes_equal_oracle":true,"questions":[{"question_id":"drink","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":6.51925802230835e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9471176210790873,"near_tie":false},{"question_id":"milk_added","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":5.587935447692871e-09,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9886021264828742,"near_tie":false},{"question_id":"cup_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":2.2817403078079224e-08,"mean_abs_dp":7.566995918750763e-09,"max_abs_dlogit":1.4901161193847656e-06,"oracle_top2_margin":0.9695334816351533,"near_tie":false},{"question_id":"beans","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.1420419216156006e-08,"mean_abs_dp":1.334895690282186e-08,"max_abs_dlogit":2.8386712074279785e-06,"oracle_top2_margin":0.9111810363829136,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2817403078079224e-08,"mean_question_max_abs_dp":1.7578713595867157e-08,"mean_abs_dp":8.27444287446829e-09,"conv":{"fast_calls":32472,"checked_vs_F_conv1d":144,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"photo_04","arm":"g448","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":634,"padded_tokens":640,"chunks":40,"n_image_tokens":196,"merged_hw":[14,14],"n_image_max_used":1024,"seconds":22.227159707981627,"host":{"start":233,"amount":182,"image_tokens":196},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0014996528625488281,"ref_absmax":48.31694412231445,"rel_max_abs_diff":3.103782513133392e-05,"min_pos_cos":0.9999999949185113,"min_pos_cos_index":134,"mean_pos_cos":0.9999999999687591},"rope_planes_equal_oracle":true,"questions":[{"question_id":"drink","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":9.313225746154785e-09,"max_abs_dlogit":1.1324882507324219e-06,"oracle_top2_margin":0.9513880060985684,"near_tie":false},{"question_id":"milk_added","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.122274160385132e-09,"mean_abs_dp":2.561137080192566e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.989408596418798,"near_tie":false},{"question_id":"cup_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.0710209608078003e-08,"mean_abs_dp":6.984919309616089e-09,"max_abs_dlogit":1.2516975402832031e-06,"oracle_top2_margin":0.970035164617002,"near_tie":false},{"question_id":"beans","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_abs_dp":1.6763806343078613e-08,"max_abs_dlogit":1.0505318641662598e-06,"oracle_top2_margin":0.9172959104180336,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_question_max_abs_dp":2.2118911147117615e-08,"mean_abs_dp":9.277405647131113e-09,"conv":{"fast_calls":33432,"checked_vs_F_conv1d":144,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"photo_04","arm":"native","device":"cpu","S":16,"perturb":null,"source":"cc0_photo","tokens":1038,"padded_tokens":1040,"chunks":65,"n_image_tokens":600,"merged_hw":[20,30],"n_image_max_used":1024,"seconds":36.85965195801691,"host":{"start":637,"amount":570,"image_tokens":600},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0019712448120117188,"ref_absmax":48.316856384277344,"rel_max_abs_diff":4.079828365351137e-05,"min_pos_cos":0.9999999972379414,"min_pos_cos_index":619,"mean_pos_cos":0.9999999999659842},"rope_planes_equal_oracle":true,"questions":[{"question_id":"drink","type":"choice","n_options":4,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.0244548320770264e-08,"mean_abs_dp":5.3551048040390015e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.951157646253705,"near_tie":false},{"question_id":"milk_added","type":"noul","n_options":2,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":2.3283064365386963e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9896592232398689,"near_tie":false},{"question_id":"cup_color","type":"choice","n_options":4,"argmax":3,"argmax_oracle":3,"argmax_equal":true,"max_abs_dp":1.1641532182693481e-08,"mean_abs_dp":2.9103830456733704e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.970263154245913,"near_tie":false},{"question_id":"beans","type":"score","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.967053731282552e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9172757267951965,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":3.64379957318306e-08,"mean_abs_dp":1.4363859708492573e-08,"conv":{"fast_calls":34992,"checked_vs_F_conv1d":144,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a3f18f3a63d45345942b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":185,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.365620124997804,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000446319580078125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.23732973903833e-06,"min_pos_cos":0.999999999799893,"min_pos_cos_index":125,"mean_pos_cos":0.9999999999828854},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.009374141693115e-08,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.9059995450079441,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.009374141693115e-08,"conv":{"fast_calls":3384,"checked_vs_F_conv1d":120,"mismatched":0,"max_abs_diff":0.0}},{"kind":"p1","id":"semif_f40beba9088c8db8bbd6","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":183,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.452694916981272,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005950927734375,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.2316459100118443e-05,"min_pos_cos":0.9999999996409217,"min_pos_cos_index":123,"mean_pos_cos":0.9999999999803806},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.195638656616211e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8385190591216087,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.195638656616211e-08,"conv":{"fast_calls":35280,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_0b43ea8e24e74d621f1b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":177,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.690391209005611,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00031507015228271484,"ref_absmax":48.31693649291992,"rel_max_abs_diff":6.520904990093553e-06,"min_pos_cos":0.999999999905977,"min_pos_cos_index":162,"mean_pos_cos":0.999999999990889},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":9.313225746154785e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9089595079421997,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_question_max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":9.313225746154785e-09,"conv":{"fast_calls":35568,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_942ec65dac0bfb9874f9","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":177,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.598866957996506,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005931854248046875,"ref_absmax":48.31693649291992,"rel_max_abs_diff":1.2276966791791723e-05,"min_pos_cos":0.9999999996131484,"min_pos_cos_index":139,"mean_pos_cos":0.9999999999838821},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_abs_dp":3.948807716369629e-07,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.7890182957053185,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_question_max_abs_dp":6.556510925292969e-07,"mean_abs_dp":3.948807716369629e-07,"conv":{"fast_calls":35856,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_e0c140e9222d1bce5647","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.6289412080077454,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004520416259765625,"ref_absmax":48.31690979003906,"rel_max_abs_diff":9.355764429904717e-06,"min_pos_cos":0.9999999998696932,"min_pos_cos_index":60,"mean_pos_cos":0.9999999999882274},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":9.934107462565104e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8911582641303539,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_question_max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":9.934107462565104e-09,"conv":{"fast_calls":36144,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_8e8c3804a3c15ebb31e3","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.718185792007716,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006313323974609375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.306650400514007e-05,"min_pos_cos":0.9999999997244188,"min_pos_cos_index":148,"mean_pos_cos":0.9999999999825812},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_abs_dp":5.016724268595377e-07,"max_abs_dlogit":8.58306884765625e-06,"oracle_top2_margin":0.7645951732993126,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.748603820800781e-07,"mean_question_max_abs_dp":7.748603820800781e-07,"mean_abs_dp":5.016724268595377e-07,"conv":{"fast_calls":36432,"checked_vs_F_conv1d":168,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_56c419b12cd37cbe790c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":188,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.64050729200244,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007390975952148438,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.529688429902531e-05,"min_pos_cos":0.9999999991488692,"min_pos_cos_index":65,"mean_pos_cos":0.9999999999738605},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.2665987014770508e-07,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9058033861219883,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.2665987014770508e-07,"conv":{"fast_calls":36720,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_75b470586834df8c568d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":181,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.769148291990859,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003490447998046875,"ref_absmax":48.316898345947266,"rel_max_abs_diff":7.224072979716935e-06,"min_pos_cos":0.9999999999158395,"min_pos_cos_index":164,"mean_pos_cos":0.9999999999905966},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.1300047238667806e-07,"max_abs_dlogit":3.814697265625e-06,"oracle_top2_margin":0.8789017200469971,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.1300047238667806e-07,"conv":{"fast_calls":37008,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_834f3b3e10cc61b33d25","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.052961209003115,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000293731689453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.079268289794456e-06,"min_pos_cos":0.9999999997751114,"min_pos_cos_index":113,"mean_pos_cos":0.9999999999866145},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.029141902923584e-08,"mean_abs_dp":3.166496753692627e-08,"max_abs_dlogit":2.5033950805664062e-06,"oracle_top2_margin":0.9390244763344526,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.029141902923584e-08,"mean_question_max_abs_dp":5.029141902923584e-08,"mean_abs_dp":3.166496753692627e-08,"conv":{"fast_calls":37272,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_76c7a79b1ff2363a60c5","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.142260124994209,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00040984153747558594,"ref_absmax":48.31694793701172,"rel_max_abs_diff":8.482355673828466e-06,"min_pos_cos":0.9999999998082248,"min_pos_cos_index":57,"mean_pos_cos":0.9999999999879218},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.842877388000488e-08,"max_abs_dlogit":3.993511199951172e-06,"oracle_top2_margin":0.9637667611241341,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.842877388000488e-08,"conv":{"fast_calls":37536,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_36c71bb649b7d5eccb1f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":174,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.071438541985117,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007715225219726562,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.5967960809689698e-05,"min_pos_cos":0.9999999992573095,"min_pos_cos_index":71,"mean_pos_cos":0.999999999987543},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.829698880513509e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9576189629733562,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_question_max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.829698880513509e-09,"conv":{"fast_calls":37800,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_eac5121a4a1bf391a08c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":173,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.066667709004832,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000823974609375,"ref_absmax":48.3169059753418,"rel_max_abs_diff":1.705354663635767e-05,"min_pos_cos":0.9999999997172614,"min_pos_cos_index":133,"mean_pos_cos":0.9999999999870733},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.592622518539429e-08,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.9559501903131604,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.592622518539429e-08,"conv":{"fast_calls":38064,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_44574c44e243ee1b2f52","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.63409400000819,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004024505615234375,"ref_absmax":48.316856384277344,"rel_max_abs_diff":8.329402855240347e-06,"min_pos_cos":0.9999999997915591,"min_pos_cos_index":140,"mean_pos_cos":0.9999999999899356},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.8362881342569985e-08,"max_abs_dlogit":5.960464477539062e-07,"oracle_top2_margin":0.9197894316166639,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.8362881342569985e-08,"conv":{"fast_calls":38352,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_2608370d01783735c3ea","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.625574625009904,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004405975341796875,"ref_absmax":48.316856384277344,"rel_max_abs_diff":9.118919713556967e-06,"min_pos_cos":0.9999999994527254,"min_pos_cos_index":148,"mean_pos_cos":0.9999999999860475},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_abs_dp":2.2351741790771484e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.8671790547668934,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.470348358154297e-08,"mean_question_max_abs_dp":4.470348358154297e-08,"mean_abs_dp":2.2351741790771484e-08,"conv":{"fast_calls":38640,"checked_vs_F_conv1d":192,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_ec94ef909f7918695ab3","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":180,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.50274537500809,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006663352251052856,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.3790921272054519e-05,"min_pos_cos":0.9999999998266549,"min_pos_cos_index":163,"mean_pos_cos":0.9999999999891593},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":3.725290298461914e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9206033553928137,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_question_max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":3.725290298461914e-09,"conv":{"fast_calls":38928,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_03d919fd0e9b3a7cd1e6","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.081232208001893,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004792213439941406,"ref_absmax":50.02640151977539,"rel_max_abs_diff":9.579368682048914e-06,"min_pos_cos":0.9999999998832587,"min_pos_cos_index":145,"mean_pos_cos":0.9999999999881263},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.1175870895385742e-07,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.843472070991993,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.1175870895385742e-07,"conv":{"fast_calls":39192,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_86a15618ba4cb90a307e","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":179,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.621086291008396,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004100799560546875,"ref_absmax":48.31689453125,"rel_max_abs_diff":8.487299526037614e-06,"min_pos_cos":0.999999999847358,"min_pos_cos_index":105,"mean_pos_cos":0.9999999999821672},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.1457672119140625e-06,"mean_abs_dp":1.4180938402811687e-06,"max_abs_dlogit":1.239776611328125e-05,"oracle_top2_margin":0.5598671287298203,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.1457672119140625e-06,"mean_question_max_abs_dp":2.1457672119140625e-06,"mean_abs_dp":1.4180938402811687e-06,"conv":{"fast_calls":39480,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_3dd9c3ac98f8ddf7bdd3","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":179,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.64174545800779,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00027942657470703125,"ref_absmax":48.31689453125,"rel_max_abs_diff":5.7832064212302814e-06,"min_pos_cos":0.9999999999095014,"min_pos_cos_index":58,"mean_pos_cos":0.9999999999892614},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.91155481338501e-08,"mean_abs_dp":1.4901161193847656e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9253548197448254,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.91155481338501e-08,"mean_question_max_abs_dp":3.91155481338501e-08,"mean_abs_dp":1.4901161193847656e-08,"conv":{"fast_calls":39768,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_36273dd4777876a26338","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.0830890420184005,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007410049438476562,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.533633591289056e-05,"min_pos_cos":0.9999999991897072,"min_pos_cos_index":160,"mean_pos_cos":0.9999999999862075},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.4901161193847656e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9322653636336327,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_question_max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.4901161193847656e-08,"conv":{"fast_calls":40032,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_07ee70cfe4b01330ce89","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.0494569999864325,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000293731689453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.079268289794456e-06,"min_pos_cos":0.999999999804683,"min_pos_cos_index":135,"mean_pos_cos":0.9999999999893523},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.849466641743978e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.7554071396589279,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.849466641743978e-08,"conv":{"fast_calls":40296,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_51ceab3ce622c7c759f1","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":191,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.606265583017375,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012187957763671875,"ref_absmax":48.316864013671875,"rel_max_abs_diff":2.522505964009406e-05,"min_pos_cos":0.9999999984812604,"min_pos_cos_index":174,"mean_pos_cos":0.9999999999792766},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.346172014872233e-08,"max_abs_dlogit":3.5762786865234375e-06,"oracle_top2_margin":0.930072607472539,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":5.960464477539063e-08,"mean_abs_dp":4.346172014872233e-08,"conv":{"fast_calls":40584,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b8091be929b1e489fcc4","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":191,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.638333790993784,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00045359134674072266,"ref_absmax":48.316864013671875,"rel_max_abs_diff":9.387847411048308e-06,"min_pos_cos":0.9999999994458875,"min_pos_cos_index":151,"mean_pos_cos":0.9999999999826737},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.685754776000977e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.8859123326838017,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.685754776000977e-08,"conv":{"fast_calls":40872,"checked_vs_F_conv1d":216,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_c89b86e3f55123bfa506","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":182,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":18.097749125008704,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004062652587890625,"ref_absmax":49.80162048339844,"rel_max_abs_diff":8.157671474254388e-06,"min_pos_cos":0.9999999997560344,"min_pos_cos_index":63,"mean_pos_cos":0.99999999998552},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.204510529836019e-07,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.8885854668915272,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.204510529836019e-07,"conv":{"fast_calls":41160,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_3b0958cabc339005cf66","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":178,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":9.522156833001645,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006765127182006836,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.400157255794021e-05,"min_pos_cos":0.9999999997215604,"min_pos_cos_index":163,"mean_pos_cos":0.9999999999875645},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.381903171539307e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9334130901843309,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.381903171539307e-08,"conv":{"fast_calls":41448,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f2b4ec4930322fe45118","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":193,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":10.100612000009278,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000518798828125,"ref_absmax":48.3168830871582,"rel_max_abs_diff":1.073742333894232e-05,"min_pos_cos":0.9999999997538469,"min_pos_cos_index":178,"mean_pos_cos":0.9999999999809052},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.0978193283081055e-08,"mean_abs_dp":2.2972623507181804e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9006375558674335,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.0978193283081055e-08,"mean_question_max_abs_dp":4.0978193283081055e-08,"mean_abs_dp":2.2972623507181804e-08,"conv":{"fast_calls":41760,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_61ac20b07a5189231e15","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":199,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.919297665997874,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012187957763671875,"ref_absmax":48.3169059753418,"rel_max_abs_diff":2.522503773294572e-05,"min_pos_cos":0.999999998120845,"min_pos_cos_index":138,"mean_pos_cos":0.9999999999714945},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":6.829698880513509e-09,"max_abs_dlogit":2.2649765014648438e-06,"oracle_top2_margin":0.8907307609915733,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_question_max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":6.829698880513509e-09,"conv":{"fast_calls":42072,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4b3d794c004566610181","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":184,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":9.131549000012456,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000881195068359375,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.8237830787669015e-05,"min_pos_cos":0.999999998595456,"min_pos_cos_index":123,"mean_pos_cos":0.9999999999637765},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.2665987014770508e-07,"mean_abs_dp":8.568167686462402e-08,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.9178899675607681,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.2665987014770508e-07,"mean_question_max_abs_dp":1.2665987014770508e-07,"mean_abs_dp":8.568167686462402e-08,"conv":{"fast_calls":42360,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b7c4e62d07986286fce0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":9.266514292015927,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005035400390625,"ref_absmax":52.73832702636719,"rel_max_abs_diff":9.547895571483502e-06,"min_pos_cos":0.9999999991846814,"min_pos_cos_index":174,"mean_pos_cos":0.9999999999669039},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.5033950805664062e-06,"mean_abs_dp":1.6589959462483723e-06,"max_abs_dlogit":6.389617919921875e-05,"oracle_top2_margin":0.6482845097780228,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.5033950805664062e-06,"mean_question_max_abs_dp":2.5033950805664062e-06,"mean_abs_dp":1.6589959462483723e-06,"conv":{"fast_calls":42648,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_9473603b44af63beab79","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":9.388842624990502,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007381439208984375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.5277122260717236e-05,"min_pos_cos":0.9999999996523513,"min_pos_cos_index":148,"mean_pos_cos":0.9999999999808473},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.408260186513265e-08,"max_abs_dlogit":1.0728836059570312e-06,"oracle_top2_margin":0.9331219494342804,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.408260186513265e-08,"conv":{"fast_calls":42936,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_d41072dc35d8cd14c056","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":195,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":10.006211458006874,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000293731689453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.079268289794456e-06,"min_pos_cos":0.9999999999345114,"min_pos_cos_index":74,"mean_pos_cos":0.9999999999884109},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.421438694000244e-08,"mean_abs_dp":1.0554989178975424e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9314822610467672,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.421438694000244e-08,"mean_question_max_abs_dp":2.421438694000244e-08,"mean_abs_dp":1.0554989178975424e-08,"conv":{"fast_calls":43248,"checked_vs_F_conv1d":240,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_33d6e5da58f0fee2490d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":17.817237290990306,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006465911865234375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.3382285391170912e-05,"min_pos_cos":0.999999999460505,"min_pos_cos_index":151,"mean_pos_cos":0.9999999999719899},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.774199962615967e-08,"mean_abs_dp":2.669791380564372e-08,"max_abs_dlogit":3.814697265625e-06,"oracle_top2_margin":0.9321309849619865,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.774199962615967e-08,"mean_question_max_abs_dp":5.774199962615967e-08,"mean_abs_dp":2.669791380564372e-08,"conv":{"fast_calls":43536,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_823bf736b92e9d8fa66d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.762685207999311,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003871917724609375,"ref_absmax":48.53725814819336,"rel_max_abs_diff":7.977207350253868e-06,"min_pos_cos":0.9999999996449984,"min_pos_cos_index":151,"mean_pos_cos":0.9999999999801835},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.940696716308594e-07,"mean_abs_dp":5.948046843210856e-07,"max_abs_dlogit":6.9141387939453125e-06,"oracle_top2_margin":0.8193516284227371,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":8.940696716308594e-07,"mean_question_max_abs_dp":8.940696716308594e-07,"mean_abs_dp":5.948046843210856e-07,"conv":{"fast_calls":43824,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_8a4c3de28be95c105ca1","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.623924292012816,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005702972412109375,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.1803263985407219e-05,"min_pos_cos":0.9999999994456398,"min_pos_cos_index":55,"mean_pos_cos":0.9999999999656338},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.662441253662109e-06,"mean_abs_dp":3.767510255177816e-06,"max_abs_dlogit":3.4332275390625e-05,"oracle_top2_margin":0.5325551182031631,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.662441253662109e-06,"mean_question_max_abs_dp":5.662441253662109e-06,"mean_abs_dp":3.767510255177816e-06,"conv":{"fast_calls":44112,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_ea7ec66a5e8eac886548","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":192,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.62439524999354,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004115104675292969,"ref_absmax":48.316898345947266,"rel_max_abs_diff":8.516905712425839e-06,"min_pos_cos":0.9999999998117347,"min_pos_cos_index":128,"mean_pos_cos":0.999999999985834},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.692344029744466e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.8700214065611362,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.692344029744466e-08,"conv":{"fast_calls":44400,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_25172cd6ddc49ab0ca70","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":183,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.630171625001822,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000446319580078125,"ref_absmax":48.316871643066406,"rel_max_abs_diff":9.237344325088832e-06,"min_pos_cos":0.9999999997695576,"min_pos_cos_index":143,"mean_pos_cos":0.9999999999821797},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.747842947642009e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8931036777794361,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":9.747842947642009e-08,"conv":{"fast_calls":44688,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_fb73a85d386cf59b0c51","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":185,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.6249729999981355,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005083084106445312,"ref_absmax":51.69054412841797,"rel_max_abs_diff":9.833682721189966e-06,"min_pos_cos":0.9999999995791539,"min_pos_cos_index":168,"mean_pos_cos":0.9999999999807141},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.632538477579753e-07,"max_abs_dlogit":8.463859558105469e-06,"oracle_top2_margin":0.049953460693359375,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.632538477579753e-07,"conv":{"fast_calls":44976,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a02fd582d2881cc462b2","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":195,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.163698125019437,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004215240478515625,"ref_absmax":48.31694793701172,"rel_max_abs_diff":8.724144753536199e-06,"min_pos_cos":0.9999999998547798,"min_pos_cos_index":133,"mean_pos_cos":0.9999999999850726},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":1.0554989178975424e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9422076493501663,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_question_max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":1.0554989178975424e-08,"conv":{"fast_calls":45288,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_19f37d5a97034bd7d04f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":195,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.195813500002259,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007190704345703125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.4882364579561753e-05,"min_pos_cos":0.9999999998180359,"min_pos_cos_index":156,"mean_pos_cos":0.9999999999832188},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.3527612686157227e-08,"mean_abs_dp":1.241763432820638e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.9404052533209324,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.3527612686157227e-08,"mean_question_max_abs_dp":3.3527612686157227e-08,"mean_abs_dp":1.241763432820638e-08,"conv":{"fast_calls":45600,"checked_vs_F_conv1d":264,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_fda0ca94652c6d739ecf","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":194,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":13.054202333005378,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008797645568847656,"ref_absmax":48.316917419433594,"rel_max_abs_diff":1.820820954382563e-05,"min_pos_cos":0.9999999998726552,"min_pos_cos_index":53,"mean_pos_cos":0.9999999999859341},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.2285849253336586e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9348341189324856,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":5.960464477539063e-08,"mean_abs_dp":3.2285849253336586e-08,"conv":{"fast_calls":45912,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_d0e36c4cb10a29b3191c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":193,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2229201669979375,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009870529174804688,"ref_absmax":48.3168830871582,"rel_max_abs_diff":2.0428737418759748e-05,"min_pos_cos":0.999999999769398,"min_pos_cos_index":89,"mean_pos_cos":0.999999999976875},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.0531158447265625e-06,"mean_abs_dp":2.6846925417582193e-06,"max_abs_dlogit":1.9431114196777344e-05,"oracle_top2_margin":0.43476974964141846,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.0531158447265625e-06,"mean_question_max_abs_dp":4.0531158447265625e-06,"mean_abs_dp":2.6846925417582193e-06,"conv":{"fast_calls":46224,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a0a39ace7e5337d038a7","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":197,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.1730420000094455,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005769729614257812,"ref_absmax":50.7144889831543,"rel_max_abs_diff":1.1376886033840011e-05,"min_pos_cos":0.9999999998476748,"min_pos_cos_index":119,"mean_pos_cos":0.9999999999817181},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9543195981532335,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_question_max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":7.450580596923828e-09,"conv":{"fast_calls":46536,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_eeae283841ce75dd25bf","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":197,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.1737384170119185,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004901885986328125,"ref_absmax":50.7144889831543,"rel_max_abs_diff":9.665651936188043e-06,"min_pos_cos":0.9999999998584486,"min_pos_cos_index":178,"mean_pos_cos":0.9999999999823348},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.829698880513509e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9557990459725261,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.3969838619232178e-08,"mean_question_max_abs_dp":1.3969838619232178e-08,"mean_abs_dp":6.829698880513509e-09,"conv":{"fast_calls":46848,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_00bb4cc043f7d2b288d4","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.670494791993406,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006771087646484375,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.4013908745215928e-05,"min_pos_cos":0.9999999997597628,"min_pos_cos_index":174,"mean_pos_cos":0.9999999999818193},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":9.934107462565104e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9388207122683525,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_question_max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":9.934107462565104e-09,"conv":{"fast_calls":47136,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_1a81ead69e93e5d0c86a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":196,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.22207029099809,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009037256240844727,"ref_absmax":49.39069747924805,"rel_max_abs_diff":1.8297486575567823e-05,"min_pos_cos":0.9999999997386576,"min_pos_cos_index":178,"mean_pos_cos":0.9999999999744242},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.725290298461914e-06,"mean_abs_dp":2.4785598119099936e-06,"max_abs_dlogit":3.4928321838378906e-05,"oracle_top2_margin":0.11752080917358398,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.725290298461914e-06,"mean_question_max_abs_dp":3.725290298461914e-06,"mean_abs_dp":2.4785598119099936e-06,"conv":{"fast_calls":47448,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4444dad25e438f128673","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.60399345800397,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007715225219726562,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.5967948202739333e-05,"min_pos_cos":0.9999999996940767,"min_pos_cos_index":173,"mean_pos_cos":0.9999999999773689},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.029141902923584e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.890910916030407,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.029141902923584e-08,"conv":{"fast_calls":47736,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_3226202ba1ada2570e9f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":193,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.160749959002715,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004253387451171875,"ref_absmax":48.3168830871582,"rel_max_abs_diff":8.80310810508874e-06,"min_pos_cos":0.9999999998494339,"min_pos_cos_index":99,"mean_pos_cos":0.9999999999856076},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.60770320892334e-08,"mean_abs_dp":8.692344029744467e-09,"max_abs_dlogit":5.4836273193359375e-06,"oracle_top2_margin":0.7694877237081528,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.60770320892334e-08,"mean_question_max_abs_dp":2.60770320892334e-08,"mean_abs_dp":8.692344029744467e-09,"conv":{"fast_calls":48048,"checked_vs_F_conv1d":288,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_3f783eec5c90812479bc","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.479052084003342,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.001392364501953125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":2.8817310724350342e-05,"min_pos_cos":0.9999999971961799,"min_pos_cos_index":175,"mean_pos_cos":0.999999999947707},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.842877388000488e-08,"mean_abs_dp":1.862645149230957e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8997703716158867,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.842877388000488e-08,"mean_question_max_abs_dp":4.842877388000488e-08,"mean_abs_dp":1.862645149230957e-08,"conv":{"fast_calls":48336,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_ffee861474a9665a093a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":191,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.629960790975019,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008730888366699219,"ref_absmax":48.316864013671875,"rel_max_abs_diff":1.8070064241397583e-05,"min_pos_cos":0.9999999989447482,"min_pos_cos_index":174,"mean_pos_cos":0.9999999999815912},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.066394805908203e-07,"mean_abs_dp":2.7815500895182293e-07,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.6625179350376129,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.066394805908203e-07,"mean_question_max_abs_dp":5.066394805908203e-07,"mean_abs_dp":2.7815500895182293e-07,"conv":{"fast_calls":48624,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_8c992e17e4fb54b044f5","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":178,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.645774459000677,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00057220458984375,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.1842739784689516e-05,"min_pos_cos":0.9999999996161243,"min_pos_cos_index":138,"mean_pos_cos":0.9999999999863824},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.395081679026286e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9447127860039473,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.395081679026286e-08,"conv":{"fast_calls":48912,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_695dd61195ca025c4f4c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":179,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.67380604200298,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00024318695068359375,"ref_absmax":48.31689453125,"rel_max_abs_diff":5.03316599799905e-06,"min_pos_cos":0.9999999997413729,"min_pos_cos_index":65,"mean_pos_cos":0.9999999999904583},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.190951585769653e-08,"mean_abs_dp":1.7074247201283772e-08,"max_abs_dlogit":2.6226043701171875e-06,"oracle_top2_margin":0.9609713619574904,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.190951585769653e-08,"mean_question_max_abs_dp":4.190951585769653e-08,"mean_abs_dp":1.7074247201283772e-08,"conv":{"fast_calls":49200,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_e6155431ab7b0f8170d4","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.64140012502321,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00031948089599609375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.612191159354359e-06,"min_pos_cos":0.9999999998887701,"min_pos_cos_index":113,"mean_pos_cos":0.9999999999917935},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":0.0,"mean_abs_dp":0.0,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9604581100866199,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":0.0,"mean_question_max_abs_dp":0.0,"mean_abs_dp":0.0,"conv":{"fast_calls":49464,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b49ecda75086d8c68d2a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.423429291986395,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000293731689453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.079268289794456e-06,"min_pos_cos":0.9999999999074888,"min_pos_cos_index":128,"mean_pos_cos":0.9999999999899298},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-06,"mean_abs_dp":9.723007678985596e-07,"max_abs_dlogit":8.702278137207031e-06,"oracle_top2_margin":0.5389100611209869,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-06,"mean_question_max_abs_dp":1.4901161193847656e-06,"mean_abs_dp":9.723007678985596e-07,"conv":{"fast_calls":49728,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_587ced261c6092cd1325","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.728485915984493,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0010042190551757812,"ref_absmax":48.316856384277344,"rel_max_abs_diff":2.0784031295185037e-05,"min_pos_cos":0.9999999994542337,"min_pos_cos_index":146,"mean_pos_cos":0.9999999999811681},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.7008354663848877e-08,"mean_abs_dp":1.5522042910257976e-08,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.970055628567934,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7008354663848877e-08,"mean_question_max_abs_dp":2.7008354663848877e-08,"mean_abs_dp":1.5522042910257976e-08,"conv":{"fast_calls":50016,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_533d4423d311de82b27d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.8795427920122165,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007929801940917969,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.6412063551615554e-05,"min_pos_cos":0.9999999992649762,"min_pos_cos_index":130,"mean_pos_cos":0.999999999976596},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_abs_dp":1.0865430037180582e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9584523839876056,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.955777406692505e-08,"mean_question_max_abs_dp":1.955777406692505e-08,"mean_abs_dp":1.0865430037180582e-08,"conv":{"fast_calls":50304,"checked_vs_F_conv1d":312,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_2bf540d515a51755b25c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":178,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.49391808299697,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009160041809082031,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.89582526053238e-05,"min_pos_cos":0.999999999721314,"min_pos_cos_index":136,"mean_pos_cos":0.9999999999843123},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":9.313225746154785e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9586424399167299,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_question_max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":9.313225746154785e-09,"conv":{"fast_calls":50592,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_fdb9fc214aea2710763a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":168,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.081408541998826,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006666183471679688,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.3796800563398964e-05,"min_pos_cos":0.9999999995029991,"min_pos_cos_index":151,"mean_pos_cos":0.9999999999827344},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.30385160446167e-07,"mean_abs_dp":9.064873059590657e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.8761075809597969,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.30385160446167e-07,"mean_question_max_abs_dp":1.30385160446167e-07,"mean_abs_dp":9.064873059590657e-08,"conv":{"fast_calls":50856,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f6d68126b43f8c9ed183","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":181,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.632674374995986,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004978179931640625,"ref_absmax":48.316898345947266,"rel_max_abs_diff":1.0303186053038908e-05,"min_pos_cos":0.9999999988127664,"min_pos_cos_index":141,"mean_pos_cos":0.9999999999807202},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.7939677238464355e-08,"max_abs_dlogit":1.6689300537109375e-06,"oracle_top2_margin":0.9405296631157398,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.7939677238464355e-08,"conv":{"fast_calls":51144,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_44bae0c8a1615a096c43","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":184,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.660806583007798,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000286102294921875,"ref_absmax":48.31687927246094,"rel_max_abs_diff":5.92137363236007e-06,"min_pos_cos":0.9999999997780564,"min_pos_cos_index":144,"mean_pos_cos":0.9999999999850523},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9683361621573567,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.6763806343078613e-08,"mean_question_max_abs_dp":1.6763806343078613e-08,"mean_abs_dp":7.450580596923828e-09,"conv":{"fast_calls":51432,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_2175379f4c69e6207626","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":175,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.0991864999814425,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00045871734619140625,"ref_absmax":51.562374114990234,"rel_max_abs_diff":8.89635813060919e-06,"min_pos_cos":0.9999999994991244,"min_pos_cos_index":76,"mean_pos_cos":0.9999999999820818},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.502533753712972e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.9050104506313801,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.502533753712972e-07,"conv":{"fast_calls":51696,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_def9fe59d262357a673a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.092156958009582,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00029659271240234375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.1384819419677786e-06,"min_pos_cos":0.9999999998405344,"min_pos_cos_index":130,"mean_pos_cos":0.9999999999907472},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_abs_dp":1.9495685895284018e-07,"max_abs_dlogit":4.291534423828125e-06,"oracle_top2_margin":0.8392161056399345,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_question_max_abs_dp":2.980232238769531e-07,"mean_abs_dp":1.9495685895284018e-07,"conv":{"fast_calls":51960,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_712f6ffb763eef7c9a12","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":181,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.647438290994614,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00039649009704589844,"ref_absmax":48.316898345947266,"rel_max_abs_diff":8.206033719446217e-06,"min_pos_cos":0.9999999990509901,"min_pos_cos_index":108,"mean_pos_cos":0.999999999983783},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":4.6566128730773926e-09,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9771416820585728,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_question_max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":4.6566128730773926e-09,"conv":{"fast_calls":52248,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_e1d610bd14e3d16c09b9","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":181,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.641386292001698,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0002726316452026367,"ref_absmax":48.316898345947266,"rel_max_abs_diff":5.642573396384095e-06,"min_pos_cos":0.9999999998022037,"min_pos_cos_index":122,"mean_pos_cos":0.9999999999899565},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.1420419216156006e-08,"mean_abs_dp":7.140139738718669e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.95653619710356,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.1420419216156006e-08,"mean_question_max_abs_dp":2.1420419216156006e-08,"mean_abs_dp":7.140139738718669e-09,"conv":{"fast_calls":52536,"checked_vs_F_conv1d":336,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_22ec57822db50ec121cf","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":171,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":11.900949375005439,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000396728515625,"ref_absmax":48.3168830871582,"rel_max_abs_diff":8.210970788602952e-06,"min_pos_cos":0.9999999998709986,"min_pos_cos_index":93,"mean_pos_cos":0.9999999999925827},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":5.960464477539062e-07,"oracle_top2_margin":0.9663570886477828,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_question_max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"conv":{"fast_calls":52800,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_3e75f2b84e8e08e3d3f2","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.075166208000155,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000396728515625,"ref_absmax":51.44828414916992,"rel_max_abs_diff":7.711209852494195e-06,"min_pos_cos":0.9999999998623593,"min_pos_cos_index":155,"mean_pos_cos":0.9999999999897685},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.364418029785156e-07,"mean_abs_dp":3.2782554626464844e-07,"max_abs_dlogit":2.1457672119140625e-06,"oracle_top2_margin":0.5234462171792984,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.364418029785156e-07,"mean_question_max_abs_dp":5.364418029785156e-07,"mean_abs_dp":3.2782554626464844e-07,"conv":{"fast_calls":53064,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_ebd74aa05324a1556288","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.606828790972941,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003032684326171875,"ref_absmax":48.316856384277344,"rel_max_abs_diff":6.2766590236171335e-06,"min_pos_cos":0.9999999999154189,"min_pos_cos_index":144,"mean_pos_cos":0.9999999999907832},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.370626846949259e-08,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.9519742447882891,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.370626846949259e-08,"conv":{"fast_calls":53352,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_d7e84e4245a29817da1f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":184,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.630591833993094,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0002689361572265625,"ref_absmax":48.31687927246094,"rel_max_abs_diff":5.566091214418466e-06,"min_pos_cos":0.9999999997779877,"min_pos_cos_index":137,"mean_pos_cos":0.9999999999894407},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.687656958897909e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9693773267790675,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.687656958897909e-08,"conv":{"fast_calls":53640,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f41ae66b3c09630c7a42","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":178,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.654581959010102,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00024127960205078125,"ref_absmax":52.76446533203125,"rel_max_abs_diff":4.57276692813013e-06,"min_pos_cos":0.9999999998851452,"min_pos_cos_index":136,"mean_pos_cos":0.9999999999911922},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.3659397761027018e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9546181615442038,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_question_max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.3659397761027018e-08,"conv":{"fast_calls":53928,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_82c2cd97c160ad27951b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":171,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.077205457986565,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00038433074951171875,"ref_absmax":48.3168830871582,"rel_max_abs_diff":7.95437795145911e-06,"min_pos_cos":0.999999999838451,"min_pos_cos_index":112,"mean_pos_cos":0.999999999986361},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.502533753712972e-07,"max_abs_dlogit":4.5299530029296875e-06,"oracle_top2_margin":0.790655829012394,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.502533753712972e-07,"conv":{"fast_calls":54192,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_7ada97b6f99f4788e5c7","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":184,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.633487166982377,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000362396240234375,"ref_absmax":48.31687927246094,"rel_max_abs_diff":7.500406600989422e-06,"min_pos_cos":0.999999999814474,"min_pos_cos_index":106,"mean_pos_cos":0.9999999999884734},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":1.5522042910257976e-09,"max_abs_dlogit":1.0728836059570312e-06,"oracle_top2_margin":0.9695109454914927,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.6566128730773926e-09,"mean_question_max_abs_dp":4.6566128730773926e-09,"mean_abs_dp":1.5522042910257976e-09,"conv":{"fast_calls":54480,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4d7a003342db5b0a17e3","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":187,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.609137750027003,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0002841949462890625,"ref_absmax":48.31692123413086,"rel_max_abs_diff":5.881892699907966e-06,"min_pos_cos":0.9999999997291371,"min_pos_cos_index":148,"mean_pos_cos":0.9999999999874886},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.991508563359578e-08,"max_abs_dlogit":3.6954879760742188e-06,"oracle_top2_margin":0.9542434560135007,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.991508563359578e-08,"conv":{"fast_calls":54768,"checked_vs_F_conv1d":360,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_19d3d8d0bf773e88be7b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":172,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":11.878879957977915,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004540681838989258,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.397700047007745e-06,"min_pos_cos":0.99999999980329,"min_pos_cos_index":74,"mean_pos_cos":0.9999999999898518},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7695128917694092e-08,"mean_abs_dp":5.89837630589803e-09,"max_abs_dlogit":1.5497207641601562e-06,"oracle_top2_margin":0.9612534679472446,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7695128917694092e-08,"mean_question_max_abs_dp":1.7695128917694092e-08,"mean_abs_dp":5.89837630589803e-09,"conv":{"fast_calls":55032,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_d6de731cca52359b1c87","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.057744957972318,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00042724609375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":8.842572057882844e-06,"min_pos_cos":0.9999999997247941,"min_pos_cos_index":131,"mean_pos_cos":0.999999999987772},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.271666208902995e-07,"max_abs_dlogit":3.0994415283203125e-06,"oracle_top2_margin":0.6572877019643784,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_question_max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.271666208902995e-07,"conv":{"fast_calls":55296,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b45c63fb92729af27f05","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":197,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.162262833007844,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00047397613525390625,"ref_absmax":49.72970962524414,"rel_max_abs_diff":9.531045703377749e-06,"min_pos_cos":0.9999999998656499,"min_pos_cos_index":116,"mean_pos_cos":0.9999999999861089},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.384185791015625e-07,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.7186497449874878,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.384185791015625e-07,"conv":{"fast_calls":55608,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_46b7029b9a704138b77a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":195,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.1630773750075605,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003864765167236328,"ref_absmax":49.729698181152344,"rel_max_abs_diff":7.77154358178084e-06,"min_pos_cos":0.9999999998930508,"min_pos_cos_index":92,"mean_pos_cos":0.9999999999856737},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.0978193283081055e-08,"mean_abs_dp":1.5522042910257976e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.926379282027483,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.0978193283081055e-08,"mean_question_max_abs_dp":4.0978193283081055e-08,"mean_abs_dp":1.5522042910257976e-08,"conv":{"fast_calls":55920,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_caf773b23934fa7362a1","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":190,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.672240041021723,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005974769592285156,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.2365784362195541e-05,"min_pos_cos":0.999999999805298,"min_pos_cos_index":56,"mean_pos_cos":0.9999999999870992},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.5390257835388184e-08,"mean_abs_dp":2.2972623507181804e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.926736993715167,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5390257835388184e-08,"mean_question_max_abs_dp":3.5390257835388184e-08,"mean_abs_dp":2.2972623507181804e-08,"conv":{"fast_calls":56208,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f46f392ef9e9e9df564b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":188,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.55878079199465,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000545501708984375,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.1290087508441906e-05,"min_pos_cos":0.999999999498559,"min_pos_cos_index":171,"mean_pos_cos":0.999999999986256},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.2914339701334635e-07,"max_abs_dlogit":4.76837158203125e-06,"oracle_top2_margin":0.8944226875901222,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.2914339701334635e-07,"conv":{"fast_calls":56496,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_005dcf6d2c0f21d64796","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":209,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.711536207993049,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003933906555175781,"ref_absmax":48.31693649291992,"rel_max_abs_diff":8.14187910227345e-06,"min_pos_cos":0.9999999995679136,"min_pos_cos_index":169,"mean_pos_cos":0.999999999979075},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.8736718334257603,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2351741790771484e-08,"mean_question_max_abs_dp":2.2351741790771484e-08,"mean_abs_dp":7.450580596923828e-09,"conv":{"fast_calls":56832,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_edb708bd0fe5fd67d6ee","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":210,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.720319790998474,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008144378662109375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.6856152985339172e-05,"min_pos_cos":0.9999999997284762,"min_pos_cos_index":84,"mean_pos_cos":0.9999999999798436},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4528632164001465e-07,"mean_abs_dp":1.0927518208821614e-07,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.8699474632740021,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4528632164001465e-07,"mean_question_max_abs_dp":1.4528632164001465e-07,"mean_abs_dp":1.0927518208821614e-07,"conv":{"fast_calls":57168,"checked_vs_F_conv1d":384,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_97a35499bd8717f3e786","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":203,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.976411166018806,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00171661376953125,"ref_absmax":48.316925048828125,"rel_max_abs_diff":3.5528208134033245e-05,"min_pos_cos":0.9999999982798438,"min_pos_cos_index":63,"mean_pos_cos":0.9999999999732704},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_abs_dp":8.071462313334147e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9219651073217392,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.862645149230957e-08,"mean_question_max_abs_dp":1.862645149230957e-08,"mean_abs_dp":8.071462313334147e-09,"conv":{"fast_calls":57480,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_31eb4533d2c89959ec46","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":202,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.175280625000596,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004940032958984375,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.0224238471875055e-05,"min_pos_cos":0.9999999999088683,"min_pos_cos_index":143,"mean_pos_cos":0.9999999999879334},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.3907750447591147e-07,"max_abs_dlogit":5.0067901611328125e-06,"oracle_top2_margin":0.8207472041249275,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.3907750447591147e-07,"conv":{"fast_calls":57792,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_0aa24a6701ac9c56c4a0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":194,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.278694334003376,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000523686408996582,"ref_absmax":48.316917419433594,"rel_max_abs_diff":1.0838572429000812e-05,"min_pos_cos":0.9999999998674095,"min_pos_cos_index":177,"mean_pos_cos":0.9999999999877368},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.891787052154541e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.9414865039288998,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.891787052154541e-08,"conv":{"fast_calls":58104,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_796d5c0da6eaadae6a3f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":188,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.61247574997833,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008778572082519531,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.8168750964197158e-05,"min_pos_cos":0.9999999996948022,"min_pos_cos_index":71,"mean_pos_cos":0.999999999989848},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":9.313225746154785e-09,"mean_abs_dp":4.6566128730773926e-09,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9685217225924134,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.313225746154785e-09,"mean_question_max_abs_dp":9.313225746154785e-09,"mean_abs_dp":4.6566128730773926e-09,"conv":{"fast_calls":58392,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_599b3092b6fd18312d23","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":187,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.698351875005756,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.001369476318359375,"ref_absmax":48.31692123413086,"rel_max_abs_diff":2.834361717136859e-05,"min_pos_cos":0.9999999876683978,"min_pos_cos_index":108,"mean_pos_cos":0.9999999999123327},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.761021455128987e-08,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.9458253737539053,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.761021455128987e-08,"conv":{"fast_calls":58680,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_960a4bcd27363e6db4b9","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.739676083001541,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0010976791381835938,"ref_absmax":48.49167251586914,"rel_max_abs_diff":2.2636446243927198e-05,"min_pos_cos":0.999999996664808,"min_pos_cos_index":130,"mean_pos_cos":0.9999999999519618},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.874301910400391e-06,"mean_abs_dp":2.5779008865356445e-06,"max_abs_dlogit":3.814697265625e-05,"oracle_top2_margin":0.6942056119441986,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.874301910400391e-06,"mean_question_max_abs_dp":3.874301910400391e-06,"mean_abs_dp":2.5779008865356445e-06,"conv":{"fast_calls":58968,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_c6a411e7f2bbdf377173","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":207,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.18370816699462,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005316734313964844,"ref_absmax":48.316890716552734,"rel_max_abs_diff":1.1003883393811597e-05,"min_pos_cos":0.9999999995841702,"min_pos_cos_index":103,"mean_pos_cos":0.9999999999798952},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.2169281641642252e-07,"max_abs_dlogit":2.0265579223632812e-06,"oracle_top2_margin":0.8326506838202477,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":1.2169281641642252e-07,"conv":{"fast_calls":59280,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_e332aa927d0c2ed044aa","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":208,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.175054667022778,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00037801265716552734,"ref_absmax":48.31690979003906,"rel_max_abs_diff":7.823609970260512e-06,"min_pos_cos":0.999999999744623,"min_pos_cos_index":103,"mean_pos_cos":0.9999999999814398},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":3.725290298461914e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9292190689593554,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1175870895385742e-08,"mean_question_max_abs_dp":1.1175870895385742e-08,"mean_abs_dp":3.725290298461914e-09,"conv":{"fast_calls":59592,"checked_vs_F_conv1d":408,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a0128e551d095f3449f8","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":183,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.482481582992477,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008032321929931641,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.6624259097876216e-05,"min_pos_cos":0.9999999984586321,"min_pos_cos_index":62,"mean_pos_cos":0.9999999999757232},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.470348358154297e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.928305683657527,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.470348358154297e-08,"conv":{"fast_calls":59880,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_46d6a0731bd288c370f0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":182,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.6587779999827035,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00029659271240234375,"ref_absmax":50.34111404418945,"rel_max_abs_diff":5.89165969076478e-06,"min_pos_cos":0.9999999998331592,"min_pos_cos_index":59,"mean_pos_cos":0.9999999999864456},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.319814999898274e-08,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.904314111918211,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.319814999898274e-08,"conv":{"fast_calls":60168,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_21ca7a8b970e36b8e20f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":214,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.713305499986745,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00039386749267578125,"ref_absmax":48.31690979003906,"rel_max_abs_diff":8.151752551794617e-06,"min_pos_cos":0.9999999994034421,"min_pos_cos_index":153,"mean_pos_cos":0.9999999999722984},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.635883331298828e-06,"mean_abs_dp":2.4164716402689614e-06,"max_abs_dlogit":1.7642974853515625e-05,"oracle_top2_margin":0.7225751206278801,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.635883331298828e-06,"mean_question_max_abs_dp":3.635883331298828e-06,"mean_abs_dp":2.4164716402689614e-06,"conv":{"fast_calls":60504,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4aa83af95bb69e90bc2d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":212,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.811193708999781,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005938075482845306,"ref_absmax":48.316898345947266,"rel_max_abs_diff":1.2289852383174305e-05,"min_pos_cos":0.9999999988658782,"min_pos_cos_index":193,"mean_pos_cos":0.9999999999393288},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":9.59634780883789e-06,"mean_abs_dp":6.397565205891927e-06,"max_abs_dlogit":2.47955322265625e-05,"oracle_top2_margin":0.5333060473203659,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.59634780883789e-06,"mean_question_max_abs_dp":9.59634780883789e-06,"mean_abs_dp":6.397565205891927e-06,"conv":{"fast_calls":60840,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_cdfa9eed51fafcf09b72","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":197,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2096296249947045,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00036907196044921875,"ref_absmax":48.31692123413086,"rel_max_abs_diff":7.638565351893903e-06,"min_pos_cos":0.999999998888376,"min_pos_cos_index":180,"mean_pos_cos":0.999999999976238},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.0132789611816406e-06,"mean_abs_dp":6.829698880513509e-07,"max_abs_dlogit":4.887580871582031e-06,"oracle_top2_margin":0.7747408449649811,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.0132789611816406e-06,"mean_question_max_abs_dp":1.0132789611816406e-06,"mean_abs_dp":6.829698880513509e-07,"conv":{"fast_calls":61152,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_d51b05bfb9d6d7f09331","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":203,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.199577958002919,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000545501708984375,"ref_absmax":48.316925048828125,"rel_max_abs_diff":1.1290075029259453e-05,"min_pos_cos":0.9999999993479318,"min_pos_cos_index":161,"mean_pos_cos":0.9999999999832228},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.3709068298339844e-06,"mean_abs_dp":9.002784887949625e-07,"max_abs_dlogit":1.5974044799804688e-05,"oracle_top2_margin":0.16769880056381226,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.3709068298339844e-06,"mean_question_max_abs_dp":1.3709068298339844e-06,"mean_abs_dp":9.002784887949625e-07,"conv":{"fast_calls":61464,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_fdd902cc6e65fad7cb4f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2603050409816206,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006809234619140625,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.4092860343780525e-05,"min_pos_cos":0.9999999987575995,"min_pos_cos_index":87,"mean_pos_cos":0.9999999999646343},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_abs_dp":8.617838223775228e-07,"max_abs_dlogit":1.7404556274414062e-05,"oracle_top2_margin":0.8473366126418114,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.2516975402832031e-06,"mean_question_max_abs_dp":1.2516975402832031e-06,"mean_abs_dp":8.617838223775228e-07,"conv":{"fast_calls":61776,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_42384af9ae56beb37ff5","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.232838708005147,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003522634506225586,"ref_absmax":48.31690979003906,"rel_max_abs_diff":7.290686679949483e-06,"min_pos_cos":0.9999999987575995,"min_pos_cos_index":87,"mean_pos_cos":0.9999999999736493},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":9.716798861821492e-08,"max_abs_dlogit":5.841255187988281e-06,"oracle_top2_margin":0.9411986172199249,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":9.716798861821492e-08,"conv":{"fast_calls":62088,"checked_vs_F_conv1d":432,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_cdf7f428857e1b841cb3","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":194,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":13.028520499996375,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006723403930664062,"ref_absmax":55.00690841674805,"rel_max_abs_diff":1.222283550227116e-05,"min_pos_cos":0.9999999994960943,"min_pos_cos_index":95,"mean_pos_cos":0.9999999999756235},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.9581550760194659,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_question_max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"conv":{"fast_calls":62400,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_774f7f9245f83c796d44","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":180,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.650576165993698,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000293731689453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":6.079268289794456e-06,"min_pos_cos":0.9999999998079174,"min_pos_cos_index":163,"mean_pos_cos":0.9999999999874521},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.3659397761027018e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9443456009030342,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.7939677238464355e-08,"mean_question_max_abs_dp":2.7939677238464355e-08,"mean_abs_dp":1.3659397761027018e-08,"conv":{"fast_calls":62688,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_92ec2ffa3dd46ada1335","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":204,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2299810420081485,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000827789306640625,"ref_absmax":48.31689453125,"rel_max_abs_diff":1.7132502299071275e-05,"min_pos_cos":0.9999999990452708,"min_pos_cos_index":145,"mean_pos_cos":0.9999999999691109},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.20881716410319e-08,"max_abs_dlogit":7.152557373046875e-07,"oracle_top2_margin":0.8247770480811596,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.20881716410319e-08,"conv":{"fast_calls":63000,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_6149a17bc154f9c5b4a0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":200,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.241117124998709,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008678436279296875,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.796147449257453e-05,"min_pos_cos":0.9999999991442491,"min_pos_cos_index":96,"mean_pos_cos":0.9999999999653835},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":7.152557373046875e-07,"mean_abs_dp":4.967053731282552e-07,"max_abs_dlogit":8.58306884765625e-06,"oracle_top2_margin":0.8216868452727795,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.152557373046875e-07,"mean_question_max_abs_dp":7.152557373046875e-07,"mean_abs_dp":4.967053731282552e-07,"conv":{"fast_calls":63312,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_be866e1dbec5d42d9423","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.201905584021006,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0013103485107421875,"ref_absmax":48.31690979003906,"rel_max_abs_diff":2.7119874106938994e-05,"min_pos_cos":0.9999999984848931,"min_pos_cos_index":163,"mean_pos_cos":0.9999999999754349},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.587935447692871e-08,"max_abs_dlogit":1.430511474609375e-06,"oracle_top2_margin":0.9001107849180698,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.587935447692871e-08,"conv":{"fast_calls":63624,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_97d2ed2591fc7638806c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":199,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.182920332998037,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0014514923095703125,"ref_absmax":48.3169059753418,"rel_max_abs_diff":3.0041085625620807e-05,"min_pos_cos":0.9999999990717194,"min_pos_cos_index":139,"mean_pos_cos":0.9999999999668031},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.3245811462402344e-06,"mean_abs_dp":1.539786656697591e-06,"max_abs_dlogit":1.0013580322265625e-05,"oracle_top2_margin":0.42948460578918457,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.3245811462402344e-06,"mean_question_max_abs_dp":2.3245811462402344e-06,"mean_abs_dp":1.539786656697591e-06,"conv":{"fast_calls":63936,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4533dd91267b4d815690","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":205,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.192324957984965,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000568389892578125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.1763778898433428e-05,"min_pos_cos":0.9999999995257689,"min_pos_cos_index":115,"mean_pos_cos":0.999999999976994},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.009374141693115e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9077414758503437,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.009374141693115e-08,"conv":{"fast_calls":64248,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_d880e65fef5e2993fdd0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":202,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.191721458017128,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00034809112548828125,"ref_absmax":48.31687927246094,"rel_max_abs_diff":7.204337919371419e-06,"min_pos_cos":0.9999999999042314,"min_pos_cos_index":112,"mean_pos_cos":0.9999999999855733},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.854534149169922e-07,"mean_abs_dp":4.6193599700927734e-07,"max_abs_dlogit":2.1338462829589844e-05,"oracle_top2_margin":0.5450654923915863,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.854534149169922e-07,"mean_question_max_abs_dp":6.854534149169922e-07,"mean_abs_dp":4.6193599700927734e-07,"conv":{"fast_calls":64560,"checked_vs_F_conv1d":456,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_90494c76078ffc3c717a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":202,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":13.03367266699206,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003561973571777344,"ref_absmax":48.95710754394531,"rel_max_abs_diff":7.275702651714082e-06,"min_pos_cos":0.9999999998047521,"min_pos_cos_index":185,"mean_pos_cos":0.9999999999839474},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.1548399925231934e-07,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.8971287794411182,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.384185791015625e-07,"mean_question_max_abs_dp":2.384185791015625e-07,"mean_abs_dp":1.1548399925231934e-07,"conv":{"fast_calls":64872,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_9ac5cc82c4126c91174a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.210699082992505,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0012693405151367188,"ref_absmax":48.31690979003906,"rel_max_abs_diff":2.6271144422369578e-05,"min_pos_cos":0.9999999970856026,"min_pos_cos_index":137,"mean_pos_cos":0.999999999956199},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":8.195638656616211e-08,"mean_abs_dp":4.842877388000488e-08,"max_abs_dlogit":3.337860107421875e-06,"oracle_top2_margin":0.902028501033783,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":8.195638656616211e-08,"mean_question_max_abs_dp":8.195638656616211e-08,"mean_abs_dp":4.842877388000488e-08,"conv":{"fast_calls":65184,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_386e10b1c1ce906473a4","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":205,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.222602082998492,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0007114410400390625,"ref_absmax":48.31694793701172,"rel_max_abs_diff":1.4724461507099558e-05,"min_pos_cos":0.9999999998620025,"min_pos_cos_index":188,"mean_pos_cos":0.999999999987749},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.30385160446167e-08,"mean_abs_dp":4.346172014872233e-09,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9533811770379543,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.30385160446167e-08,"mean_question_max_abs_dp":1.30385160446167e-08,"mean_abs_dp":4.346172014872233e-09,"conv":{"fast_calls":65496,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4da6aedc1139224eeff2","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":208,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.220100375008769,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009436607360839844,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.9530651694917128e-05,"min_pos_cos":0.999999999608675,"min_pos_cos_index":109,"mean_pos_cos":0.999999999980066},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.929420391718546e-08,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9697301546111703,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.929420391718546e-08,"conv":{"fast_calls":65808,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a22dab8ba073ff6cdb8d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2422912909823935,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005589723587036133,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.1568876427168571e-05,"min_pos_cos":0.999999999673407,"min_pos_cos_index":128,"mean_pos_cos":0.9999999999804102},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.9581627165898681,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_question_max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"conv":{"fast_calls":66120,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_42c54fa2ad45743e24d8","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":188,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.625554875005037,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00042438507080078125,"ref_absmax":48.316871643066406,"rel_max_abs_diff":8.78337227492421e-06,"min_pos_cos":0.9999999998547434,"min_pos_cos_index":146,"mean_pos_cos":0.999999999984837},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.67447845141093e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9540273305028677,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":6.67447845141093e-08,"conv":{"fast_calls":66408,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f3d13cf184f916677def","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":208,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.191941291996045,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0082855224609375,"ref_absmax":48.31690979003906,"rel_max_abs_diff":0.0001714828720823042,"min_pos_cos":0.9999999664146623,"min_pos_cos_index":148,"mean_pos_cos":0.999999999796517},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.682209014892578e-07,"mean_abs_dp":1.7136335372924805e-07,"max_abs_dlogit":3.0994415283203125e-06,"oracle_top2_margin":0.8051065430045128,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.682209014892578e-07,"mean_question_max_abs_dp":2.682209014892578e-07,"mean_abs_dp":1.7136335372924805e-07,"conv":{"fast_calls":66720,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_1105577da4c8dad4609d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":205,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.190892957994947,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004730224609375,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.789990492656008e-06,"min_pos_cos":0.9999999991588685,"min_pos_cos_index":81,"mean_pos_cos":0.9999999999787015},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.4156103134155273e-06,"mean_abs_dp":9.785095850626628e-07,"max_abs_dlogit":1.1444091796875e-05,"oracle_top2_margin":0.5260652750730515,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4156103134155273e-06,"mean_question_max_abs_dp":1.4156103134155273e-06,"mean_abs_dp":9.785095850626628e-07,"conv":{"fast_calls":67032,"checked_vs_F_conv1d":480,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_0ebd299739302d8b057b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.966766582976561,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000698089599609375,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.444814253732121e-05,"min_pos_cos":0.9999999989125693,"min_pos_cos_index":186,"mean_pos_cos":0.9999999999802994},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":3.129243850708008e-06,"mean_abs_dp":2.04394261042277e-06,"max_abs_dlogit":7.62939453125e-06,"oracle_top2_margin":0.15257352590560913,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.129243850708008e-06,"mean_question_max_abs_dp":3.129243850708008e-06,"mean_abs_dp":2.04394261042277e-06,"conv":{"fast_calls":67344,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_eafc22c8c40df3932a8e","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":189,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.655737791006686,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009965896606445312,"ref_absmax":49.97416687011719,"rel_max_abs_diff":1.994209654829598e-05,"min_pos_cos":0.9999999998466619,"min_pos_cos_index":172,"mean_pos_cos":0.999999999982222},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.1457672119140625e-06,"mean_abs_dp":1.4329950014750163e-06,"max_abs_dlogit":1.1444091796875e-05,"oracle_top2_margin":0.7760530337691307,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.1457672119140625e-06,"mean_question_max_abs_dp":2.1457672119140625e-06,"mean_abs_dp":1.4329950014750163e-06,"conv":{"fast_calls":67632,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_0aa1447680e4f25862bb","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":208,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.235529708006652,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00045299530029296875,"ref_absmax":54.96559143066406,"rel_max_abs_diff":8.241434113638098e-06,"min_pos_cos":0.9999999987675093,"min_pos_cos_index":193,"mean_pos_cos":0.9999999999697586},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":2.0191073417663574e-06,"mean_abs_dp":1.3485550880432129e-06,"max_abs_dlogit":2.2411346435546875e-05,"oracle_top2_margin":0.695934608578682,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.0191073417663574e-06,"mean_question_max_abs_dp":2.0191073417663574e-06,"mean_abs_dp":1.3485550880432129e-06,"conv":{"fast_calls":67944,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_980e76145c925747220c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":209,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.762268583988771,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00040435791015625,"ref_absmax":54.965579986572266,"rel_max_abs_diff":7.3565658773187146e-06,"min_pos_cos":0.9999999997920644,"min_pos_cos_index":167,"mean_pos_cos":0.9999999999882169},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":4.76837158203125e-07,"mean_abs_dp":3.203749656677246e-07,"max_abs_dlogit":5.7220458984375e-06,"oracle_top2_margin":0.6095224618911743,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.76837158203125e-07,"mean_question_max_abs_dp":4.76837158203125e-07,"mean_abs_dp":3.203749656677246e-07,"conv":{"fast_calls":68280,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4aee37a6a155747766aa","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":206,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.204032958979951,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00035858154296875,"ref_absmax":52.0065803527832,"rel_max_abs_diff":6.894926383091059e-06,"min_pos_cos":0.9999999998578114,"min_pos_cos_index":164,"mean_pos_cos":0.9999999999858183},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.940696716308594e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.8270759135484695,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.940696716308594e-08,"conv":{"fast_calls":68592,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_6db17cc22bd656d78558","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":199,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.188639416999649,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008590221405029297,"ref_absmax":57.03696060180664,"rel_max_abs_diff":1.5060797970986558e-05,"min_pos_cos":0.9999999995508757,"min_pos_cos_index":55,"mean_pos_cos":0.9999999999813016},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":9.437402089436849e-08,"max_abs_dlogit":3.6954879760742188e-06,"oracle_top2_margin":0.9279375076293945,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":9.437402089436849e-08,"conv":{"fast_calls":68904,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_ef3e0f8e657bf59f8a29","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":200,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.199069499998586,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00036984682083129883,"ref_absmax":48.73970031738281,"rel_max_abs_diff":7.5882046549924005e-06,"min_pos_cos":0.9999999998133804,"min_pos_cos_index":99,"mean_pos_cos":0.9999999999850339},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.2351741790771484e-07,"mean_abs_dp":1.2914339701334635e-07,"max_abs_dlogit":2.1696090698242188e-05,"oracle_top2_margin":0.5817935168743134,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.2351741790771484e-07,"mean_question_max_abs_dp":2.2351741790771484e-07,"mean_abs_dp":1.2914339701334635e-07,"conv":{"fast_calls":69216,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_f756e1cb04e11766341e","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":206,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.191842624975834,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008177757263183594,"ref_absmax":48.73971176147461,"rel_max_abs_diff":1.677842762633559e-05,"min_pos_cos":0.9999999986114414,"min_pos_cos_index":144,"mean_pos_cos":0.999999999976407},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.5083621342976886e-07,"max_abs_dlogit":1.0311603546142578e-05,"oracle_top2_margin":0.5247095376253128,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.5762786865234375e-07,"mean_question_max_abs_dp":3.5762786865234375e-07,"mean_abs_dp":2.5083621342976886e-07,"conv":{"fast_calls":69528,"checked_vs_F_conv1d":504,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_d167e51bb1f499099645","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":186,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.464521749992855,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005316734313964844,"ref_absmax":48.316856384277344,"rel_max_abs_diff":1.1003891212787899e-05,"min_pos_cos":0.999999999161045,"min_pos_cos_index":144,"mean_pos_cos":0.9999999999705572},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":6.9141387939453125e-06,"mean_abs_dp":4.598249991734822e-06,"max_abs_dlogit":2.300739288330078e-05,"oracle_top2_margin":0.3043373227119446,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.9141387939453125e-06,"mean_question_max_abs_dp":6.9141387939453125e-06,"mean_abs_dp":4.598249991734822e-06,"conv":{"fast_calls":69816,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_674ca4e9346216255466","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":176,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.112509124999633,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005216598510742188,"ref_absmax":48.31689453125,"rel_max_abs_diff":1.0796634513354825e-05,"min_pos_cos":0.9999999998411362,"min_pos_cos_index":76,"mean_pos_cos":0.9999999999922554},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.30385160446167e-08,"mean_abs_dp":7.450580596923828e-09,"max_abs_dlogit":5.960464477539062e-07,"oracle_top2_margin":0.9540956951677799,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.30385160446167e-08,"mean_question_max_abs_dp":1.30385160446167e-08,"mean_abs_dp":7.450580596923828e-09,"conv":{"fast_calls":70080,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_1df851a202994a07f123","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":194,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.258945582987508,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00042629241943359375,"ref_absmax":48.316917419433594,"rel_max_abs_diff":8.822839746439085e-06,"min_pos_cos":0.9999999995803316,"min_pos_cos_index":89,"mean_pos_cos":0.9999999999887462},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":5.21540641784668e-08,"mean_abs_dp":1.7384688059488933e-08,"max_abs_dlogit":5.960464477539062e-07,"oracle_top2_margin":0.8496682047843933,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.21540641784668e-08,"mean_question_max_abs_dp":5.21540641784668e-08,"mean_abs_dp":1.7384688059488933e-08,"conv":{"fast_calls":70392,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b5e6788d95b8395d2888","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":193,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.208041459001834,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00034046173095703125,"ref_absmax":48.3168830871582,"rel_max_abs_diff":7.046434066180898e-06,"min_pos_cos":0.9999999998592185,"min_pos_cos_index":178,"mean_pos_cos":0.9999999999904524},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.725290298461914e-08,"mean_abs_dp":2.2041300932566326e-08,"max_abs_dlogit":2.3245811462402344e-06,"oracle_top2_margin":0.9533842094242573,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.725290298461914e-08,"mean_question_max_abs_dp":3.725290298461914e-08,"mean_abs_dp":2.2041300932566326e-08,"conv":{"fast_calls":70704,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_32f36cd0fbbfb3604586","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":183,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.632822624989785,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003299713134765625,"ref_absmax":48.316871643066406,"rel_max_abs_diff":6.82931866769388e-06,"min_pos_cos":0.9999999997903384,"min_pos_cos_index":166,"mean_pos_cos":0.999999999986868},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":9.685754776000977e-08,"mean_abs_dp":5.960464477539063e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9057800695300102,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":9.685754776000977e-08,"mean_question_max_abs_dp":9.685754776000977e-08,"mean_abs_dp":5.960464477539063e-08,"conv":{"fast_calls":70992,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_843185706adff0c60f8b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":184,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.640128458006075,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.00064849853515625,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.3421780233349492e-05,"min_pos_cos":0.9999999997345377,"min_pos_cos_index":64,"mean_pos_cos":0.9999999999868797},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.341104507446289e-07,"mean_abs_dp":9.189049402872722e-08,"max_abs_dlogit":4.0531158447265625e-06,"oracle_top2_margin":0.6781404465436935,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.341104507446289e-07,"mean_question_max_abs_dp":1.341104507446289e-07,"mean_abs_dp":9.189049402872722e-08,"conv":{"fast_calls":71280,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b531dd603bf080dab820","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":199,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.1872876250126865,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009765625,"ref_absmax":48.3169059753418,"rel_max_abs_diff":2.021161082827576e-05,"min_pos_cos":0.9999999997238413,"min_pos_cos_index":110,"mean_pos_cos":0.9999999999850618},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.146728992462158e-08,"mean_abs_dp":2.483526865641276e-08,"max_abs_dlogit":2.4437904357910156e-06,"oracle_top2_margin":0.9287196863442659,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.146728992462158e-08,"mean_question_max_abs_dp":6.146728992462158e-08,"mean_abs_dp":2.483526865641276e-08,"conv":{"fast_calls":71592,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_be2dd67bc733c1213bef","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":202,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.189235833007842,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006030797958374023,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.2481762169212332e-05,"min_pos_cos":0.9999999995050771,"min_pos_cos_index":52,"mean_pos_cos":0.9999999999774946},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.754432201385498e-08,"max_abs_dlogit":2.2649765014648438e-06,"oracle_top2_margin":0.8430927470326424,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":8.754432201385498e-08,"conv":{"fast_calls":71904,"checked_vs_F_conv1d":528,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b8e2750307df9761e716","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":196,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":13.061728208005661,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0003027915954589844,"ref_absmax":48.31687927246094,"rel_max_abs_diff":6.266787094247741e-06,"min_pos_cos":0.9999999998776165,"min_pos_cos_index":134,"mean_pos_cos":0.9999999999880905},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":8.692344029744467e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9213596116751432,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.4901161193847656e-08,"mean_question_max_abs_dp":1.4901161193847656e-08,"mean_abs_dp":8.692344029744467e-09,"conv":{"fast_calls":72216,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_5d31b4bf2d0fc193f53b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":201,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.2727495830040425,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0008182525634765625,"ref_absmax":48.31690979003906,"rel_max_abs_diff":1.693511789210601e-05,"min_pos_cos":0.9999999988111258,"min_pos_cos_index":159,"mean_pos_cos":0.9999999999805663},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_abs_dp":2.5331974029541016e-07,"max_abs_dlogit":6.9141387939453125e-06,"oracle_top2_margin":0.8883221484720707,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":4.172325134277344e-07,"mean_question_max_abs_dp":4.172325134277344e-07,"mean_abs_dp":2.5331974029541016e-07,"conv":{"fast_calls":72528,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_a9230edfc6ec9e524af0","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":185,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.661724291974679,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0009942054748535156,"ref_absmax":48.31694793701172,"rel_max_abs_diff":2.057674413022961e-05,"min_pos_cos":0.9999999996109424,"min_pos_cos_index":79,"mean_pos_cos":0.9999999999843628},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9519330970942974,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":7.450580596923828e-09,"mean_question_max_abs_dp":7.450580596923828e-09,"mean_abs_dp":2.483526865641276e-09,"conv":{"fast_calls":72816,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_c100e84585149a04be49","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":191,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.653090582985897,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005594491958618164,"ref_absmax":48.316864013671875,"rel_max_abs_diff":1.1578756346925022e-05,"min_pos_cos":0.9999999998760283,"min_pos_cos_index":79,"mean_pos_cos":0.9999999999892816},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"max_abs_dlogit":1.9073486328125e-06,"oracle_top2_margin":0.9623699542135,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.587935447692871e-09,"mean_question_max_abs_dp":5.587935447692871e-09,"mean_abs_dp":1.862645149230957e-09,"conv":{"fast_calls":73104,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_579beed184ffc1104aa9","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":182,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.64166095899418,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005357265472412109,"ref_absmax":48.316864013671875,"rel_max_abs_diff":1.1087775628186883e-05,"min_pos_cos":0.9999999998310316,"min_pos_cos_index":167,"mean_pos_cos":0.9999999999846811},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.1288634141286216e-08,"max_abs_dlogit":4.76837158203125e-07,"oracle_top2_margin":0.9717829953879118,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.1288634141286216e-08,"conv":{"fast_calls":73392,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_c4e9bd463b6fde5be743","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":170,"padded_tokens":176,"chunks":11,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.091630416980479,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004444122314453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.197853970922782e-06,"min_pos_cos":0.9999999999086012,"min_pos_cos_index":91,"mean_pos_cos":0.9999999999882766},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":3.91155481338501e-08,"mean_abs_dp":2.0489096641540527e-08,"max_abs_dlogit":2.86102294921875e-06,"oracle_top2_margin":0.9437617193907499,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":3.91155481338501e-08,"mean_question_max_abs_dp":3.91155481338501e-08,"mean_abs_dp":2.0489096641540527e-08,"conv":{"fast_calls":73656,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b863998d4be2721a5dfd","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":205,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.197788083984051,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005707740783691406,"ref_absmax":50.43629455566406,"rel_max_abs_diff":1.1316732987575153e-05,"min_pos_cos":0.9999999995952268,"min_pos_cos_index":165,"mean_pos_cos":0.9999999999856226},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.043081283569336e-07,"mean_abs_dp":6.457169850667317e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.692500114440918,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.043081283569336e-07,"mean_question_max_abs_dp":1.043081283569336e-07,"mean_abs_dp":6.457169850667317e-08,"conv":{"fast_calls":73968,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_37cebc3792e98049439f","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":202,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.174659250013065,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005617141723632812,"ref_absmax":50.43626403808594,"rel_max_abs_diff":1.1137109043982996e-05,"min_pos_cos":0.9999999997592774,"min_pos_cos_index":133,"mean_pos_cos":0.9999999999833032},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":9.12696123123169e-08,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.9405502211302519,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":9.12696123123169e-08,"conv":{"fast_calls":74280,"checked_vs_F_conv1d":552,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_64d63f86e328aff1481c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":210,"padded_tokens":224,"chunks":14,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":13.562426333985059,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004730224609375,"ref_absmax":50.43629455566406,"rel_max_abs_diff":9.378612467564372e-06,"min_pos_cos":0.9999999998726192,"min_pos_cos_index":170,"mean_pos_cos":0.9999999999889754},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.8560558954874676e-08,"max_abs_dlogit":7.152557373046875e-07,"oracle_top2_margin":0.9369588885456324,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":5.960464477539063e-08,"mean_question_max_abs_dp":5.960464477539063e-08,"mean_abs_dp":2.8560558954874676e-08,"conv":{"fast_calls":74616,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_6d64a8459f64eac7f4cc","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":207,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.235338541999226,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.009288787841796875,"ref_absmax":50.43628692626953,"rel_max_abs_diff":0.00018416874849204747,"min_pos_cos":0.9999993875949071,"min_pos_cos_index":83,"mean_pos_cos":0.9999999970233373},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.388492425282796e-08,"max_abs_dlogit":2.384185791015625e-06,"oracle_top2_margin":0.9439202304929495,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.388492425282796e-08,"conv":{"fast_calls":74928,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_2708193212d8a4a523c7","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":207,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.203224124998087,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.001003265380859375,"ref_absmax":48.316890716552734,"rel_max_abs_diff":2.0764278619353903e-05,"min_pos_cos":0.9999999990324512,"min_pos_cos_index":117,"mean_pos_cos":0.9999999999687951},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.6093254089355469e-06,"mean_abs_dp":1.0480483373006184e-06,"max_abs_dlogit":5.245208740234375e-06,"oracle_top2_margin":0.5487232059240341,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.6093254089355469e-06,"mean_question_max_abs_dp":1.6093254089355469e-06,"mean_abs_dp":1.0480483373006184e-06,"conv":{"fast_calls":75240,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_5a37d1dfcad20810a76a","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":196,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.195050250011263,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000354766845703125,"ref_absmax":48.31687927246094,"rel_max_abs_diff":7.342503304126487e-06,"min_pos_cos":0.9999999998210595,"min_pos_cos_index":106,"mean_pos_cos":0.999999999984308},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_abs_dp":1.73225998878479e-07,"max_abs_dlogit":3.2186508178710938e-06,"oracle_top2_margin":0.8613027483224869,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.980232238769531e-07,"mean_question_max_abs_dp":2.980232238769531e-07,"mean_abs_dp":1.73225998878479e-07,"conv":{"fast_calls":75552,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_acacd8b46acd25cb954b","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":206,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.184101541002747,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0006003379821777344,"ref_absmax":48.316871643066406,"rel_max_abs_diff":1.2425017633853462e-05,"min_pos_cos":0.9999999997063979,"min_pos_cos_index":57,"mean_pos_cos":0.9999999999788958},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.594524701436361e-08,"max_abs_dlogit":9.5367431640625e-07,"oracle_top2_margin":0.8688179031014442,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":4.594524701436361e-08,"conv":{"fast_calls":75864,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_4089c199b8b796ec3ed8","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":205,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.182459125004243,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000640869140625,"ref_absmax":48.85005187988281,"rel_max_abs_diff":1.3119108700249295e-05,"min_pos_cos":0.9999999998386694,"min_pos_cos_index":186,"mean_pos_cos":0.9999999999750139},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":2.205371856689453e-06,"mean_abs_dp":1.4627973238627117e-06,"max_abs_dlogit":1.4781951904296875e-05,"oracle_top2_margin":0.7102116644382477,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":2.205371856689453e-06,"mean_question_max_abs_dp":2.205371856689453e-06,"mean_abs_dp":1.4627973238627117e-06,"conv":{"fast_calls":76176,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_02f2b95a5c5a4777a45c","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":191,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.63266141602071,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0005011558532714844,"ref_absmax":48.316864013671875,"rel_max_abs_diff":1.0372276088317237e-05,"min_pos_cos":0.9999999997502692,"min_pos_cos_index":134,"mean_pos_cos":0.9999999999884742},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":0,"argmax_oracle":0,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.543712854385376e-08,"max_abs_dlogit":3.5762786865234375e-06,"oracle_top2_margin":0.9560115858912468,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":7.543712854385376e-08,"conv":{"fast_calls":76464,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_ebea4681e3dcc7d92ca2","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":196,"padded_tokens":208,"chunks":13,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":7.207848332996946,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.000553131103515625,"ref_absmax":48.31687927246094,"rel_max_abs_diff":1.1447989022562802e-05,"min_pos_cos":0.9999999998341532,"min_pos_cos_index":179,"mean_pos_cos":0.9999999999852639},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.071462313334148e-08,"max_abs_dlogit":4.291534423828125e-06,"oracle_top2_margin":0.7813310697674751,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.7881393432617188e-07,"mean_question_max_abs_dp":1.7881393432617188e-07,"mean_abs_dp":8.071462313334148e-08,"conv":{"fast_calls":76776,"checked_vs_F_conv1d":576,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false},{"kind":"p1","id":"semif_b8c5b9b285cbdddc9f8d","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":187,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":12.470214708999265,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004023313522338867,"ref_absmax":48.31692123413086,"rel_max_abs_diff":8.32692443883783e-06,"min_pos_cos":0.999999999853492,"min_pos_cos_index":114,"mean_pos_cos":0.9999999999875249},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":1,"argmax_oracle":1,"argmax_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.805244048436483e-08,"max_abs_dlogit":2.9802322387695312e-06,"oracle_top2_margin":0.9581476468592882,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":1.1920928955078125e-07,"mean_question_max_abs_dp":1.1920928955078125e-07,"mean_abs_dp":5.805244048436483e-08,"conv":{"fast_calls":77064,"checked_vs_F_conv1d":600,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":true},{"kind":"p1","id":"semif_7ed75d854d99b3a81916","arm":"text","device":"cpu","S":16,"perturb":null,"source":"semif_authored144","tokens":185,"padded_tokens":192,"chunks":12,"n_image_tokens":0,"merged_hw":null,"n_image_max_used":1024,"seconds":6.696550666994881,"host":{"start":1073741824,"amount":0,"image_tokens":0},"head_floor":{"max_abs_dlogit":0.0,"bit_equal":true},"hidden":{"max_abs_diff":0.0004444122314453125,"ref_absmax":48.31694793701172,"rel_max_abs_diff":9.197853970922782e-06,"min_pos_cos":0.9999999992510765,"min_pos_cos_index":168,"mean_pos_cos":0.9999999999780107},"rope_planes_equal_oracle":true,"questions":[{"question_id":"decision","type":"choice","n_options":3,"argmax":2,"argmax_oracle":2,"argmax_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.315127929051717e-07,"max_abs_dlogit":7.867813110351562e-06,"oracle_top2_margin":0.7615668922662735,"near_tie":false}],"argmax_all_equal":true,"max_abs_dp":6.556510925292969e-07,"mean_question_max_abs_dp":6.556510925292969e-07,"mean_abs_dp":4.315127929051717e-07,"conv":{"fast_calls":77352,"checked_vs_F_conv1d":600,"mismatched":0,"max_abs_diff":0.0},"conv_checked_this_run":false}]},"p2":{"result":"PASS","kind":"p2","id":null,"arm":null,"device":"cpu","S":16,"runs":[{"id":"own_t01","arm":"text","tokens":436,"chunks":28,"hidden_bit_equal":true,"max_abs_diff":0.0,"chunk_layer_hashes_equal":true,"first_difference":null,"seconds":22.601297749992227,"module_run":{"npz":"<lane>/parity/own_t01__text__cpu_s16.npz","device":"cpu"}},{"id":"own_j01","arm":"text","tokens":476,"chunks":30,"hidden_bit_equal":true,"max_abs_diff":0.0,"chunk_layer_hashes_equal":true,"first_difference":null,"seconds":28.069894874992315,"module_run":{"npz":"<lane>/parity/own_j01__text__cpu_s16.npz","device":"cpu"}},{"id":"semif_a3f18f3a63d45345942b","arm":"text","tokens":185,"chunks":12,"hidden_bit_equal":true,"max_abs_diff":0.0,"chunk_layer_hashes_equal":true,"first_difference":null,"seconds":17.879269416007446,"module_run":{"npz":"<lane>/parity/semif_a3f18f3a63d45345942b__text__cpu_s16.npz","device":"cpu"}}],"load_seconds":5.6700499590078834,"unrolled_linear_layers":24,"conv":{"fast_calls":1680,"checked_vs_F_conv1d":72,"mismatched":0,"max_abs_diff":0.0},"plain_class":"coreai_models.models.macos.qwen3_5.Qwen3_5StatefulForCausalLM","plain_entry":"model.forward_stateful (embed_tokens + forward_stateful_core, in-graph plain RoPE)"},"p3":{"runs":[{"id":"own_t01","arm":"text","tokens":436,"stages":[{"stage":"layer-0 input (hs[0])","max_abs_diff":0.0,"ref_absmax":0.462890625,"min_pos_cos":0.9999999999999998,"mean_pos_cos":1.0},{"stage":"layer 0 output (hs[1])","max_abs_diff":3.4332275390625e-05,"ref_absmax":22.78901481628418,"min_pos_cos":0.9999999999996432,"mean_pos_cos":0.9999999999999802},{"stage":"layer 1 output (hs[2])","max_abs_diff":3.24249267578125e-05,"ref_absmax":31.73855209350586,"min_pos_cos":0.9999999999997372,"mean_pos_cos":0.9999999999999285},{"stage":"layer 2 output (hs[3])","max_abs_diff":5.14984130859375e-05,"ref_absmax":32.855560302734375,"min_pos_cos":0.9999999999988266,"mean_pos_cos":0.9999999999997875},{"stage":"layer 3 output (hs[4])","max_abs_diff":4.9591064453125e-05,"ref_absmax":30.969675064086914,"min_pos_cos":0.9999999999988156,"mean_pos_cos":0.9999999999997821},{"stage":"layer 4 output (hs[5])","max_abs_diff":4.57763671875e-05,"ref_absmax":28.546762466430664,"min_pos_cos":0.9999999999978126,"mean_pos_cos":0.9999999999996794},{"stage":"layer 5 output (hs[6])","max_abs_diff":3.814697265625e-05,"ref_absmax":28.99304962158203,"min_pos_cos":0.9999999999974135,"mean_pos_cos":0.9999999999995526},{"stage":"layer 6 output (hs[7])","max_abs_diff":3.814697265625e-05,"ref_absmax":63.745628356933594,"min_pos_cos":0.9999999999967402,"mean_pos_cos":0.9999999999993479},{"stage":"layer 7 output (hs[8])","max_abs_diff":3.62396240234375e-05,"ref_absmax":44.75798416137695,"min_pos_cos":0.9999999999963658,"mean_pos_cos":0.999999999999158},{"stage":"layer 8 output (hs[9])","max_abs_diff":2.6702880859375e-05,"ref_absmax":42.81202697753906,"min_pos_cos":0.9999999999960273,"mean_pos_cos":0.9999999999989657},{"stage":"layer 9 output (hs[10])","max_abs_diff":2.47955322265625e-05,"ref_absmax":43.58789825439453,"min_pos_cos":0.9999999999940886,"mean_pos_cos":0.9999999999988645},{"stage":"layer 10 output (hs[11])","max_abs_diff":2.6702880859375e-05,"ref_absmax":68.33386993408203,"min_pos_cos":0.9999999999921526,"mean_pos_cos":0.9999999999987395},{"stage":"layer 11 output (hs[12])","max_abs_diff":2.6702880859375e-05,"ref_absmax":67.96086883544922,"min_pos_cos":0.999999999986045,"mean_pos_cos":0.9999999999983245},{"stage":"layer 12 output (hs[13])","max_abs_diff":2.47955322265625e-05,"ref_absmax":67.7262191772461,"min_pos_cos":0.9999999999725822,"mean_pos_cos":0.9999999999979051},{"stage":"layer 13 output (hs[14])","max_abs_diff":3.62396240234375e-05,"ref_absmax":67.68138885498047,"min_pos_cos":0.9999999999596528,"mean_pos_cos":0.9999999999974938},{"stage":"layer 14 output (hs[15])","max_abs_diff":7.05718994140625e-05,"ref_absmax":74.43360137939453,"min_pos_cos":0.9999999999269855,"mean_pos_cos":0.9999999999969025},{"stage":"layer 15 output (hs[16])","max_abs_diff":0.0001068115234375,"ref_absmax":70.99459075927734,"min_pos_cos":0.9999999998343649,"mean_pos_cos":0.9999999999955997},{"stage":"layer 16 output (hs[17])","max_abs_diff":0.0001010894775390625,"ref_absmax":69.69348907470703,"min_pos_cos":0.999999999763129,"mean_pos_cos":0.9999999999941982},{"stage":"layer 17 output (hs[18])","max_abs_diff":0.000202178955078125,"ref_absmax":68.98184204101562,"min_pos_cos":0.9999999996825502,"mean_pos_cos":0.99999999999293},{"stage":"layer 18 output (hs[19])","max_abs_diff":9.72747802734375e-05,"ref_absmax":79.57122802734375,"min_pos_cos":0.9999999995659704,"mean_pos_cos":0.9999999999903293},{"stage":"layer 19 output (hs[20])","max_abs_diff":0.0001621246337890625,"ref_absmax":53.280818939208984,"min_pos_cos":0.9999999995877067,"mean_pos_cos":0.9999999999884926},{"stage":"layer 20 output (hs[21])","max_abs_diff":0.000213623046875,"ref_absmax":59.358970642089844,"min_pos_cos":0.9999999994859263,"mean_pos_cos":0.9999999999865072},{"stage":"layer 21 output (hs[22])","max_abs_diff":0.0002422332763671875,"ref_absmax":67.45379638671875,"min_pos_cos":0.999999999395574,"mean_pos_cos":0.999999999984558},{"stage":"layer 22 output (hs[23])","max_abs_diff":0.000537872314453125,"ref_absmax":157.20408630371094,"min_pos_cos":0.9999999990732287,"mean_pos_cos":0.9999999999804116},{"stage":"layer 23 output (hs[24])","max_abs_diff":0.0005130767822265625,"ref_absmax":120.97592163085938,"min_pos_cos":0.99999999883224,"mean_pos_cos":0.999999999978108},{"stage":"layer 24 output (hs[25])","max_abs_diff":0.00139617919921875,"ref_absmax":126.42041015625,"min_pos_cos":0.9999999986516616,"mean_pos_cos":0.9999999999758205},{"stage":"layer 25 output (hs[26])","max_abs_diff":0.00225067138671875,"ref_absmax":138.5946807861328,"min_pos_cos":0.9999999982792424,"mean_pos_cos":0.9999999999728995},{"stage":"layer 26 output (hs[27])","max_abs_diff":0.00191497802734375,"ref_absmax":294.0437927246094,"min_pos_cos":0.9999999977242038,"mean_pos_cos":0.9999999999679544},{"stage":"layer 27 output (hs[28])","max_abs_diff":0.00304412841796875,"ref_absmax":153.15187072753906,"min_pos_cos":0.9999999971634439,"mean_pos_cos":0.9999999999660923},{"stage":"layer 28 output (hs[29])","max_abs_diff":0.003292083740234375,"ref_absmax":159.9530792236328,"min_pos_cos":0.9999999972070555,"mean_pos_cos":0.9999999999652416},{"stage":"layer 29 output (hs[30])","max_abs_diff":0.00281524658203125,"ref_absmax":165.33294677734375,"min_pos_cos":0.9999999974374257,"mean_pos_cos":0.9999999999668803},{"stage":"layer 30 output (hs[31])","max_abs_diff":0.003017425537109375,"ref_absmax":162.252685546875,"min_pos_cos":0.9999999972160991,"mean_pos_cos":0.9999999999661379},{"stage":"layer 31 output (hs[32])","max_abs_diff":0.01593017578125,"ref_absmax":2268.877197265625,"min_pos_cos":0.9999999992441753,"mean_pos_cos":0.999999999988716},{"stage":"final norm (last_hidden)","max_abs_diff":0.0012989044189453125,"ref_absmax":48.316856384277344,"min_pos_cos":0.9999999983907859,"mean_pos_cos":0.9999999999827827}],"worst_max_abs_diff":{"value":0.01593017578125,"stage":"layer 31 output (hs[32])"},"worst_min_pos_cos":{"value":0.9999999971634439,"stage":"layer 27 output (hs[28])"}},{"id":"img_01","arm":"g256","tokens":430,"stages":[{"stage":"layer-0 input (hs[0])","max_abs_diff":0.0,"ref_absmax":17.857091903686523,"min_pos_cos":0.9999999999999998,"mean_pos_cos":1.0},{"stage":"layer 0 output (hs[1])","max_abs_diff":3.4332275390625e-05,"ref_absmax":32.62185287475586,"min_pos_cos":0.9999999999996432,"mean_pos_cos":0.9999999999999819},{"stage":"layer 1 output (hs[2])","max_abs_diff":3.24249267578125e-05,"ref_absmax":38.02134323120117,"min_pos_cos":0.9999999999991824,"mean_pos_cos":0.9999999999999357},{"stage":"layer 2 output (hs[3])","max_abs_diff":5.14984130859375e-05,"ref_absmax":32.855560302734375,"min_pos_cos":0.999999999998462,"mean_pos_cos":0.9999999999998237},{"stage":"layer 3 output (hs[4])","max_abs_diff":4.9591064453125e-05,"ref_absmax":30.969675064086914,"min_pos_cos":0.9999999999978404,"mean_pos_cos":0.9999999999998096},{"stage":"layer 4 output (hs[5])","max_abs_diff":4.57763671875e-05,"ref_absmax":28.936403274536133,"min_pos_cos":0.9999999999970404,"mean_pos_cos":0.9999999999997319},{"stage":"layer 5 output (hs[6])","max_abs_diff":3.814697265625e-05,"ref_absmax":28.993045806884766,"min_pos_cos":0.9999999999965055,"mean_pos_cos":0.9999999999996391},{"stage":"layer 6 output (hs[7])","max_abs_diff":3.24249267578125e-05,"ref_absmax":63.745628356933594,"min_pos_cos":0.9999999999971534,"mean_pos_cos":0.9999999999994773},{"stage":"layer 7 output (hs[8])","max_abs_diff":3.0517578125e-05,"ref_absmax":44.75798416137695,"min_pos_cos":0.9999999999962327,"mean_pos_cos":0.999999999999417},{"stage":"layer 8 output (hs[9])","max_abs_diff":2.288818359375e-05,"ref_absmax":42.81202697753906,"min_pos_cos":0.9999999999961984,"mean_pos_cos":0.9999999999992938},{"stage":"layer 9 output (hs[10])","max_abs_diff":2.09808349609375e-05,"ref_absmax":43.587894439697266,"min_pos_cos":0.9999999999962185,"mean_pos_cos":0.9999999999992721},{"stage":"layer 10 output (hs[11])","max_abs_diff":3.3855438232421875e-05,"ref_absmax":68.33386993408203,"min_pos_cos":0.9999999999959743,"mean_pos_cos":0.9999999999991946},{"stage":"layer 11 output (hs[12])","max_abs_diff":3.3855438232421875e-05,"ref_absmax":67.96086883544922,"min_pos_cos":0.999999999988349,"mean_pos_cos":0.9999999999990505},{"stage":"layer 12 output (hs[13])","max_abs_diff":3.528594970703125e-05,"ref_absmax":67.7262191772461,"min_pos_cos":0.9999999999798523,"mean_pos_cos":0.9999999999988494},{"stage":"layer 13 output (hs[14])","max_abs_diff":3.62396240234375e-05,"ref_absmax":67.68138885498047,"min_pos_cos":0.9999999999633843,"mean_pos_cos":0.9999999999985885},{"stage":"layer 14 output (hs[15])","max_abs_diff":6.198883056640625e-05,"ref_absmax":74.43360137939453,"min_pos_cos":0.9999999999327538,"mean_pos_cos":0.9999999999982189},{"stage":"layer 15 output (hs[16])","max_abs_diff":0.0001773834228515625,"ref_absmax":70.99459075927734,"min_pos_cos":0.9999999997584652,"mean_pos_cos":0.9999999999971674},{"stage":"layer 16 output (hs[17])","max_abs_diff":0.000324249267578125,"ref_absmax":69.69348907470703,"min_pos_cos":0.9999999992514425,"mean_pos_cos":0.9999999999952288},{"stage":"layer 17 output (hs[18])","max_abs_diff":0.000247955322265625,"ref_absmax":68.98184204101562,"min_pos_cos":0.9999999990649938,"mean_pos_cos":0.999999999994109},{"stage":"layer 18 output (hs[19])","max_abs_diff":0.000148773193359375,"ref_absmax":79.57122802734375,"min_pos_cos":0.9999999986314215,"mean_pos_cos":0.9999999999916819},{"stage":"layer 19 output (hs[20])","max_abs_diff":0.00020599365234375,"ref_absmax":53.413963317871094,"min_pos_cos":0.9999999989547819,"mean_pos_cos":0.9999999999911849},{"stage":"layer 20 output (hs[21])","max_abs_diff":0.000308990478515625,"ref_absmax":61.58170700073242,"min_pos_cos":0.9999999989464332,"mean_pos_cos":0.9999999999901947},{"stage":"layer 21 output (hs[22])","max_abs_diff":0.000316619873046875,"ref_absmax":67.45378875732422,"min_pos_cos":0.9999999988594699,"mean_pos_cos":0.9999999999894328},{"stage":"layer 22 output (hs[23])","max_abs_diff":0.00086212158203125,"ref_absmax":157.20413208007812,"min_pos_cos":0.9999999986082141,"mean_pos_cos":0.9999999999883356},{"stage":"layer 23 output (hs[24])","max_abs_diff":0.00051116943359375,"ref_absmax":124.89531707763672,"min_pos_cos":0.9999999989908317,"mean_pos_cos":0.9999999999904693},{"stage":"layer 24 output (hs[25])","max_abs_diff":0.000396728515625,"ref_absmax":128.6216583251953,"min_pos_cos":0.9999999990595485,"mean_pos_cos":0.9999999999906934},{"stage":"layer 25 output (hs[26])","max_abs_diff":0.0003662109375,"ref_absmax":138.59469604492188,"min_pos_cos":0.9999999990560166,"mean_pos_cos":0.9999999999906266},{"stage":"layer 26 output (hs[27])","max_abs_diff":0.0013275146484375,"ref_absmax":294.0438232421875,"min_pos_cos":0.9999999989354892,"mean_pos_cos":0.9999999999890612},{"stage":"layer 27 output (hs[28])","max_abs_diff":0.000637054443359375,"ref_absmax":163.96519470214844,"min_pos_cos":0.9999999991511438,"mean_pos_cos":0.9999999999910592},{"stage":"layer 28 output (hs[29])","max_abs_diff":0.000728607177734375,"ref_absmax":170.21034240722656,"min_pos_cos":0.9999999993796341,"mean_pos_cos":0.9999999999915395},{"stage":"layer 29 output (hs[30])","max_abs_diff":0.0007781982421875,"ref_absmax":176.71734619140625,"min_pos_cos":0.9999999994045023,"mean_pos_cos":0.9999999999915758},{"stage":"layer 30 output (hs[31])","max_abs_diff":0.000911712646484375,"ref_absmax":172.61424255371094,"min_pos_cos":0.9999999993754266,"mean_pos_cos":0.9999999999899175},{"stage":"layer 31 output (hs[32])","max_abs_diff":0.005859375,"ref_absmax":2534.841064453125,"min_pos_cos":0.9999999998592494,"mean_pos_cos":0.9999999999956098},{"stage":"final norm (last_hidden)","max_abs_diff":0.000446319580078125,"ref_absmax":48.31694793701172,"min_pos_cos":0.9999999997278849,"mean_pos_cos":0.9999999999924615}],"worst_max_abs_diff":{"value":0.005859375,"stage":"layer 31 output (hs[32])"},"worst_min_pos_cos":{"value":0.9999999986082141,"stage":"layer 22 output (hs[23])"}}]},"p4":{"zero_embeds_moves_argmax_every_run":true,"runs":[{"id":"img_07","arm":"g448","device":"cpu","arms":{"image_embeds_zero":{"argmax_changed":2,"questions":3,"max_abs_dp_vs_base":0.6637070178985596,"rope_planes_equal_oracle":true,"argmax_equal_oracle":1},"image_rc_row_col_swapped":{"argmax_changed":0,"questions":3,"max_abs_dp_vs_base":0.0615348219871521,"rope_planes_equal_oracle":false,"argmax_equal_oracle":3},"pos1d_index_all_planes":{"argmax_changed":0,"questions":3,"max_abs_dp_vs_base":0.0070198774337768555,"rope_planes_equal_oracle":false,"argmax_equal_oracle":3},"rope_shift_amount_plus_1":{"argmax_changed":0,"questions":3,"max_abs_dp_vs_base":0.001384437084197998,"rope_planes_equal_oracle":false,"argmax_equal_oracle":3}}},{"id":"img_08","arm":"g448","device":"cpu","arms":{"image_embeds_zero":{"argmax_changed":1,"questions":4,"max_abs_dp_vs_base":0.5346691906452179,"rope_planes_equal_oracle":true,"argmax_equal_oracle":3},"image_rc_row_col_swapped":{"argmax_changed":0,"questions":4,"max_abs_dp_vs_base":0.015324532985687256,"rope_planes_equal_oracle":false,"argmax_equal_oracle":4},"pos1d_index_all_planes":{"argmax_changed":0,"questions":4,"max_abs_dp_vs_base":0.0745273232460022,"rope_planes_equal_oracle":false,"argmax_equal_oracle":4},"rope_shift_amount_plus_1":{"argmax_changed":0,"questions":4,"max_abs_dp_vs_base":0.0008138418197631836,"rope_planes_equal_oracle":false,"argmax_equal_oracle":4}}},{"id":"img_01","arm":"g256","device":"cpu","arms":{"image_embeds_zero":{"argmax_changed":2,"questions":3,"max_abs_dp_vs_base":0.752471849322319,"rope_planes_equal_oracle":true,"argmax_equal_oracle":1},"image_rc_row_col_swapped":{"argmax_changed":0,"questions":3,"max_abs_dp_vs_base":0.006321609020233154,"rope_planes_equal_oracle":false,"argmax_equal_oracle":3},"pos1d_index_all_planes":{"argmax_changed":1,"questions":3,"max_abs_dp_vs_base":0.011417746543884277,"rope_planes_equal_oracle":false,"argmax_equal_oracle":2},"rope_shift_amount_plus_1":{"argmax_changed":0,"questions":3,"max_abs_dp_vs_base":0.0012584328651428223,"rope_planes_equal_oracle":false,"argmax_equal_oracle":3}}}]},"p5":{"runs":[{"id":"own_t01","arm":"text","tokens":436,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":1.2516975402832031e-06,"vs_oracle_hidden_max_abs_diff":0.0012989044189453125,"seconds":22.52041866601212,"chunks":28},"32":{"vs_oracle_max_abs_dp":1.1920928955078125e-06,"vs_oracle_hidden_max_abs_diff":0.000396728515625,"seconds":14.86291454199818,"chunks":14,"vs_s16_hidden_max_abs_diff":0.0015506744384765625,"vs_s16_max_abs_dp":2.086162567138672e-07},"64":{"vs_oracle_max_abs_dp":3.8743019104003906e-07,"vs_oracle_hidden_max_abs_diff":0.0008087158203125,"seconds":12.900609959004214,"chunks":7,"vs_s16_hidden_max_abs_diff":0.0005407333374023438,"vs_s16_max_abs_dp":8.642673492431641e-07}}},{"id":"own_t14","arm":"text","tokens":2389,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":3.5762786865234375e-07,"vs_oracle_hidden_max_abs_diff":0.0008497238159179688,"seconds":93.57261433399981,"chunks":150},"32":{"vs_oracle_max_abs_dp":2.980232238769531e-07,"vs_oracle_hidden_max_abs_diff":0.0010623931884765625,"seconds":55.91649237499223,"chunks":75,"vs_s16_hidden_max_abs_diff":0.0008792877197265625,"vs_s16_max_abs_dp":3.129243850708008e-07},"64":{"vs_oracle_max_abs_dp":6.556510925292969e-07,"vs_oracle_hidden_max_abs_diff":0.001415252685546875,"seconds":45.42656412499491,"chunks":38,"vs_s16_hidden_max_abs_diff":0.001094818115234375,"vs_s16_max_abs_dp":5.364418029785156e-07}}},{"id":"img_04","arm":"g448","tokens":690,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":3.5762786865234375e-07,"vs_oracle_hidden_max_abs_diff":0.0011761188507080078,"seconds":30.112736875016708,"chunks":44},"32":{"vs_oracle_max_abs_dp":1.7881393432617188e-07,"vs_oracle_hidden_max_abs_diff":0.000904083251953125,"seconds":20.094218917016406,"chunks":22,"vs_s16_hidden_max_abs_diff":0.0009326934814453125,"vs_s16_max_abs_dp":2.384185791015625e-07},"64":{"vs_oracle_max_abs_dp":1.7881393432617188e-07,"vs_oracle_hidden_max_abs_diff":0.0010857582092285156,"seconds":17.076649791997625,"chunks":11,"vs_s16_hidden_max_abs_diff":0.0009431838989257812,"vs_s16_max_abs_dp":2.384185791015625e-07}}},{"id":"photo_02","arm":"native","tokens":1035,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":1.1920928955078125e-07,"vs_oracle_hidden_max_abs_diff":0.00079345703125,"seconds":42.35778200000641,"chunks":65},"32":{"vs_oracle_max_abs_dp":1.7881393432617188e-07,"vs_oracle_hidden_max_abs_diff":0.0009136199951171875,"seconds":27.49469745901297,"chunks":33,"vs_s16_hidden_max_abs_diff":0.0008039474487304688,"vs_s16_max_abs_dp":1.1920928955078125e-07},"64":{"vs_oracle_max_abs_dp":2.384185791015625e-07,"vs_oracle_hidden_max_abs_diff":0.0010967254638671875,"seconds":23.27737241599243,"chunks":17,"vs_s16_hidden_max_abs_diff":0.0012025833129882812,"vs_s16_max_abs_dp":1.1920928955078125e-07}}},{"id":"semif_a3f18f3a63d45345942b","arm":"text","tokens":185,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":1.7881393432617188e-07,"vs_oracle_hidden_max_abs_diff":0.000446319580078125,"seconds":12.365620124997804,"chunks":12},"32":{"vs_oracle_max_abs_dp":4.0978193283081055e-08,"vs_oracle_hidden_max_abs_diff":0.0005083084106445312,"seconds":9.712922499980778,"chunks":6,"vs_s16_hidden_max_abs_diff":0.0006084442138671875,"vs_s16_max_abs_dp":1.7881393432617188e-07},"64":{"vs_oracle_max_abs_dp":1.1920928955078125e-07,"vs_oracle_hidden_max_abs_diff":0.000537872314453125,"seconds":8.908925249997992,"chunks":3,"vs_s16_hidden_max_abs_diff":0.00044345855712890625,"vs_s16_max_abs_dp":5.960464477539063e-08}}},{"id":"semif_f40beba9088c8db8bbd6","arm":"text","tokens":183,"device":"cpu","widths":{"16":{"vs_oracle_max_abs_dp":1.1920928955078125e-07,"vs_oracle_hidden_max_abs_diff":0.0005950927734375,"seconds":12.452694916981272,"chunks":12},"32":{"vs_oracle_max_abs_dp":2.980232238769531e-07,"vs_oracle_hidden_max_abs_diff":0.0012807846069335938,"seconds":9.75020420801593,"chunks":6,"vs_s16_hidden_max_abs_diff":0.0006856918334960938,"vs_s16_max_abs_dp":1.7881393432617188e-07},"64":{"vs_oracle_max_abs_dp":2.980232238769531e-07,"vs_oracle_hidden_max_abs_diff":0.00106048583984375,"seconds":8.859883917000843,"chunks":3,"vs_s16_hidden_max_abs_diff":0.000553131103515625,"vs_s16_max_abs_dp":1.7881393432617188e-07}}}]},"device_check":{"schema":"clef-flash-decoder-device-check/1","what":"the module in fp32, S=16 chunks, fresh zero states: CPU (bmm depthwise conv, checked against F.conv1d) vs MPS (F.conv1d), hidden at every position and the author's head on each","bar":{"max_abs_dp":1e-05,"hidden_max_abs":0.01},"runs":[{"id":"own_t01","arm":"text","tokens":436,"hidden_max_abs_diff":0.00152587890625,"hidden_min_pos_cos":0.9999999976074015,"max_abs_dp":9.5367431640625e-07,"argmax_equal":true,"cpu_vs_oracle":{"max_abs_dp":1.2516975402832031e-06,"min_pos_cos":0.9999999983907859},"mps_vs_oracle":{"max_abs_dp":8.195638656616211e-07,"min_pos_cos":0.9999999994811214},"seconds":{"cpu":22.52041866601212,"mps":11.890400667005451},"chunks":28},{"id":"own_j01","arm":"text","tokens":476,"hidden_max_abs_diff":0.0006448030471801758,"hidden_min_pos_cos":0.9999999997174034,"max_abs_dp":1.1324882507324219e-06,"argmax_equal":true,"cpu_vs_oracle":{"max_abs_dp":1.2516975402832031e-06,"min_pos_cos":0.9999999999120478},"mps_vs_oracle":{"max_abs_dp":1.0132789611816406e-06,"min_pos_cos":0.9999999997663191},"seconds":{"cpu":22.376559665979585,"mps":11.169664000015473},"chunks":30},{"id":"img_01","arm":"g256","tokens":430,"hidden_max_abs_diff":0.00045013427734375,"hidden_min_pos_cos":0.9999999992456257,"max_abs_dp":8.940696716308594e-07,"argmax_equal":true,"cpu_vs_oracle":{"max_abs_dp":8.642673492431641e-07,"min_pos_cos":0.9999999997278849},"mps_vs_oracle":{"max_abs_dp":1.1026859283447266e-06,"min_pos_cos":0.999999999712665},"seconds":{"cpu":20.93862583299051,"mps":9.78567841698532},"chunks":27},{"id":"img_04","arm":"g448","tokens":690,"hidden_max_abs_diff":0.0032806396484375,"hidden_min_pos_cos":0.9999999831004369,"max_abs_dp":2.60770320892334e-07,"argmax_equal":true,"cpu_vs_oracle":{"max_abs_dp":3.5762786865234375e-07,"min_pos_cos":0.9999999990149874},"mps_vs_oracle":{"max_abs_dp":1.7881393432617188e-07,"min_pos_cos":0.9999999902135579},"seconds":{"cpu":30.112736875016708,"mps":16.65437554201344},"chunks":44},{"id":"semif_a3f18f3a63d45345942b","arm":"text","tokens":185,"hidden_max_abs_diff":0.00043785572052001953,"hidden_min_pos_cos":0.99999999983505,"max_abs_dp":0.0,"argmax_equal":true,"cpu_vs_oracle":{"max_abs_dp":1.7881393432617188e-07,"min_pos_cos":0.999999999799893},"mps_vs_oracle":{"max_abs_dp":1.7881393432617188e-07,"min_pos_cos":0.9999999998518024},"seconds":{"cpu":12.365620124997804,"mps":4.486746665992541},"chunks":12}],"max_abs_dp":1.1324882507324219e-06,"hidden_max_abs_diff":0.0032806396484375,"generated_at":"2026-10-03T00:57:51+00:00","result":"PASS"},"generated_at":"2026-10-03T02:08:56+00:00"},"host":{"schema":"clef-flash-host-test/1","what":"host.build_ids / static_inputs / rope_positions / preprocess vs every oracle run","oracle":{"records_oracle.json":{"sha256":"d8adaa746a5d575e654212ca78f5634af334aaa62eb53ef150d5d5925b1fa7c0","runs":214,"complete":true},"records_oracle_large.json":{"sha256":"93e0c7115ced533a01a5f7ee5b10fa6267d1dc5d28ca75cf0b3cb2724f58ab42","runs":28,"complete":true}},"fixtures_records_json_sha256":"84e24684b5ce29e8da7582a23acf83c923a02fc874681c9247c735ddaaadecbc","tokenizer_json_sha256":"06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523","host_py_sha256":"356ae136ee62252a052a23bd13415228281065980754603aa89d75e225eb8e2a","runs":242,"arms":{"g256":14,"g448":14,"g672":14,"g896":14,"native":14,"text":172},"tokenizers":{"tokenizers 0.23.0-rc0 Tokenizer.from_file(tokenizer.json)":{"pass":242,"of":242,"fail":[]},"transformers 4.57.6 AutoTokenizer":{"pass":242,"of":242,"fail":[]}},"tokenizer_errors":{},"negative_control":{"run":"own_t01:text","question":"department","changed_instructions":"Which zqxvortmund should handle this ticket?","red":true,"span_changed":true,"messages":["ids differ at 125 (len 440 vs 436)","question 0 differs: ('department', 'choice', [124, 135], [[146, 162], [168, 184], [190, 207], [213, 236]], ['account', 'billing', 'shipping', 'technical']) vs ('department', 'choice', [124, 131], [[142, 158], [164, 180], [186, 203], [209, 232]], ['account', 'billing', 'shipping', 'technical'])","rope planes differ from the oracle's rotary input"],"question_span_host":[124,135],"question_span_oracle":[124,131]},"pixel_values":{"summary":{"pil":{"runs":56,"max_abs":0.0,"max_levels":0.0,"worst":"img_01:g256","runs_bit_equal":56,"runs_with_diff_gt_half_level":0},"numpy":{"runs":56,"max_abs":0.01568630337715149,"max_levels":2.000003680586815,"worst":"photo_03:g672","runs_bit_equal":31,"runs_with_diff_gt_half_level":25}},"runs":{"img_01:g256":{"image":"shapes_224.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_01:g448":{"image":"shapes_224.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"img_02:g256":{"image":"bars_640x480.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_02:g448":{"image":"bars_640x480.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.007843151688575745,"max_levels":1.0000018402934074,"n_diff":2810,"n_diff_gt_half_level":2810,"n":1204224}},"img_03:g256":{"image":"sevenseg_47_300x200.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_03:g448":{"image":"sevenseg_47_300x200.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"img_04:g256":{"image":"receipt_1024x768.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_04:g448":{"image":"receipt_1024x768.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":2,"n_diff_gt_half_level":2,"n":1204224}},"img_05:g256":{"image":"signal_green_256x240.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_05:g448":{"image":"signal_green_256x240.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"img_06:g256":{"image":"gauge_300x200.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_06:g448":{"image":"gauge_300x200.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"img_07:g256":{"image":"arrows_640x480.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_07:g448":{"image":"arrows_640x480.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":204,"n_diff_gt_half_level":204,"n":1204224}},"img_08:g256":{"image":"floorplan_1024x768.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_08:g448":{"image":"floorplan_1024x768.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"img_09:g256":{"image":"circles_640x480.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_09:g448":{"image":"circles_640x480.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.015686274506151676,"max_levels":1.9999999995343387,"n_diff":214,"n_diff_gt_half_level":214,"n":1204224}},"img_10:g256":{"image":"clock_3oclock_256x240.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216}},"img_10:g448":{"image":"clock_3oclock_256x240.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224}},"photo_01:g256":{"image":"photo_fruit.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":16,"n_diff_gt_half_level":16,"n":393216}},"photo_01:g448":{"image":"photo_fruit.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":76,"n_diff_gt_half_level":76,"n":1204224}},"photo_02:g256":{"image":"photo_cat.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.007843144237995148,"max_levels":1.0000008903443813,"n_diff":58,"n_diff_gt_half_level":58,"n":393216}},"photo_02:g448":{"image":"photo_cat.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":998,"n_diff_gt_half_level":998,"n":1204224}},"photo_03:g256":{"image":"photo_lake.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.007843137253075838,"max_levels":0.9999999997671694,"n_diff":68,"n_diff_gt_half_level":68,"n":393216}},"photo_03:g448":{"image":"photo_lake.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":3876,"n_diff_gt_half_level":3876,"n":1204224}},"photo_04:g256":{"image":"photo_coffee.png","tile":256,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":393216},"numpy":{"max_abs":0.007843144237995148,"max_levels":1.0000008903443813,"n_diff":34,"n_diff_gt_half_level":34,"n":393216}},"photo_04:g448":{"image":"photo_coffee.png","tile":448,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":1204224},"numpy":{"max_abs":0.007843137718737125,"max_levels":1.0000000591389835,"n_diff":44,"n_diff_gt_half_level":44,"n":1204224}},"img_01:g672":{"image":"shapes_224.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_01:g896":{"image":"shapes_224.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_02:g672":{"image":"bars_640x480.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_02:g896":{"image":"bars_640x480.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_03:g672":{"image":"sevenseg_47_300x200.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":6,"n_diff_gt_half_level":6,"n":2709504}},"img_03:g896":{"image":"sevenseg_47_300x200.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_04:g672":{"image":"receipt_1024x768.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0156862773001194,"max_levels":2.0000003557652235,"n_diff":558,"n_diff_gt_half_level":558,"n":2709504}},"img_04:g896":{"image":"receipt_1024x768.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_05:g672":{"image":"signal_green_256x240.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_05:g896":{"image":"signal_green_256x240.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_06:g672":{"image":"gauge_300x200.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_06:g896":{"image":"gauge_300x200.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_07:g672":{"image":"arrows_640x480.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_07:g896":{"image":"arrows_640x480.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_08:g672":{"image":"floorplan_1024x768.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.00784314051270485,"max_levels":1.0000004153698683,"n_diff":2832,"n_diff_gt_half_level":2832,"n":2709504}},"img_08:g896":{"image":"floorplan_1024x768.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_09:g672":{"image":"circles_640x480.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504}},"img_09:g896":{"image":"circles_640x480.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896}},"img_10:g672":{"image":"clock_3oclock_256x240.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":6,"n_diff_gt_half_level":6,"n":2709504}},"img_10:g896":{"image":"clock_3oclock_256x240.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":2,"n_diff_gt_half_level":2,"n":4816896}},"photo_01:g672":{"image":"photo_fruit.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":1668,"n_diff_gt_half_level":1668,"n":2709504}},"photo_01:g896":{"image":"photo_fruit.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":2044,"n_diff_gt_half_level":2044,"n":4816896}},"photo_02:g672":{"image":"photo_cat.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0156862735748291,"max_levels":1.9999998807907104,"n_diff":2982,"n_diff_gt_half_level":2982,"n":2709504}},"photo_02:g896":{"image":"photo_cat.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.007843166589736938,"max_levels":1.0000037401914597,"n_diff":216,"n_diff_gt_half_level":216,"n":4816896}},"photo_03:g672":{"image":"photo_lake.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.01568630337715149,"max_levels":2.000003680586815,"n_diff":8082,"n_diff_gt_half_level":8082,"n":2709504}},"photo_03:g896":{"image":"photo_lake.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.00784313678741455,"max_levels":0.9999999403953552,"n_diff":126,"n_diff_gt_half_level":126,"n":4816896}},"photo_04:g672":{"image":"photo_coffee.png","tile":672,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":2709504},"numpy":{"max_abs":0.0156862735748291,"max_levels":1.9999998807907104,"n_diff":4844,"n_diff_gt_half_level":4844,"n":2709504}},"photo_04:g896":{"image":"photo_coffee.png","tile":896,"pil":{"max_abs":0.0,"max_levels":0.0,"n_diff":0,"n_diff_gt_half_level":0,"n":4816896},"numpy":{"max_abs":0.007843151688575745,"max_levels":1.0000018402934074,"n_diff":168,"n_diff_gt_half_level":168,"n":4816896}}}},"pass":true,"generated_at":"2026-10-03T02:17:51+00:00"}}