2026-09-06T16:49:04.507804Z  INFO retained operator-plan finalist finalist=0 values=11 operations=7 estimated_cycles=450230 estimated_exchange_cycles=173640 exchange_row_bytes=35532 peak_standard_bytes=76236 peak_interleaved_bytes=114688 peak_total_bytes=166604 maximum_standard_allocation_bytes=16384 active_tile_counts={12, 64} padding_group_counts={} conversion_sources=[]
2026-09-06T16:49:04.507944Z  INFO retained operator-plan finalist finalist=1 values=11 operations=7 estimated_cycles=460868 estimated_exchange_cycles=191640 exchange_row_bytes=38796 peak_standard_bytes=79500 peak_interleaved_bytes=53248 peak_total_bytes=132748 maximum_standard_allocation_bytes=16384 active_tile_counts={12, 30, 64} padding_group_counts={} conversion_sources=[]
2026-09-06T16:49:04.507968Z  INFO retained operator-plan finalist finalist=2 values=11 operations=7 estimated_cycles=462949 estimated_exchange_cycles=187440 exchange_row_bytes=38244 peak_standard_bytes=87396 peak_interleaved_bytes=53248 peak_total_bytes=132196 maximum_standard_allocation_bytes=16384 active_tile_counts={12, 32, 64} padding_group_counts={} conversion_sources=[]
2026-09-06T16:49:04.508033Z  INFO retained operator-plan finalist finalist=3 values=12 operations=8 estimated_cycles=463488 estimated_exchange_cycles=187600 exchange_row_bytes=38280 peak_standard_bytes=78984 peak_interleaved_bytes=85824 peak_total_bytes=154504 maximum_standard_allocation_bytes=16384 active_tile_counts={12, 30, 60, 64} padding_group_counts={} conversion_sources=[Some(OperationId(1))]
2026-09-06T16:49:04.514147Z  INFO ipu_codegen.package.phase{phase="lower_mid"}: package build phase finished phase="lower_mid" elapsed_ms=434 success=true
2026-09-06T16:49:04.996913Z  INFO screened tile mappings by exchange resource load candidates=15 baseline_cycles=63344 candidate_cycles=63344 baseline_pressure=61298 candidate_pressure=61298
2026-09-06T16:49:05.000736Z  INFO modelled expanded operator plan finalist=0 modelled_cycles=333088 estimated_cycles=354824 estimated_exchange_cycles=94080
2026-09-06T16:49:05.000872Z  INFO screened tile mappings by exchange resource load candidates=19 baseline_cycles=69296 candidate_cycles=69296 baseline_pressure=64224 candidate_pressure=64224
2026-09-06T16:49:05.004780Z  INFO modelled expanded operator plan finalist=2 modelled_cycles=394772 estimated_cycles=455388 estimated_exchange_cycles=140712
2026-09-06T16:49:05.060506Z  INFO screened tile mappings by exchange resource load candidates=23 baseline_cycles=71376 candidate_cycles=71376 baseline_pressure=67290 candidate_pressure=67290
2026-09-06T16:49:05.066286Z  INFO modelled expanded operator plan finalist=1 modelled_cycles=358102 estimated_cycles=419166 estimated_exchange_cycles=145040
2026-09-06T16:49:05.080936Z  INFO screened tile mappings by exchange resource load candidates=23 baseline_cycles=68572 candidate_cycles=68572 baseline_pressure=64804 candidate_pressure=64804
2026-09-06T16:49:05.084864Z  INFO modelled expanded operator plan finalist=3 modelled_cycles=339784 estimated_cycles=386444 estimated_exchange_cycles=125432
2026-09-06T16:49:05.171703Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=1 transfers=1038 maximum_identical_destinations=1
2026-09-06T16:49:05.171742Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=1 initial_horizon=1121 selected_horizon=1121 endpoint_lower_bound=736 lower_bound_gap=385 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T16:49:05.172704Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=1 tile=49 row_words=162 horizon=1121
2026-09-06T16:49:05.329248Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: scheduled operator-plan finalist finalist=0 mapped=false analytical_cycles=354824 analytical_exchange_cycles=94080 scheduled_exchange_cycles=74373 refined_cycles=335117
2026-09-06T16:49:05.339597Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: selected physically scheduled operator-plan finalist selected=0 mapped=false
2026-09-06T16:49:05.339638Z  INFO ipu_codegen.package.phase{phase="select_finalist"}: package build phase finished phase="select_finalist" elapsed_ms=825 success=true
2026-09-06T16:49:05.339650Z  INFO lowered graph for package construction logical_shards=5300 exchange_phases=15
2026-09-06T16:49:05.339895Z  INFO ipu_codegen.package.phase{phase="compile_runtime"}: package build phase finished phase="compile_runtime" elapsed_ms=0 success=true
2026-09-06T16:49:05.340517Z  INFO ipu_codegen.package.phase{phase="plan_kernels"}: package build phase finished phase="plan_kernels" elapsed_ms=0 success=true
2026-09-06T16:49:05.341118Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/attention_softmax_f16.S name="attention_softmax_d72_p64_tail" target=ipu21
2026-09-06T16:49:05.410414Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/6f1dcf9e5a40ba122715fc8fac5e48f8389fc3c474ada70cb241ad2424fd19a1/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/6f1dcf9e5a40ba122715fc8fac5e48f8389fc3c474ada70cb241ad2424fd19a1/kernel.json
2026-09-06T16:49:05.410773Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/attention_softmax_f16.S name="attention_softmax_d72_p64_full" target=ipu21
2026-09-06T16:49:05.465468Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/9116b185a352cd7962d9014b82beb75ac70ef7219e513e8bc6bf3791fbc4cbe0/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/9116b185a352cd7962d9014b82beb75ac70ef7219e513e8bc6bf3791fbc4cbe0/kernel.json
2026-09-06T16:49:05.465756Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/attention_merge_f16.S name="attention_merge_v72_p80_k64" target=ipu21
2026-09-06T16:49:05.520422Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/e886be24a3adfbfda0d1b88ddca8c1b9cd7a388aeb06ebe1ebab9252fb1fc7e9/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/e886be24a3adfbfda0d1b88ddca8c1b9cd7a388aeb06ebe1ebab9252fb1fc7e9/kernel.json
2026-09-06T16:49:05.520515Z  INFO ipu_codegen.package.phase{phase="compile_kernels"}: package build phase finished phase="compile_kernels" elapsed_ms=179 success=true
2026-09-06T16:49:05.520964Z  INFO ipu_codegen.package.phase{phase="link_runtime"}: package build phase finished phase="link_runtime" elapsed_ms=0 success=true
2026-09-06T16:49:05.543810Z  INFO ipu_codegen.package.phase{phase="size_tile_code"}: package build phase finished phase="size_tile_code" elapsed_ms=4 success=true
2026-09-06T16:49:05.543843Z  INFO allocated package support memory linked_end=320900 profile_bytes=0 exchange_table_bytes=3852 host_code_bytes=1076 generated_code_bytes=2148 code_address=321984 standard_ranges=[(393216, 524288)]
2026-09-06T16:49:05.551680Z  INFO ipu_codegen.package.phase{phase="place_storage"}: package build phase finished phase="place_storage" elapsed_ms=7 success=true
2026-09-06T16:49:05.558475Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=0
2026-09-06T16:49:05.589328Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=1
2026-09-06T16:49:05.589885Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=1 transfers=1038 maximum_identical_destinations=1
2026-09-06T16:49:05.589895Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=1 initial_horizon=1121 selected_horizon=1121 endpoint_lower_bound=736 lower_bound_gap=385 selected_kind="reused" neighborhood_improvements=0
2026-09-06T16:49:05.590894Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=1 tile=49 row_words=162 horizon=1121
2026-09-06T16:49:05.613940Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=2
2026-09-06T16:49:05.621180Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=3
2026-09-06T16:49:05.626692Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=4
2026-09-06T16:49:05.632151Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=5
2026-09-06T16:49:05.637632Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=6
2026-09-06T16:49:05.642966Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=7
2026-09-06T16:49:05.648085Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=8
2026-09-06T16:49:05.653540Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=9
2026-09-06T16:49:05.659003Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=10
2026-09-06T16:49:05.664199Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=11
2026-09-06T16:49:05.669788Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=12
2026-09-06T16:49:05.675245Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=13
2026-09-06T16:49:05.680567Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=14
2026-09-06T16:49:05.681282Z  INFO ipu_codegen.package.phase{phase="lower_exchanges"}: package build phase finished phase="lower_exchanges" elapsed_ms=129 success=true
2026-09-06T16:49:05.712262Z  INFO ipu_codegen.package.phase{phase="optimize_exchange_placement"}: package build phase finished phase="optimize_exchange_placement" elapsed_ms=30 success=true
2026-09-06T16:49:05.713394Z  INFO costed final placed program final_cycles=335117 final_exchange=74373
2026-09-06T16:49:05.727662Z  INFO ipu_codegen.package.phase{phase="prepare_tile_code"}: package build phase finished phase="prepare_tile_code" elapsed_ms=4 success=true
2026-09-06T16:49:05.734787Z  INFO ipu_codegen.package.phase{phase="emit_tile_code"}: package build phase finished phase="emit_tile_code" elapsed_ms=7 success=true
2026-09-06T16:49:05.863785Z  INFO ipu_codegen.package.phase{phase="build_tile_images"}: package build phase finished phase="build_tile_images" elapsed_ms=126 success=true
2026-09-06T16:49:05.874796Z  INFO writing application package tiles=1472
2026-09-06T16:49:05.888526Z  INFO application package written
2026-09-06T16:49:05.889369Z  INFO reading application package
2026-09-06T16:49:05.906772Z  INFO application package read tiles=1472 compiler=0.1.0
2026-09-06T16:49:05.907059Z  INFO waiting for hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T16:49:05.907069Z  INFO acquired hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T16:49:05.907075Z  INFO opening IPU device path="/dev/ipu0"
2026-09-06T16:49:05.907112Z  INFO IPU device attached path=/dev/ipu0
2026-09-06T16:49:05.907115Z  INFO initializing IPU device
2026-09-06T16:49:05.915067Z  INFO IPU device initialized
2026-09-06T16:49:05.915115Z  INFO replaying device configuration register_writes=2003
2026-09-06T16:49:05.919881Z  INFO loading application tile_count=1472 final_mark=23
2026-09-06T16:49:06.119474Z  INFO application loaded and initial synchronization completed tile_count=1472
2026-09-06T16:49:06.318936Z  INFO starting host exchange session startup_mark=23
2026-09-06T16:49:06.318968Z  INFO attaching host exchange pages pages=2
2026-09-06T16:49:06.319695Z  INFO host exchange session started
2026-09-06T16:49:06.319705Z  INFO invoking streaming host exchange call call="initialize" command=0 phases=24
2026-09-06T16:49:06.323878Z  INFO invoking streaming host exchange call call="run" command=0 phases=5
checkpoint=0 operation=0 tensors=1 samples=256 maximumAbsoluteError=0.000244
checkpoint=1 operation=1 tensors=1 samples=256 maximumAbsoluteError=0.000244
checkpoint=2 operation=2 tensors=1 samples=256 maximumAbsoluteError=0.000244
2026-09-06T16:49:16.764438Z ERROR diagnostic run failed error=device timeout: host call run phase 4/5: device timeout: HSP register 0x3030: expected 0, observed 1 completed=3 device=counts=[0, 64, 1408, 0] firstUnexpected=[(0, 1), (1, 2), (2, 1), (3, 2), (4, 2), (5, 2), (6, 2), (7, 2), (8, 2), (9, 2), (10, 2), (11, 2), (12, 2), (13, 2), (14, 2), (15, 2)] contexts=[(0, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 23)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (2, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (3, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (4, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (5, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (6, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74"))))]), (1, 2, Ok(322580), Some("generated tile program+0x254"), Some((321984, 624, 588, [1098907649, 413233152, 1098911745, 421527553, 422969520])), Some((156, 0, [])), None, Ok([321336, 327680, 0, 393392, 321336, 0, 327688, 393224, 0, 322208, 322580, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (2, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 23)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (2, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (3, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (4, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (5, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74")))), (6, Ok((318756, Some("ipu_stack_attention_softmax_d72_p64_tail_f16+0x74"))))]), (3, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (4, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (5, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (6, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (7, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (8, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (9, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (10, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (11, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (12, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (13, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (14, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), []), (15, 2, Ok(322380), Some("generated tile program+0x18c"), Some((321984, 420, 388, [1098907649, 413233152, 1098911745, 421527553, 422897032])), Some((105, 0, [])), None, Ok([321984, 4, 0, 320904, 949216, 327680, 0, 4294967295, 0, 322008, 322380, 362208, 0, 0, 0, 0]), Ok((0, 23)), Ok(None), Ok([0, 0, 0, 0, 0, 0]), [])]
Error: device timeout: host call run phase 4/5: device timeout: HSP register 0x3030: expected 0, observed 1