2026-09-06T20:52:33.063324Z  INFO retained operator-plan finalist finalist=0 values=16 operations=1 estimated_cycles=958938 estimated_exchange_cycles=539760 exchange_row_bytes=39696 peak_standard_bytes=362704 peak_interleaved_bytes=79360 peak_total_bytes=392464 maximum_standard_allocation_bytes=114688 active_tile_counts={1472} padding_group_counts={} conversion_sources=[]
2026-09-06T20:52:33.630793Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_mid"}: package build phase finished phase="lower_mid" elapsed_ms=16778 success=true
2026-09-06T20:52:50.681722Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: screened tile mappings by exchange resource load candidates=52 baseline_cycles=211872 candidate_cycles=211872 baseline_pressure=175464 candidate_pressure=175464
2026-09-06T20:52:50.914181Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: modelled expanded operator plan finalist=0 modelled_cycles=593136 estimated_cycles=828288 estimated_exchange_cycles=456024
2026-09-06T20:52:52.176002Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T20:52:52.176066Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=1 initial_horizon=9994 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="critical-neighborhood" neighborhood_improvements=1
2026-09-06T20:52:52.205727Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T20:52:54.779450Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: compared ordinary and paired exchange schedules phase=2 candidates=4297 ordinary_horizon=29091 paired_horizon=28544 use_paired=true
2026-09-06T20:52:54.856458Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T20:52:54.856510Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=2 initial_horizon=28544 selected_horizon=28544 endpoint_lower_bound=24400 lower_bound_gap=4144 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T20:52:54.895628Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=2 tile=646 row_words=235 horizon=28544
2026-09-06T20:53:36.609438Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T20:53:36.609524Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T20:53:36.809491Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T20:54:26.199331Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T20:54:26.199374Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T20:54:26.330098Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T20:54:26.372439Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: scheduled operator-plan finalist finalist=0 mapped=false analytical_cycles=828288 analytical_exchange_cycles=456024 scheduled_exchange_cycles=261738 refined_cycles=634002
2026-09-06T20:54:26.385358Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: selected physically scheduled operator-plan finalist selected=0 mapped=false
2026-09-06T20:54:26.385371Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: package build phase finished phase="select_finalist" elapsed_ms=112754 success=true
2026-09-06T20:54:26.385382Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: lowered graph for package construction logical_shards=34600 exchange_phases=5
2026-09-06T20:54:26.385874Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_runtime"}: package build phase finished phase="compile_runtime" elapsed_ms=0 success=true
2026-09-06T20:54:26.387373Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="plan_kernels"}: package build phase finished phase="plan_kernels" elapsed_ms=1 success=true
2026-09-06T20:54:26.388391Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gemm_f16_amp.S name="gemm_f16_k144_c32_r366_r366" target=ipu21
2026-09-06T20:54:26.441014Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/6993ea37713ff6d2f3085dad9b15c38da3e497ce882a3f938a313c1c53fad65b/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/6993ea37713ff6d2f3085dad9b15c38da3e497ce882a3f938a313c1c53fad65b/kernel.json
2026-09-06T20:54:26.443375Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gemm_f16_amp.S name="gemm_f16_k144_c48_r366_r366" target=ipu21
2026-09-06T20:54:26.490722Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/243929b6ffbfb18c0a1bacdaf920e5aa7bd3964218f90c9f21215e63341e2588/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/243929b6ffbfb18c0a1bacdaf920e5aa7bd3964218f90c9f21215e63341e2588/kernel.json
2026-09-06T20:54:26.491139Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gemm_f16_amp.S name="gemm_f16_interleaved_k144_c32_r364_r364" target=ipu21
2026-09-06T20:54:26.535904Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/2cbf15a3d8be7897ad04a1783d2803686300804b44712798536b8a0e5f14fd99/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/2cbf15a3d8be7897ad04a1783d2803686300804b44712798536b8a0e5f14fd99/kernel.json
2026-09-06T20:54:26.536410Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gemm_f16_amp.S name="gemm_f16_interleaved_k144_c48_r364_r364" target=ipu21
2026-09-06T20:54:26.581254Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/03ff50470b979de68faf39c2928b6505b4ea07688b473f6f6540147d064efb11/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/03ff50470b979de68faf39c2928b6505b4ea07688b473f6f6540147d064efb11/kernel.json
2026-09-06T20:54:26.581604Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gemm_f16_amp.S name="gemm_f16_interleaved_k160_c64_r242_r244" target=ipu21
2026-09-06T20:54:26.627914Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/e4f6ed87a10ee61123dcfd23e04e42d3d08a74a7371e8bdcb3581035fa780bfd/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/e4f6ed87a10ee61123dcfd23e04e42d3d08a74a7371e8bdcb3581035fa780bfd/kernel.json
2026-09-06T20:54:26.628795Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: package build phase finished phase="compile_kernels" elapsed_ms=241 success=true
2026-09-06T20:54:26.630520Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="link_runtime"}: package build phase finished phase="link_runtime" elapsed_ms=0 success=true
2026-09-06T20:54:27.463927Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="size_tile_code"}: package build phase finished phase="size_tile_code" elapsed_ms=13 success=true
2026-09-06T20:54:27.464008Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: allocated package support memory linked_end=319620 profile_bytes=24 exchange_table_bytes=8584 host_code_bytes=7352 generated_code_bytes=2080 code_address=409600 standard_ranges=[(425984, 524288)]
2026-09-06T20:54:28.778574Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="place_storage"}: package build phase finished phase="place_storage" elapsed_ms=1314 success=true
2026-09-06T20:54:28.782355Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=0
2026-09-06T20:54:29.642623Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T20:54:29.642658Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=1 initial_horizon=9994 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="critical-neighborhood" neighborhood_improvements=1
2026-09-06T20:54:29.659941Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T20:54:31.249862Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: compared ordinary and paired exchange schedules phase=2 candidates=4127 ordinary_horizon=29091 paired_horizon=28723 use_paired=true
2026-09-06T20:54:31.291302Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T20:54:31.291325Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=2 initial_horizon=28723 selected_horizon=28723 endpoint_lower_bound=24640 lower_bound_gap=4083 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T20:54:31.314830Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=2 tile=924 row_words=234 horizon=28723
2026-09-06T20:54:40.600090Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=3
2026-09-06T20:54:40.642540Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T20:54:40.642560Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="reused" neighborhood_improvements=0
2026-09-06T20:54:40.753125Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T20:55:13.513900Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=4
2026-09-06T20:55:13.606002Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T20:55:13.606026Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="reused" neighborhood_improvements=0
2026-09-06T20:55:13.742153Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T20:55:13.780500Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: package build phase finished phase="lower_exchanges" elapsed_ms=45001 success=true
2026-09-06T20:55:24.098690Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=0
2026-09-06T20:55:24.261878Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=1
2026-09-06T20:55:24.270737Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T20:55:24.270753Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=1 initial_horizon=9600 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="reused" neighborhood_improvements=0
2026-09-06T20:55:24.288090Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T20:55:26.392818Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: compared ordinary and paired exchange schedules phase=2 candidates=4127 ordinary_horizon=28429 paired_horizon=27256 use_paired=true
2026-09-06T20:55:26.458320Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T20:55:26.458349Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=2 initial_horizon=27256 selected_horizon=27256 endpoint_lower_bound=24640 lower_bound_gap=2616 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T20:55:26.496760Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=2 tile=932 row_words=238 horizon=27256
2026-09-06T20:55:42.158703Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=3
2026-09-06T20:55:42.228702Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T20:55:42.228732Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="reused" neighborhood_improvements=0
2026-09-06T20:55:42.420390Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T20:56:36.476695Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=4
2026-09-06T20:56:36.609121Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T20:56:36.609150Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="reused" neighborhood_improvements=0
2026-09-06T20:56:36.812280Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T20:56:36.882230Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: evaluated exchange placement finalist offset=28672 baseline_score=127530434560 score=100812718080 baseline_cycles=253275 candidate_cycles=248874 row_bytes=8560 row_capacity=16320 accepted=true
2026-09-06T20:56:36.909285Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: package build phase finished phase="optimize_exchange_placement" elapsed_ms=83128 success=true
2026-09-06T20:56:36.919296Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: costed final placed program final_cycles=630138 final_exchange=257874
2026-09-06T20:56:37.626486Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="prepare_tile_code"}: package build phase finished phase="prepare_tile_code" elapsed_ms=38 success=true
2026-09-06T20:56:37.642720Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="emit_tile_code"}: package build phase finished phase="emit_tile_code" elapsed_ms=16 success=true
2026-09-06T20:56:37.770017Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="build_tile_images"}: package build phase finished phase="build_tile_images" elapsed_ms=121 success=true
2026-09-06T20:56:37.895993Z  INFO writing application package tiles=1472
2026-09-06T20:56:37.923357Z  INFO application package written
2026-09-06T20:56:37.923386Z  INFO reading application package
2026-09-06T20:56:37.991970Z  INFO application package read tiles=1472 compiler=0.1.0
2026-09-06T20:56:37.992725Z  INFO waiting for hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T20:56:37.992739Z  INFO acquired hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T20:56:37.992745Z  INFO opening IPU device path="/dev/ipu0"
2026-09-06T20:56:37.992776Z  INFO IPU device attached path=/dev/ipu0
2026-09-06T20:56:37.992781Z  INFO initializing IPU device
2026-09-06T20:56:38.000939Z  INFO IPU device initialized
2026-09-06T20:56:38.000949Z  INFO replaying device configuration register_writes=2003
2026-09-06T20:56:38.013861Z  INFO loading application tile_count=1472 final_mark=23
2026-09-06T20:56:38.239652Z  INFO application loaded and initial synchronization completed tile_count=1472
2026-09-06T20:56:42.689628Z  INFO starting host exchange session startup_mark=23
2026-09-06T20:56:42.689671Z  INFO attaching host exchange pages pages=2
2026-09-06T20:56:42.690390Z  INFO host exchange session started
2026-09-06T20:56:42.690600Z  INFO invoking streaming host exchange call call="initialize" command=0 phases=30
2026-09-06T20:56:42.713161Z  INFO invoking streaming host exchange call call="run" command=0 phases=109
runFailureDiagnostics=counts=[0, 1472, 0, 0] firstUnexpected=[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1), (13, 1), (14, 1), (15, 1)] contexts=[(0, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (2, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (3, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (4, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (5, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (1, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (2, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c"))))]), (2, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (4, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (5, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (6, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c"))))]), (3, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc"))))]), (4, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (2, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (3, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (4, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (5, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (5, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (3, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (4, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (5, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (6, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (2, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (3, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (4, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (7, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (4, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (5, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (6, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc"))))]), (8, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (2, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (3, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (4, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (9, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (3, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (4, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (5, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (6, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c"))))]), (10, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (5, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (6, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94"))))]), (11, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (5, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (6, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94"))))]), (12, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (2, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (3, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (4, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (5, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (13, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (2, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104"))))]), (14, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 21)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (2, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (3, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (4, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (5, Ok((319004, Some("ipu_stack_gelu_tanh_approx_f16+0x13c")))), (6, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94"))))]), (15, 1, Err(Timeout("stopping tile context")), None, None, None, None, Err(Timeout("stopping tile context")), Ok((0, 19)), Ok(None), Ok([3, 3, 3, 3, 3, 3]), [(1, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (2, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (3, Ok((318892, Some("ipu_stack_gelu_tanh_approx_f16+0xcc")))), (4, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94")))), (5, Ok((318948, Some("ipu_stack_gelu_tanh_approx_f16+0x104")))), (6, Ok((318836, Some("ipu_stack_gelu_tanh_approx_f16+0x94"))))])]
Error: device timeout: host call run phase 52/109: device timeout: HSP register 0x3030: expected 0, observed 1