2026-09-06T21:09:18.781356Z  INFO retained operator-plan finalist finalist=0 values=16 operations=1 estimated_cycles=968964 estimated_exchange_cycles=539760 exchange_row_bytes=39696 peak_standard_bytes=362704 peak_interleaved_bytes=79360 peak_total_bytes=392464 maximum_standard_allocation_bytes=114688 active_tile_counts={1472} padding_group_counts={} conversion_sources=[]
2026-09-06T21:09:19.411920Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_mid"}: package build phase finished phase="lower_mid" elapsed_ms=17385 success=true
2026-09-06T21:09:36.212330Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: screened tile mappings by exchange resource load candidates=52 baseline_cycles=211872 candidate_cycles=211872 baseline_pressure=175464 candidate_pressure=175464
2026-09-06T21:09:36.446745Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: modelled expanded operator plan finalist=0 modelled_cycles=603162 estimated_cycles=838314 estimated_exchange_cycles=456024
2026-09-06T21:09:37.757964Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T21:09:37.758027Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=1 initial_horizon=9994 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="critical-neighborhood" neighborhood_improvements=1
2026-09-06T21:09:37.787738Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T21:09:40.428575Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: compared ordinary and paired exchange schedules phase=2 candidates=4297 ordinary_horizon=29091 paired_horizon=28544 use_paired=true
2026-09-06T21:09:40.504940Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T21:09:40.504999Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=2 initial_horizon=28544 selected_horizon=28544 endpoint_lower_bound=24400 lower_bound_gap=4144 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T21:09:40.543843Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=2 tile=646 row_words=235 horizon=28544
2026-09-06T21:10:05.964140Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T21:10:05.964184Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T21:10:06.086784Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T21:10:49.046736Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T21:10:49.046797Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T21:10:49.264732Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T21:10:49.327386Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: scheduled operator-plan finalist finalist=0 mapped=false analytical_cycles=838314 analytical_exchange_cycles=456024 scheduled_exchange_cycles=261738 refined_cycles=644028
2026-09-06T21:10:49.346544Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: selected physically scheduled operator-plan finalist selected=0 mapped=false
2026-09-06T21:10:49.346563Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="select_finalist"}: package build phase finished phase="select_finalist" elapsed_ms=89934 success=true
2026-09-06T21:10:49.346574Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: lowered graph for package construction logical_shards=34600 exchange_phases=5
2026-09-06T21:10:49.350389Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_runtime"}: package build phase finished phase="compile_runtime" elapsed_ms=3 success=true
2026-09-06T21:10:49.352097Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="plan_kernels"}: package build phase finished phase="plan_kernels" elapsed_ms=1 success=true
2026-09-06T21:10:49.352762Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: compiling kernel source=/srv/home/gc-sdk/ipu-stack/crates/ipu-tests/../../device/gelu_f16.S name="gelu_f16" target=ipu21
2026-09-06T21:10:49.404939Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: kernel artifact created object=/srv/home/.cache/ipu-stack/kernels/feafe9e2c70537dd203eb9a69f74e565f79683df07a32ff1b97876af4a64a369/kernel.o metadata=/srv/home/.cache/ipu-stack/kernels/feafe9e2c70537dd203eb9a69f74e565f79683df07a32ff1b97876af4a64a369/kernel.json
2026-09-06T21:10:49.405142Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="compile_kernels"}: package build phase finished phase="compile_kernels" elapsed_ms=53 success=true
2026-09-06T21:10:49.406940Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="link_runtime"}: package build phase finished phase="link_runtime" elapsed_ms=0 success=true
2026-09-06T21:10:50.678106Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="size_tile_code"}: package build phase finished phase="size_tile_code" elapsed_ms=13 success=true
2026-09-06T21:10:50.678174Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: allocated package support memory linked_end=319916 profile_bytes=24 exchange_table_bytes=8584 host_code_bytes=7352 generated_code_bytes=2080 code_address=409600 standard_ranges=[(425984, 524288)]
2026-09-06T21:10:51.777723Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="place_storage"}: package build phase finished phase="place_storage" elapsed_ms=1099 success=true
2026-09-06T21:10:51.786324Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=0
2026-09-06T21:10:53.159332Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T21:10:53.159386Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=1 initial_horizon=9994 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="critical-neighborhood" neighborhood_improvements=1
2026-09-06T21:10:53.188528Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T21:10:55.646283Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: compared ordinary and paired exchange schedules phase=2 candidates=4127 ordinary_horizon=29091 paired_horizon=28723 use_paired=true
2026-09-06T21:10:55.716111Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T21:10:55.716170Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=2 initial_horizon=28723 selected_horizon=28723 endpoint_lower_bound=24640 lower_bound_gap=4083 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T21:10:55.765560Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=2 tile=924 row_words=234 horizon=28723
2026-09-06T21:11:11.493921Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=3
2026-09-06T21:11:11.573997Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T21:11:11.574029Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="reused" neighborhood_improvements=0
2026-09-06T21:11:11.764560Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T21:12:06.375843Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: reused exchange optimization after validating relocated rows phase=4
2026-09-06T21:12:06.507976Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T21:12:06.508011Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="reused" neighborhood_improvements=0
2026-09-06T21:12:06.713301Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T21:12:06.762117Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="lower_exchanges"}: package build phase finished phase="lower_exchanges" elapsed_ms=74984 success=true
2026-09-06T21:12:15.855948Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=0
2026-09-06T21:12:16.109162Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=1
2026-09-06T21:12:16.123496Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=1 transfers=32280 maximum_identical_destinations=1
2026-09-06T21:12:16.123517Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=1 initial_horizon=9600 selected_horizon=9600 endpoint_lower_bound=7776 lower_bound_gap=1824 selected_kind="reused" neighborhood_improvements=0
2026-09-06T21:12:16.152258Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=1 tile=705 row_words=140 horizon=9600
2026-09-06T21:12:19.132433Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: compared ordinary and paired exchange schedules phase=2 candidates=4127 ordinary_horizon=28429 paired_horizon=27256 use_paired=true
2026-09-06T21:12:19.200103Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=2 transfers=5814 maximum_identical_destinations=1
2026-09-06T21:12:19.200132Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=2 initial_horizon=27256 selected_horizon=27256 endpoint_lower_bound=24640 lower_bound_gap=2616 selected_kind="full-duplex" neighborhood_improvements=0
2026-09-06T21:12:19.240279Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=2 tile=932 row_words=238 horizon=27256
2026-09-06T21:12:35.053114Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=3
2026-09-06T21:12:35.125154Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=3 transfers=155880 maximum_identical_destinations=1
2026-09-06T21:12:35.125194Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=3 initial_horizon=15700 selected_horizon=15700 endpoint_lower_bound=8640 lower_bound_gap=7060 selected_kind="reused" neighborhood_improvements=0
2026-09-06T21:12:35.315835Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=3 tile=447 row_words=633 horizon=15700
2026-09-06T21:13:30.053230Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: reused exchange optimization after validating relocated rows phase=4
2026-09-06T21:13:30.193950Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: prepared large physical exchange phase phase=4 transfers=5904 maximum_identical_destinations=1
2026-09-06T21:13:30.193983Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: optimized physical exchange schedule phase=4 initial_horizon=26732 selected_horizon=26732 endpoint_lower_bound=26352 lower_bound_gap=380 selected_kind="reused" neighborhood_improvements=0
2026-09-06T21:13:30.402795Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: finished large physical exchange phase phase=4 tile=431 row_words=1320 horizon=26732
2026-09-06T21:13:30.496780Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: evaluated exchange placement finalist offset=28672 baseline_score=127530434560 score=100812718080 baseline_cycles=253275 candidate_cycles=248874 row_bytes=8560 row_capacity=16320 accepted=true
2026-09-06T21:13:30.529814Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="optimize_exchange_placement"}: package build phase finished phase="optimize_exchange_placement" elapsed_ms=83767 success=true
2026-09-06T21:13:30.542543Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}: costed final placed program final_cycles=640164 final_exchange=257874
2026-09-06T21:13:31.268382Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="prepare_tile_code"}: package build phase finished phase="prepare_tile_code" elapsed_ms=39 success=true
2026-09-06T21:13:31.285847Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="emit_tile_code"}: package build phase finished phase="emit_tile_code" elapsed_ms=17 success=true
2026-09-06T21:13:31.419776Z  INFO ipu_codegen.package.build{tile_count=1472 operations=1}:ipu_codegen.package.phase{phase="build_tile_images"}: package build phase finished phase="build_tile_images" elapsed_ms=128 success=true
2026-09-06T21:13:31.563878Z  INFO writing application package tiles=1472
2026-09-06T21:13:31.592830Z  INFO application package written
2026-09-06T21:13:31.592867Z  INFO reading application package
2026-09-06T21:13:31.662383Z  INFO application package read tiles=1472 compiler=0.1.0
2026-09-06T21:13:31.663015Z  INFO waiting for hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T21:13:31.663030Z  INFO acquired hardware ownership path=artifacts/layout-sweep/device.lock
2026-09-06T21:13:31.663035Z  INFO opening IPU device path="/dev/ipu0"
2026-09-06T21:13:31.663064Z  INFO IPU device attached path=/dev/ipu0
2026-09-06T21:13:31.663069Z  INFO initializing IPU device
2026-09-06T21:13:31.671554Z  INFO IPU device initialized
2026-09-06T21:13:31.671595Z  INFO replaying device configuration register_writes=2003
2026-09-06T21:13:31.686169Z  INFO loading application tile_count=1472 final_mark=23
2026-09-06T21:13:31.917619Z  INFO application loaded and initial synchronization completed tile_count=1472
2026-09-06T21:13:36.523601Z  INFO starting host exchange session startup_mark=23
2026-09-06T21:13:36.523630Z  INFO attaching host exchange pages pages=2
2026-09-06T21:13:36.524215Z  INFO host exchange session started
2026-09-06T21:13:36.524523Z  INFO invoking streaming host exchange call call="initialize" command=0 phases=30
2026-09-06T21:13:36.548147Z  INFO invoking streaming host exchange call call="run" command=0 phases=109
Error: numerical comparison failed for 1344212/77262336 logical shard values (maximum absolute error 0.625): [(18, -2.0253906, -1.8935547, 0.13183594), (46, 0.13500977, 0.015625, 0.119384766), (1154, 2.2871094, 2.09375, 0.19335938), (1180, -0.44213867, -0.37890625, 0.06323242), (1201, -0.6723633, -0.8046875, 0.13232422), (1232, -1.6142578, -1.40625, 0.20800781), (2345, 1.8417969, 1.6796875, 0.16210938), (2348, -0.30371094, -0.515625, 0.21191406), (2401, 2.5800781, 2.765625, 0.18554688), (2416, -0.7451172, -0.6015625, 0.14355469), (2422, 0.50097656, 0.3984375, 0.10253906), (3558, 1.7402344, 1.8769531, 0.13671875), (5778, 0.008483887, 0.140625, 0.13214111), (5831, 1.9931641, 2.1445313, 0.15136719), (5835, 1.1689453, 1.3994141, 0.23046875), (5866, -0.66064453, -0.7578125, 0.09716797)]