# JDL Exchange Notes This directory has an independent host-side exchange plan assembler for point-to-point and single-source multicast IPU21 exchange. It does not include `JDL.hpp`, load `JDL.gp`, or run JDL setup vertices: ```bash make no_blob_exchange exchange_plan_dump ./exchange_plan_dump 512 1286 1024 ./no_blob_exchange 512 1286 1024 ``` `exchange_plan.hpp` contains the assembler. The only SDK dependency in the demo is Poplar graph construction and loading; exchange plan generation itself is ordinary C++17. ## Recovered Plan Shape For one sender and receiver, the common plan shape is: ```text sender: 0x41800003 delay(111 - XCOM_TimeToMux(senderPhysical, receiverPhysical)) send(min(count, 64) - 1, direction) sendoff(count - 65, direction) # only when count > 64 0x43a00000 0 0 0 0 0 receiver: 0x00000001 0x41800003 0x641c0000 delayxpic(count - 1, 0, 0x640) # count <= 51 delaypic(51 - count + 2 * receiverPhysicalRow) delay(count + 4) 0x43a00000 0 0 ``` The `delayxpic` remains required when `count == 1`. Its zero encoded delay still advances the exchange event and restores the neutral `0x640` mux. Omitting it leaves the receiver connected to the old source after the phase; a later repeated phase can then consume unrelated traffic and report an exchange address error. At count 53 and above, the last two exchange delays reverse order: ```text delaypic(51 + 2 * receiverPhysicalRow) delayxpic(count - 53, 0, 0x640) delay(56) ``` The SDK emits an aligned dual-issue bundle at count 52. Copying that bundle to a differently aligned plan buffer raises `TEXCPT_INVALID_INSTR`. The independent assembler emits the equivalent address-independent sequence `delaypic(50 + 2 * receiverPhysicalRow); delayxpic(0, 0, 0x640)`. The physical-row phase term was initially hidden because the first validation set mapped every receiver to physical row zero. JDL emits `50`, `70`, and `72` for one-word receives on physical rows 0, 10, and 11. Hardware validation now includes logical `21 -> 20` (physical `682 -> 680`). For counts above 64, the sender uses one 64-word `send` followed by a `sendoff` for the remainder. The recovered fields support counts 1 through 4148. Routes whose send event falls directly after `sync` omit the leading delay and may add a short trailing delay; this boundary scheduling is implemented in `exchange_plan.hpp`. ## Recovered Instruction Encodings ```text delay(n) = 0x40a00000 | (n & 0x7ffff) delaypic(a,b,c) = 0x60000000 | ((a << 19) & 0x03f80000) | ((b << 18) & 0x00040000) | (c & 0x3ffff) delayxpic(a,b,c) = 0x64000000 | ((a << 14) & 0x03ffc000) | ((b << 13) & 0x00002000) | (c & 0x1fff) send(a,b,dir) = 0x78000000 | ((a << 21) & 0x07e00000) | ((b << 3) & 0x001ffff8) | (dir & 7) sendoff(a,b,dir) = 0x70000000 | ((a << 21) & 0x07e00000) | (((a >> 6) << 14) & 0x000fc000) | ((b << 3) & 0x00003ff8) | (dir & 7) ``` `exchange_plan.hpp` also contains the translated `XCOM_TimeToMux` and east/west route calculations. Generated sender rows matched the SDK oracle for all 1471 possible senders into receiver tiles 0, 736, 1286, and 1471. ## Runtime Vertex Behavior The JDL exchange vertices branch into the plan buffer as executable code. `JDLSend`: - reads `elementSelector` - sets CSR `0xa7` to `data + elementSelector * 4` - branches to `planBuf[0]` - restores CSR `0xa7` `JDLRecv`: - reads `planBuf[0]`, which is the patch index - patches one exchange instruction using `tileSelector` through a logical to physical tile lookup - sets CSR `0xa4` to the result buffer - branches to the receiver plan code - restores CSR `0xa4` - masks the patched instruction back with `andc 0x1fff` ## Point-To-Point Limits - one sender and one receiver per plan - 32-bit word transfers - count 1..4148 - one C600/IPU21 ## Proper Multicast `assembleMulticast` emits one mode-3 source stream and one receive row per destination. No intermediate or forwarding vertex is required, including for receiver sets that fail the older same-direction `send` experiment. For physical source `s` and receivers `r[i]`: ```text t[i] = XCOM_TimeToMux(s, r[i]) start = max(0, -min(t[i])) sendDelay = start - 1 recvTime[i]= start + t[i] ``` The sender uses `send(..., direction=3)` and the normal `sendoff` split above 64 words. A `sendDelay` of -1 means the delay is omitted and the send occurs at cycle 0. Each receiver starts with `delayxpic(recvTime[i], 0, sourcePhysical)`. Its remaining count schedule is the point-to-point receive schedule, but the `delaypic` word is ORed with `0x00014000`. This flag selects multicast receive/address behavior. Multicast uses absolute tile-local SRAM addressing rather than the relative `A7`/`A4` convention used by the point-to-point vertices: - source field: `sourceByteAddress / 4`, encoded by `send` - destination field: `(destinationByteAddress - 0x50000) / 4` in the low field - CSR `A6` must remain `1` until the configured exchange has completed `NoBlobMulticastSend` and `NoBlobMulticastRecv` patch these fields from the actual Poplar-assigned pointers immediately before branching into the plan. Thus relocation means different SRAM placement on each tile, not tile-ID translation. Placement-differential SDK builds established the address fields. Moving only a destination by four words changes only its multicast receive word, for example `0x61814048 -> 0x6181404c`. Moving the source by four words increments the source word address by four. It does not change route timing. The SDK inspection executable also contained nontrivial programs on physical tiles 2, 11, and 55. Replaying those as forwarding programs caused receive address faults, while omitting them produced correct multicast. They are other co-resident exchange phases or SDK runtime work, not a requirement of this multicast packet. Per-tile ELF activity alone is not enough to assign a routine to one graph copy; the placement differential identifies the relevant source and endpoint routines. Hardware validation passed with exact data, source preservation, and guards: - arbitrary physical source/receiver rows and columns - count boundaries 3, 52, 64/65, 512, 1024, and maximum 4148 - 64 receivers in one stream at 64 words, three iterations - eight receivers spread across the device at 1024 words, three iterations - the formerly incompatible logical receivers 274 and 1286 ```bash ./proper_multicast 736 1024 0 2 100 274 700 900 1000 1286 ``` `dispatchFanOut` now selects `ProperMulticast` at a cost of one launch and one payload (`156 + count` estimated cycles). The older serialized, relay-tree, and same-direction candidate assemblers remain for comparison and fallback experiments. `tools/sdk_multicast_oracle.cpp` can vary source and destination prefixes. `tools/extract_gc_exe.sh` extracts the serialized per-physical-tile ELFs. ### Relay-Tree Fan-Out `buildRelayTree` remains a fallback experiment. Every informed tile sends to one new receiver per round, so `N` destinations require `ceil(log2(N + 1))` launches. It is slower and sends `N` payload copies. ### Serialized Fan-Out `assembleSerializedFanOut` emits back-to-back sends in one sender plan and retimes each receiver's first `delayxpic` to its actual arrival cycle. This is one compute-set execution and one source pointer setup, but costs one send per receiver. It supports different route timings and can switch east/west between sends. The fixed nine-word sender row fits up to six receivers for counts at most 64, or three receivers when each transfer also needs `sendoff`. The 12-bit receive cycle field can impose a lower count/fan-out limit depending on receiver order. Hardware tests passed for: - sender 0 to receivers 274 and 1286 at counts 3 and 65 - sender 0 to receivers 736 and 1286 with different timings - sender 1286 to receivers 736 and 1471 with different directions - two receivers at count 52 and count 1024 The test harness currently instantiates two receivers: ```bash ./one_to_many_exchange 0 1 1471 1024 4 ./one_to_many_exchange 1286 736 1471 1024 4 serialized ``` `plan_dump_many.cpp` is a reverse-engineering helper that runs JDL setup once for a chosen receiver and dumps sender rows for every other tile. The no-blob harness can consume this dump as a precomputed plan-row source: ```bash ./plan_dump_many 1286 3 16 > /tmp/plan_many_1286.txt ./no_blob_exchange 512 1286 3 /tmp/plan_many_1286.txt ``` This keeps JDL out of the exchange runtime while allowing arbitrary sender rows from an offline recovered slice. ## No-Blob Replacement Status `no_blob_codelets.cpp` and `no_blob_exchange.cpp` execute the same generated plan without `JDL.hpp` or `JDL.gp`. Recovered requirements: - exchange vertices must be `SupervisorVertex` codelets with `__attribute__((target("supervisor")))` - the JDL memory constraints are required: `region(*nonexecutableDummy) != region(*planBuf)` and `elem(*data/result) != elem(*planBuf)` - `nonexecutableDummy` must use the same interleaved vector field shape as JDL: `Input>`. Without this, Poplar can place `planBuf` in a non-executable region and the first branch into the plan raises `TEXCPT_INVALID_PC`. - inactive tiles must run the two raw nonparticipation opcodes `0x40c00000, 0x41800001` - the plan rows end in `br $m10`, so the replacement vertices must explicitly load `$m10` with a local resume label before branching into `planBuf` - logical tile IDs are patched into receiver code as physical tile IDs. On this C600/IPU21, the full mapping is: ```c++ pair = logical / 2; lane = logical & 1; block = pair / 23; row = pair % 23; if (block & 1) row = 22 - row; col = (block / 2) * 4 + (block & 1); physical = row * 64 + col + lane * 2; ``` Validated on the local C600 for the basic no-blob path: ```bash g++ no_blob_exchange.cpp -std=c++17 -lpoplar -Wall -Wextra -o no_blob_exchange ./no_blob_exchange 0 ``` The test passes for sender tile 0, receiver tile 1286, count 3. Hardware integrity validation writes deterministic nonuniform data, repeats each transfer with shifted source offsets, checks result guards, and rereads the entire source. Tested cases include: - count boundaries 51, 52, 53, 64, and 65 - counts 256, 1024, 4096, and the maximum 4148 - routes crossing both directions, including immediate-send schedules near physical tile 32 - all checks return exact values and preserve source data and both result guards `plan_dump_counts.cpp`, `plan_dump_many.cpp`, and `exchange_plan_verify.cpp` remain as oracle/reverse-engineering tools. They are not needed by the independent assembler or runtime. ## One-Codelet Matrix Multiply `multi_tile_matmul.cpp` computes a signed integer 4x4 matrix product across 24 active logical tiles: - tiles 0..3 own the four A rows - tiles 4..7 own the four B columns - tiles 8..23 each compute one output element Eight multicast streams distribute four words each. The streams are generated with `assembleMulticast(..., scheduleOffset)` at 256-cycle intervals, so all eight plans coexist in one exchange program without overlapping. Every one of the 1,472 tiles runs an instance of the same `MultiTileMatMul` supervisor vertex in one compute set; inactive roles emit the recovered nonparticipation instructions. Each output instance receives one A row and one B column, then performs its local four-term dot product before returning. The graph contains no `program::Copy` and no cross-tile vertex field for matrix data. Poplar sees one vertex type and one compute-set execution; matrix exchange is entirely the plan table generated by this repository. Hardware output is: ```text C = 4 24 9 14 20 48 25 38 -12 14 3 0 22 10 8 31 vertexTypes=1 instances=1472 computeSets=1 customPasses=8 sdkMatrixCopies=0 PASS ``` The executable can be compiled once and run without graph construction: ```bash ./multi_tile_matmul --save /tmp/custom-matmul.popef ./run_matmul_executable /tmp/custom-matmul.popef ``` See `BYPASS_NOTES.md` for the remaining runtime layers. ## All-Tile Parallel Sum `parallel_sum.cpp` initializes logical tile `t` with `t + 1`, then performs an 11-round binary reduction across all 1,472 tiles. Each round assigns custom sender, receiver, or inactive supervisor roles, and every pair plan comes from `exchange_plan.hpp`. Poplar constructs and loads the graph, but its exchange assembler is not used for the reduction. All 736 first-round transfers pass in one simultaneous exchange launch with no cycle staggering. The complete hardware result is: ```text tiles=1472 rounds=11 sum=1084128 expected=1084128 PASS ``` `XCOM_LAUNCH_WIDTH` can conservatively split a round into smaller launches for scheduler experiments. The default is one simultaneous launch per round.