# Host Exchange Packet Notes

`host_exchange_plan.hpp` independently encodes the two-word packet headers
used by IPU21 host tensor copies. It has no Poplar dependency. For example:

```bash
make host_exchange_plan_dump
./host_exchange_plan_dump h2d 0 0x50120 0x40 64
# 0xec000209 0x00000011
```

## Packet Fields

Host offsets and lengths have two encodings:

```text
short:  hostAddressLength = (hostOffset / 4) << 4 | bytes / 4
        bytes = 4..60, both values aligned to 4 bytes
long:   hostAddressLength = (hostOffset / 64) << 4 | lengthCode
        bytes = 64..1024, both values aligned to 64 bytes
        lengthCode = bytes / 64, except 1024 bytes is encoded as zero
```

For physical tile `p`, routing contributes `(p >> 1) << 16` and
`(p & 1) << 15` to word 0, and `(p & 1) << 31` to word 1.

```text
tile-to-host word 0: short 0x80000000, long 0xa0000000, plus route
host-to-tile word 0: short 0xcc000200, long 0xec000200, plus route and X address
word 1:              route plus hostAddressLength
X address:           (tileAddress - 0x50000) / 32, a 9-bit field
```

A zero-byte read with the host-to-tile short opcode closes a tile-to-host
sequence. Its X address names an aligned dummy receive region and word 1 has
only routing bits.

## Device Schedule

For a 64-byte SDK host-to-tile copy, the destination tile owns the schedule
and it has this semantic shape:

```text
select the destination tile-pair host endpoint
sync the stream's start ID
set CSR A6 to 16                 # receive downcount in 32-bit words
send two-word host-to-tile request header
sync receive
sync the stream's end ID
```

The SDK schedule does not emit the direct runtime's `[1, 0]` XREQ. That packet
is part of the independently recovered one-shot protocol, not the native SDK
stream protocol. SDK packet dumps for logical tile 1471 identify tile 1471 as
the owner of both H2D and D2H. Disassembly gives the IPU21 host endpoint mux
selector as `0x600 + (physicalTile & ~2)`: physical tiles separated only by bit
1 share an endpoint.

Host synchronization is a graph, not an undifferentiated sequence of mark-1
writes. For the write/read oracle, the compiler assigns program-command syncs
`0 -> 1`, D2H syncs `2 -> 3 -> 0`, and H2D syncs `4 -> 5 -> 0`. The SDK runtime
selects those transitions through its host-interaction metadata. A direct
runtime must either install an equivalent transition table or define a complete
independent packet protocol; repeatedly releasing GS2 without that state only
executes the first stream pair reliably.

Fresh extraction of `sdk_host_tensor_oracle` gives a second important boundary.
The archive names tile members by physical tile. Logical tile 0 contains the
complete three-command dispatcher. Generic nonparticipants execute only this
loop:

```text
sans 255
sync 1
branch to sans
```

For a tensor on logical tile 100, the specialized target is `t_260.elf` because
logical tile 100 maps to physical tile 260. That member contains the D2H
downcount, command-header send, payload send, close delay, zero-byte-read send,
and sync-receive sequence inline. It also contains the H2D downcount, request
header send, and sync-receive sequence. Inspecting `t_100.elf` as though archive
names were logical tile IDs led to the earlier, incorrect conclusion that this
code was hidden resident state.

The target command handler selects `0x600 + (physicalTile & 0x3d)`, enters the
operation through sync 15, and restores mux `0x640` after sync 7. The operation
itself ends at sync 0. Controller activation and intermediate host-hierarchy
roles remain to be generated independently.

Printing the schedules for all specialized members of the logical-tile-100
oracle narrows those roles further. Logical tile 92 (physical tile 4) sends the
XREQ for both the H2D and D2H data commands, while logical tile 100 owns the
request or payload operation. Logical tiles 1, 93, and 101 have no semantic
send schedule; their specialized code is synchronization support. Tile 0 owns
the separate command read from the host.

Larger copies use multiple sends in the schedule. A tile-to-host copy sends a
command header, sends payload directly from tile SRAM in chunks of at most 256
bytes, then emits the zero-byte read close. These sends use the same `send` and
`sendoff` instruction encodings documented in `EXCHANGE_NOTES.md`.

The installed public `Program.capnp` schema names the corresponding lowered
operations: `SetDowncount`, `Send`, `MultiSend`, `RecvData`, `SyncRecv`,
`SyncInternal`, address patches, and `ExitSequence`. It also identifies packet
types including `xreq`, `zeroByteRead`, `commandPacketWrite`,
`commandPacketRead`, and `streamCopy`.

## Validation And Boundary

Before issuing a call with output slices, the direct runtime fills those host
page ranges with `0xa5`. A successful byte-for-byte round trip therefore
requires the device-to-host phase to overwrite the transport page; data left
behind by the preceding host-to-device call cannot satisfy the check.

After an SDK round trip on logical tile 200, a directly loaded saved image for
logical tile 100 still passed a fresh poisoned-output round trip. This excludes
source selection left over from the immediately preceding SDK graph.

`tools/test_host_exchange_codegen.sh` changes only one independently generated
header in a synthetic executable. It redirects 64 random host bytes to an
adjacent SRAM allocation, runs a vertex that copies only that allocation to a
second tile, and reads it back through the direct non-Poplar runtime. The C600
result compares byte-for-byte.

`tools/test_direct_d2h_redirect.sh` performs the complementary D2H check on
logical tile 100. It locates the physical-tile-260 target operation from its
instruction structure, decodes its payload source, and redirects that send to
an adjacent initialized region. After a separate random H2D call and host-page
poisoning, direct D2H returns the adjacent pattern byte-for-byte. This verifies
the recovered D2H send address field on arbitrary-tile hardware. The enclosing
controller and host-hierarchy routines in this test still come from the SDK
oracle image.

This establishes the request/header format and its use by device exchange
instructions. The current assembler does not yet construct an entire
multi-tile host phase from scratch: XREQ ownership, phase-wide ordering, and
the complete address-patch schedule are still supplied by the synthetic
fixture. Runtime loading, synchronization, and payload transport are direct.