# IPU21 Exchange Reverse Engineering

This repository contains a small, independent assembler and runtime for
point-to-point and proper single-packet multicast exchange on a Graphcore
C600/IPU21.
Plan generation does not depend on JIT Dynamic Lookup (JDL), `JDL.hpp`, or the
JDL codelet blob.

The implementation was recovered from public Graphcore artifacts and verified
against a physical C600. It is exploratory software, not an official Graphcore
interface.

## Build

Activate a Poplar SDK environment, then run:

```bash
make
```

`exchange_plan_dump` only needs a C++17 compiler. `no_blob_exchange` also needs
Poplar and compiles `no_blob_codelets.cpp` for the IPU when its graph is built.

## Run

```bash
./exchange_plan_dump 512 1286 1024
./no_blob_exchange 512 1286 1024
./proper_multicast 736 1024 0 2 100 274 700 900 1000 1286
./multi_tile_matmul
./multi_tile_matmul --save /tmp/custom-matmul.popef
./run_matmul_executable /tmp/custom-matmul.popef
./one_to_many_exchange 1286 736 1471 1024 4 serialized
./parallel_sum
./tools/test_host_exchange.sh
./tools/test_direct_d2h_redirect.sh
./popef_to_ipu_image /tmp/custom-matmul.popef /tmp/custom-matmul.ipuimg 24
./ipu_image verify /tmp/custom-matmul.ipuimg
./direct_ipu_loader inspect /tmp/custom-matmul.ipuimg
./ipu_bootloader_frames /tmp/custom-matmul.ipuimg 1408 64 /tmp/batch.frames
./direct_ipu_loader load /tmp/custom-matmul.ipuimg \
  "$POPLAR_SDK_ENABLED/bin/ipu/tile_bootloader_ipu2.elf" \
  artifacts/c600-init.ipucfg

# Build a result-producing direct-entry supervisor image.
./build_baremetal /tmp/bare-result.popef result
./popef_to_ipu_image /tmp/bare-result.popef \
  /tmp/bare-result-runtime.ipuimg 24
./ipu_image redirect /tmp/bare-result-runtime.ipuimg \
  /tmp/bare-result-direct.ipuimg 0x19689abc 0x1969abcd
./direct_ipu_loader load-bare-result /tmp/bare-result-direct.ipuimg \
  "$POPLAR_SDK_ENABLED/bin/ipu/tile_bootloader_ipu2.elf" \
  artifacts/c600-init.ipucfg 23 0 0x52000 6912

# Repeatable two-image hardware acceptance test (6912, then 3333 at 0x52000).
./tools/test_direct_result.sh

# Non-Poplar H2D, tile copy, and D2H through a manifest-defined protocol.
./tools/test_direct_host_tensor.sh
```

The runtime test sends deterministic nonuniform data, repeats transfers at
shifted source offsets, checks both result guards, and verifies that the source
was not modified.

See [EXCHANGE_NOTES.md](EXCHANGE_NOTES.md) for recovered encodings, routing,
plan layout, validation results, and current limits. [BYPASS_NOTES.md](BYPASS_NOTES.md)
describes the precompiled runner, direct kernel-driver probe, and remaining
work below Poplar. [RESEARCH_STATUS.md](RESEARCH_STATUS.md) is the concise
checkpoint separating hardware-proven results from current experiments.

## Layout

- `exchange_plan.hpp`: independent plan assembler.
- `no_blob_codelets.cpp`: supervisor send, receive, and nonparticipation codelets.
- `no_blob_exchange.cpp`: Poplar runtime and integrity test.
- `proper_multicast.cpp`: arbitrary-receiver single-packet multicast test.
- `multi_tile_matmul.cpp`: one-codelet 4x4 distributed matrix multiply.
- `run_matmul_executable.cpp`: graph-compiler-free POPEF runner.
- `parallel_sum.cpp`: 11-round reduction over all 1,472 tiles using custom plans.
- `ipu_image.hpp`: independent compact physical-tile executable format reader.
- `direct_ipu.hpp`: direct kernel attach and configuration-BAR access.
- `host_exchange.hpp`: manifest parser and direct host-exchange runtime.
- `host_exchange_plan.hpp`: independent IPU21 host packet-header assembler;
  see [HOST_EXCHANGE_NOTES.md](HOST_EXCHANGE_NOTES.md).
- `ipu_bootloader_frames.hpp`: independent 1 KiB secondary-loader framing.
- `tools/direct_ipu_loader.cpp`: Graphcore-library-free reset, load, and start.
- `tools/test_host_exchange.sh`: fresh host-data upload, custom exchange, and
  readback checks at small and large transfer sizes.
- `tools/sdk_host_tensor_oracle.cpp`: minimal `writeTensor`/run/`readTensor`
  executable used to recover the host-tensor command protocol.
- `tools/test_direct_host_tensor.sh`: randomized direct host-exchange test.
- `tools/test_host_exchange_codegen.sh`: hardware validation of an independently
  encoded host-to-tile request redirected to a different SRAM allocation.
- `baremetal_codelets.cpp`: direct-entry acceptance and H2D diagnostic probes.
- `build_baremetal.cpp`: build-time linker wrapper for direct-entry images.
- `load_executable.cpp`: supported-runtime load-only oracle for executables
  whose test vertex does not terminate.
- `tools/popef_to_ipu_image.cpp`: build-time POPEF-to-IPUIMG converter.
- `artifacts/c600-init.ipucfg`: captured C600/IPU21 initialization writes.
- `tools/`: oracle dumpers and comparison utilities used during recovery.
- `reference/`: original JDL public artifacts and an extracted ELF retained as
  reverse-engineering evidence; none are used by the independent path.

## Status

Point-to-point and proper multicast support counts 1 through 4148 on one
C600/IPU21. Proper multicast sends the payload once and arms any number of
receivers without forwarding vertices. Dispatch selects it by default;
serialized and relay-tree plans remain available as fallback experiments.
The direct loader installs and starts custom supervisor code on all 1,472 tiles
of the C600 using only the Linux kernel ABI and MMIO. The converter reproduces
the SDK's IPU21 packing convention: the omitted `.secondaryLower` reservation
is not transmitted, while the first enumerated module byte retains its linked
address (`0x4c010` for the acceptance image). The loader sends the SDK's
otherwise undocumented 24th all-`0xff`
sentinel transaction, final mark-23 execution signal, and startup mark.

The `BareResult` acceptance image computes `1234 + 5678` on each tile, stores
the result in tile SRAM, and spins. The direct loader stops tile 0 through TDI
and reads back `6912`; an alternate image subsequently writes and returns
`3333` at the same address, excluding stale SRAM. All 1,472 supervisor
contexts report active. Complete first and final 64-tile transport buffers
also compare byte-for-byte equal to buffers generated by the supported runtime.

The direct loader also runs manifest-defined host calls without Poplar. Each
operation is supplied as a `CALL INPUT OUTPUT` triple, with `-` for a missing
input or output:

```bash
./direct_ipu_loader host-exchange application.ipuimg tile_bootloader_ipu2.elf \
  artifacts/c600-init.ipucfg artifacts/sdk_host_tensor.ipuhsp \
  source-write input.bin - program-0 - - destination-read - output.bin
```

The manifest records arbitrary page sizes, attach order, command location,
callable IDs, phase counts, and non-contiguous input/output slices. The C600
test uses two randomized inputs and checks each returned file byte-for-byte.
Static extraction of a manifest from Poplar's serialized host-sync metadata is
the remaining build-time step; none of its values are hardcoded in the loader.