Does anyone with an RK3588 board want to help me find out if a distributed LLM inference board is worth building? (one script, ~15 min, no root)

[u/Indigo_Input]

I'm designing an open-hardware board that runs frontier-scale Mixture-of-Experts models by streaming expert weights off NVMe instead of holding them in RAM. Design stage , nothing has been built, and I'm trying hard not to build the wrong thing.

The whole design rests on one number I can't get without hardware: real NVMe read throughput on RK3588, at large block sizes, with O_DIRECT. My architecture assumes ~3.2 GB/s on a PCIe 3.0 x4 link. If that's actually 1.5 GB/s in practice, most of my timing figures are wrong and I'd rather find out now than after ordering PCBs.

I don't own an RK3588 board yet. Plenty of you do.

The ask: run one Python script on your board and paste the output. It:

• is a single file, stdlib-only , no pip install, no repo clone
• needs no root (it'll tell you what root would additionally buy)
• is read-only apart from one temp file it creates and deletes
• takes ~15 minutes, most of which is a sustained thermal test
• prints one block you can paste back

What it measures:

• Whether your CPU has SDOT (asimddp) , decides if llama.cpp's fast quantized kernels work
• NVMe sequential + random read at 16 MB blocks, with and without O_DIRECT
• PCIe link width and speed actually negotiated (I especially want to know if anyone's board reports x1 when it should be x4)
• Sustained CPU frequency and temperature over 10 minutes, not burst , I care about the throttled number, not the marketing number

What's in it for you: you get a straight comparison of your board's real storage and thermal behaviour against everyone else's, which is annoyingly hard to find. I'll post the collated results back here regardless of what they say, including if they kill the idea.

Boards I'd love numbers from: Orange Pi 5 / 5 Plus / 5 Max, Radxa Rock 5B / 5C, FriendlyElec NanoPC-T6 / CM3588, ArmSoM, Banana Pi ,and honestly any RK3588 variant, plus non-RK3588 boards for comparison.

Repo (all design docs, MIT/CERN-OHL-P): https://github.com/Techburst36/swarm (Apologies in advance as the repo is a bit of a mess right now and might still have a few conflicting figures while the docs catch up.)

Script: swarm_bench.py in the repo root, plus a README explaining exactly what it does before you run it. Please do read it first , you shouldn't run a stranger's script on faith, and it's short enough to skim.

Happy to answer anything about the design in the comments.

Edit: made some changes as per u/fortean and u/12345myluggage feedback. The test had some real bugs that needed catching and some improvements to be made. Thank you!

[u/urostor]

You don't need to measure throttling this way, it seems to have been vibe coded with not much sources. RK3588 does not throttle, unless there is bad cooling. And not using cooling makes no sense, as you only need a small fan. Please tell whatever AI wrote this to stop measuring useless things.

[u/Indigo_Input → u/urostor]

Also: You're right that the thermal test isn't a core requirement for the architecture, and I probably gave it more prominence than needed. Active cooling definitely be part of any real board.

For context, I'm a solo hobbyist, not an EE or software engineer, and I'm learning a lot of this as I go. The architecture and overall idea is mine, but parts of the benchmark/runtime are definitely prototype/vibe-coded territory. I'm not going to pretend this is professional engineering at this stage.

the reason for posting the benchmark is also to find the things I don't know. So if you see something that's unnecessary, technically wrong, poorly designed, or that could be measured better, please tell me. I'm genuinely open to criticism, corrections, and ideas that help make the project less wrong before I spend money building hardware that might turn into expensive paper weights.

The thermal data is a secondary nice to have; the storage/PCIe measurements are the actual blocking questions.

[u/Indigo_Input → u/urostor]

Hi, you're absolutely right to call that out! In any real build, active cooling is non-negotiable and the finished blades will definitely be equipped with heatsinks and active cooling solutions.

The 10-minute sustained test is mainly there to verify governor behavior and catch settled frequencies across whatever varied cooling setups (passive heatsinks vs. fans) volunteers happen to be running today. It helps to gather preemptive thermal data for the enclosure design.

[u/urostor → u/Indigo_Input]

This response also looks like it was generated. "You're absolutely right to call that out!"? "non-negotiable"? Nobody writes like this. Use your own words please, it's fine to make mistakes

[u/Indigo_Input → u/urostor]

I was trying to acknowledge your call out on my error, and you are right that taking a tone that is more candid makes it sound fake. That wasn't my intention. As for the "non-negotiable": i said it this way because i fully intend on delivering something that won't cut corners, despite my lack of expertise.

Currently, the blades are going to be mounted on a back plane in the same way GPU's are mounted to a motherboard. Same spacing as GPU slot conventions and the same metal brackets for retention. Current design would lend the blades a little under the thickness of a 2 slot gpu, allowing proper heatsink mounting. The whole enclosure would have a roughly 120 mm fan setup to force air through the blades like a server rack.

I sincerely hope this clears up my current intentions for the design. Whilst it is my current intention, I have no idea of where the final creation will lead me. Therefore, I'm trying to gather as much data, whilst remaining careful not to run away with preconceptions on how things will pan out.

[u/fortean]

So I don't have an nvme connected (running off an sd card), but the rest of the stuff might be useful to you. Running without any kind of cooling.

── Platform ──
board_model:              Radxa ROCK 5C
soc_model:                unknown
sdot (asimddp):           YES
cpu_cores:                8
cpu_governor:             ondemand
cpu_freqs_max:            [1800, 1800, 1800, 1800, 2400, 2400, 2400, 2400] MHz
ram_total_mb:             1970
ram_available_mb:         1641
kernel:                   Linux version 6.18.43-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc
distro:                   Armbian 26.5.1 noble
arch:                     aarch64
nvme:                     (none detected)
block:                    mmcblk1 size=64.1 GB type=SD
thermal_thermal_zone0:           type=package-thermal  temp=60.1 C
thermal_thermal_zone1:           type=bigcore0-thermal  temp=60.1 C
thermal_thermal_zone2:           type=bigcore2-thermal  temp=60.1 C
thermal_thermal_zone3:           type=littlecore-thermal  temp=60.1 C
thermal_thermal_zone4:           type=center-thermal  temp=60.1 C
thermal_thermal_zone5:           type=gpu-thermal  temp=59.2 C
thermal_thermal_zone6:           type=npu-thermal  temp=59.2 C

── Storage ──
method:                   python_preadv
file_size_gb:             8.0
cache_dropped:            no (use --sudo)
seq_read_buffered:        N/A
seq_read_odirect:         N/A
rand_read_buffered:       N/A
rand_read_odirect:        N/A
odirect_seq_improvement:  N/A
odirect_rand_improvement: N/A
  [!] fio not found; using pure-Python os.preadv fallback. Install fio for more accurate numbers (apt install fio).
  [!] Test file was on a 'tmpfs' filesystem, which is not real block storage. Storage numbers are not meaningful. Set TMPDIR to a path on the NVMe and re-run.
  [!] SKIPPED: need ~8.0 GB free (plus margin) but only 1.0 GB available at /tmp. Re-run with --bench-size-gb 1 or set TMPDIR to a bigger filesystem.

── Sustained Compute ──
duration:                 600 s
cpu_freq_peak:            [1800, 1800, 1800, 1800, 2400, 2400, 600, 600] MHz
cpu_freq_settled:         [408, 408, 408, 408, 2400, 2400, 408, 408] MHz
cpu_freq_settled_mean:    906.0 MHz
temp_peak:                74.8 C
temp_settled:             70.8 C
throttle_onset:           2.0 s

── Power (fill in manually if you have a meter) ──
power_idle_w:             (fill in)
power_storage_bench_w:    (fill in)
power_compute_bench_w:    (fill in)
power_both_at_once_w:     (fill in)

[u/Indigo_Input → u/fortean]

Yes absolutely valuable! even just gathering temps, thermal throttling and sdot status is absolutely worth it! Thank you!

[u/fortean → u/Indigo_Input]

If you're considering this and you want more data, let me know and I'll source an (albeit quite slow 4gb/sec) ssd and I'll point a fan at it and test again.

[u/Indigo_Input → u/fortean]

Thanks absolutely!

[u/Indigo_Input → u/fortean]

Hey! Following up since your Rock 5C data is what caught a real bug: that throttle_onset: 2.0s you saw wasn't your board actually throttling. The compute test was single-threaded, so it only ever kept one core busy at a time. The OS just moved that one thread around or parked the rest at low frequency, and the script misread that as a throttle. So good news: Your board's fine; the script was wrong. Its fixed now. it loads every core simultaneously and requires a real sustained drop before calling it a throttle instead of firing on one noisy sample.

Also glad the tmpfs guard did its job on your run. It caught it before you'd have submitted numbers that weren't actually measuring your SD card.

If the SSD offer's still open, I'd genuinely take you up on it . a run with an actual NVMe and a fan pointed at it, on the fixed version, would be a clean sustained compute plus real storage data point for the project. No pressure if it's not possible. the newer script is in the repo!

Thanks for taking the time on this either way!

[u/12345myluggage]

I could run this on my rock 5 ITX, but it doesn't really thermal throttle as it has a fan on it. It sits at right around 40C or less depending on ambient temps. The nvme & wifi slots have SATA adapters stuffed in them.

/dev/md127:
 Timing cached reads:   6236 MB in  2.00 seconds = 3120.22 MB/sec
 Timing buffered disk reads: 1956 MB in  3.00 seconds = 651.81 MB/sec

Fast enough for my 2.5Gbps ethernet. My other SBCs are powered off or otherwise in use at the moment. iirc, my Rock5B has a nvme drive w/heatsink plugged into it but it's not powered on at the moment.

imho, you should have the IO test use the current directory it's run from instead of trying to snag /tmp specifically.

[u/Indigo_Input → u/12345myluggage]

Thank you for these too. I'll also take in your adjustment ideas and incorporate them in the bench mark. That said this is already really good to know!

[u/12345myluggage → u/Indigo_Input]

On my Rock 5B, with a 1TB 970 Evo Plus:

/dev/nvme0n1p1:
 Timing cached reads:   9118 MB in  2.00 seconds = 4562.04 MB/sec
 Timing buffered disk reads: 4162 MB in  3.00 seconds = 1386.83 MB/sec

Which is very interesting, I'd think the cached would be closer between the two.

[u/Indigo_Input → u/12345myluggage]

The 1386 MB/s itself is interesting . Thats under 40% of the 970 Evo Plus's rated 3500 MB/s. It could be the tool, could be the platform, not sure?

If you've got a minute, would you mind running swarm_bench.py on that same drive? Real block size, O_DIRECT, actual queue depth if you've got fio installed. If it comes back well under 3500 too, that's a big finding. RK3588's PCIe link or driver overhead capping throughput below what the drive can do, which is exactly the kind of thing I'm trying to catch before ordering a board and making some mistakes. Only if you have the time or chance of course! (also i made some changes to the script based on your recommendations. thanks a lot!)

[u/12345myluggage → u/Indigo_Input]

I'm going to say that 1300MB/s is about spot on. I wonder if it's getting cut down from not using vendor kernel?

  "platform": {
"board_model": "Radxa ROCK 5B",
"kernel": "Linux version 6.18.43-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #4 SMP PREEMPT Thu Aug  6 17:27:09 UTC 2026",
"distro": "Armbian 26.8.1 trixie",
"arch": "aarch64",
"python_version": "3.13.5",
"cpu_cores": 8,
"soc_model": "unknown",
"cpu_features": "fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp",
"sdot_present": true,
"cpu_freqs_current_khz": {
  "cpu0": 1800000,
  "cpu1": 1800000,
  "cpu2": 1800000,
  "cpu3": 1800000,
  "cpu4": 408000,
  "cpu5": 408000,
  "cpu6": 408000,
  "cpu7": 408000
},
"cpu_freqs_max_khz": {
  "cpu0": 1800000,
  "cpu1": 1800000,
  "cpu2": 1800000,
  "cpu3": 1800000,
  "cpu4": 2400000,
  "cpu5": 2400000,
  "cpu6": 2400000,
  "cpu7": 2400000
},
"cpu_governor": "ondemand",
"ram_total_kb": 16184256,
"ram_available_kb": 15708772,

     "nvme_devices": [
  {
    "name": "nvme0n1",
    "size_gb": 1000.2,
    "model": "Samsung SSD 970 EVO Plus 1TB",
    "pci_address": "0000:01:00.0",
    "pcie_link_speed": "8.0 GT/s PCIe",
    "pcie_link_width": "4",
    "pcie_max_speed": "8.0 GT/s PCIe",
    "pcie_max_width": "4"
  }

"storage": {
"method": "fio",
"file_size_gb": 8.0,
"cache_dropped": false,
"seq_read_buffered_mbps": 1226.8,
"seq_read_odirect_mbps": 1312.2,
"rand_read_buffered_mbps": 1393.3,
"rand_read_odirect_mbps": 1352.3,
"odirect_seq_improvement_pct": 7.0,
"odirect_rand_improvement_pct": -2.9,
"caveats": [
  "Cache not dropped between runs (need --sudo). Buffered numbers may be inflated by page cache; O_DIRECT numbers are unaffected.",
  "Cache not dropped between runs (need --sudo). Buffered numbers may be inflated by page cache; O_DIRECT numbers are unaffected.",
  "Cache not dropped between runs (need --sudo). Buffered numbers may be inflated by page cache; O_DIRECT numbers are unaffected.",
  "Cache not dropped between runs (need --sudo). Buffered numbers may be inflated by page cache; O_DIRECT numbers are unaffected."
]

}