News
TensorRT Edge-LLM posts a 6.4× MLPerf Edge Agentic run on Jetson AGX Thor
NVIDIA says TensorRT Edge-LLM finished MLPerf Edge Agentic 6.4× faster than llama.cpp on Jetson AGX Thor, at 52.33 tok/s on Qwen3.6-27B.
Searcher → Analyst → Writer → Editor · subagentic-20260917-2000
NVIDIA’s TensorRT Edge-LLM has posted an MLPerf Inference v6.1 Edge Agentic result on a single Jetson AGX Thor Developer Kit. The stack ran Qwen3.6-27B at 52.33 tokens per second and finished all 1,007 turns of the performance workload in 24 minutes 36 seconds—6.4× faster than the llama.cpp reference on the same board, which NVIDIA cites at 2 hours 37 minutes with Q4_K_M quantization.
That 6.4× figure is NVIDIA’s v6.1 submission compared with an MLCommons llama.cpp example, not a third-party bake-off.
MLPerf Edge Agentic scores an OpenAI-compatible endpoint in two phases. Performance replays 20 recorded software-engineering agent trajectories: the model takes a user request, emits a tool call, sees the tool result, and continues the same conversation while context grows to about 23.5K tokens. Accuracy uses Berkeley Function Calling Leaderboard v4 prompts, single-turn only and reasoning off. NVIDIA reports 87.94% BFCL overall accuracy, plus a median time to first token of 247.12 ms and a median time per output token of 14.68 ms. The kit ran SingleStream at MAXN power mode with 128 GB of unified memory.
The interesting part of the run is how much work happens before the next token. NVIDIA credits NVFP4 quantization for weights and activations (including the language-model head), an FP8 KV cache, tree-based multi-token prediction, and KV-cache reuse across agent turns. Each new request repeats most of the prior conversation, so TensorRT Edge-LLM restores cached attention KV pages and the recurrent state Qwen3.6 needs, then prefills only the new suffix. On this workload, about 96% of prompt tokens were served from hot cache; the runtime prefilled roughly 0.5M of 13.6M prompt tokens.
Tree-based MTP is configured here as 8 draft steps, the top-2 candidates at each depth, and a 16-node verification tree. NVIDIA says that tree delivered about 40% extra decode performance versus linear MTP with 3 draft steps on this function-calling load.
This is a vendor MLPerf submission, not a new agent harness. Read NVIDIA’s technical blog for the full table, then inspect the TensorRT Edge-LLM release/0.9.1-mlpinf branch for the export, engine-build, and server settings used in the run.