
trnrt
Open LLM inference engine for AWS Trainium
AIOpen sourceOpen source · verified#llm-inference#aws-trainium#open-source
Listed by the ProductPerch team from Hacker News · not launched here yet
What is trnrt?
Trnrt is an open source inference engine built for running large language models on AWS Trainium hardware. It serves models through an OpenAI-compatible API while optimizing performance using specialized NKI megakernels. Each layer half uses a single fused kernel for decoding, which improves throughput and reduces latency.
The engine supports a context window of up to one million tokens. It achieves this through sparse attention and linear attention mechanisms, combined with a fixed-size state. A prefix cache stores key-value pages in FP8 format and moves evicted data to host memory. Two separate engines can run per instance, handling tensor parallelism across 32 logical NeuronCores behind a session-affine router.
Graph compilation happens offline on any x86_64 host before deployment. The serving instance loads precompiled graphs without performing runtime compilation. Currently, GLM-5.3-Flash is the only supported model, though additional open models are planned. Setup requires Docker, a trn2.48xlarge instance for serving, and a separate host for building the compile cache.
trnrt features
- NKI megakernels for efficient decode operations
- Supports up to 1 million token context window
- Prefix cache using radix tree and host memory
- OpenAI compatible API for chat and completions
- Offline graph compilation on CPU hosts
- Tensor parallel execution across 32 NeuronCores
Who uses trnrt and for what
- Deploying large language models on AWS Trainium hardware
- Running long-context applications requiring 1M token support
- Integrating custom LLMs with existing OpenAI client libraries
- Optimizing inference costs through fused kernel execution
- Testing model performance with offline compilation workflows
trnrt FAQ
Is trnrt free?
Yes, the repository is licensed under Apache-2.0 and available at no cost.
Is trnrt open source?
Yes, the source code is publicly available on GitHub under an Apache-2.0 license.
What is trnrt used for?
It is used to run large language model inference on AWS Trainium hardware with optimized kernels and an OpenAI-compatible API.
Does trnrt work on Mac?
The website does not say. It requires Docker and an x86_64 host for compilation, targeting AWS Trainium for serving.
What hardware does trnrt require?
It requires a trn2.48xlarge instance for serving and any x86_64 host for offline graph compilation.
Does trnrt support OpenAI clients?
Yes, it provides an OpenAI-compatible API that handles chat, completions, streaming, tool calls, and reasoning output.
Comments (0)
Comments and upvotes open when this product launches on ProductPerch.
More AI launches
Generate and edit images in plain language.
Create personalized AI portraits, images and videos.
Star in the viral Bulin 47 freestyle video from one photo
Agent BarnOpen source · verified
Run your AI agents from one place
Try on 1000+ real hairstyles with realistic AI previews
Turn a portrait into a 23-second AI party video