Free launch days are booked ahead. A paid launch can go live tomorrow for $3.90.

Launch sooner

trnrt

Open LLM inference engine for AWS Trainium

AIOpen sourceOpen source · verified#llm-inference#aws-trainium#open-source

Listed by the ProductPerch team from Hacker News · not launched here yet

What is trnrt?

Trnrt is an open source inference engine built for running large language models on AWS Trainium hardware. It serves models through an OpenAI-compatible API while optimizing performance using specialized NKI megakernels. Each layer half uses a single fused kernel for decoding, which improves throughput and reduces latency.

The engine supports a context window of up to one million tokens. It achieves this through sparse attention and linear attention mechanisms, combined with a fixed-size state. A prefix cache stores key-value pages in FP8 format and moves evicted data to host memory. Two separate engines can run per instance, handling tensor parallelism across 32 logical NeuronCores behind a session-affine router.

Graph compilation happens offline on any x86_64 host before deployment. The serving instance loads precompiled graphs without performing runtime compilation. Currently, GLM-5.3-Flash is the only supported model, though additional open models are planned. Setup requires Docker, a trn2.48xlarge instance for serving, and a separate host for building the compile cache.

trnrt features

  • NKI megakernels for efficient decode operations
  • Supports up to 1 million token context window
  • Prefix cache using radix tree and host memory
  • OpenAI compatible API for chat and completions
  • Offline graph compilation on CPU hosts
  • Tensor parallel execution across 32 NeuronCores

Who uses trnrt and for what

  • Deploying large language models on AWS Trainium hardware
  • Running long-context applications requiring 1M token support
  • Integrating custom LLMs with existing OpenAI client libraries
  • Optimizing inference costs through fused kernel execution
  • Testing model performance with offline compilation workflows

trnrt FAQ

Is trnrt free?

Yes, the repository is licensed under Apache-2.0 and available at no cost.

Is trnrt open source?

Yes, the source code is publicly available on GitHub under an Apache-2.0 license.

What is trnrt used for?

It is used to run large language model inference on AWS Trainium hardware with optimized kernels and an OpenAI-compatible API.

Does trnrt work on Mac?

The website does not say. It requires Docker and an x86_64 host for compilation, targeting AWS Trainium for serving.

What hardware does trnrt require?

It requires a trn2.48xlarge instance for serving and any x86_64 host for offline graph compilation.

Does trnrt support OpenAI clients?

Yes, it provides an OpenAI-compatible API that handles chat, completions, streaming, tool calls, and reasoning output.

Comments (0)

Comments and upvotes open when this product launches on ProductPerch.

More AI launches

  • Seedream 5.5

    Generate and edit images in plain language.

    AIFreemium0

  • APOB AI

    Create personalized AI portraits, images and videos.

    AIFreemium0

  • Bulin 47 AI

    Star in the viral Bulin 47 freestyle video from one photo

    AIFreemium0

  • Agent BarnOpen source · verified

    Run your AI agents from one place

    AIOpen source1110

  • AI Hairstyle

    Try on 1000+ real hairstyles with realistic AI previews

    AIFreemium0

  • Burning Bridges Studio

    Turn a portrait into a 23-second AI party video

    AIPaid0

All AI tools

Launch packs

Buy paid launches