The AI Stack, Part 3: The Engine Room

GPUs, vLLM, and KV Caches: How you can scale AI infrastructure without going broke

Your models might be incredible, but if latency and cloud compute bills ruin the user experience, you don’t have a viable business.

If you are a Software Engineer or Tech Lead and you are looking for a clear blueprint to orchestrate and scale the infrastructure behind modern AI, then this email is for you.

In today’s issue, I'll share the Infrastructure Layer, the grand finale of our technical series on the AI stack.

Read the previous issues of this series

[The A-Side] The AI Stack, Part 1: Building Product-First (#169)

The AI Stack, Part 2: The Art of Specialization

Before you continue, if you enjoy my work and would like to support the newsletter, I’d really appreciate it if you shared it with a fellow engineer:

💚 On WhatsApp | 💙 On LinkedIn | 🩵 Bluesky

Or simply forward this email to someone who might find it useful.

The moment of truth has arrived... Let’s step right into the engine room where the magic happens (and where you spend your budget).

This is the technical bedrock and economic engine of your entire system. In modern AI engineering, this layer is strictly dictated by one huge bottleneck: GPU/TPU scarcity and compute costs.

It breaks down into three vital areas for you to master:

👉🏼 Model Serving

You host models to deliver lightning-fast responses. You leverage optimized execution engines like vLLM, TensorRT-LLM, or SGLang that use architectures like PagedAttention to manage Key-Value (KV) cache memory with pinpoint efficiency.

👉🏼 Compute Management & Scaling

You handle heavy concurrent traffic using continuous batching and dynamically route queries between commercial API providers and your self-hosted instances to minimize latency and bill size.

👉🏼 Data Pipelines & Storage

Your high-speed data plumbing. You connect transactional company systems in real time to vector databases and context buffers that feed your Layer 1 RAG pipelines.

🧐 Real-world scenario

You route simple user queries to a cheap commercial API while reserving your expensive, self-hosted vLLM clusters for heavy, latency-critical corporate workloads.

☝🏼 Key takeaway

Your inference cost is your biggest long-term margin killer. Optimizations you make in this layer determine whether your AI product survives its first year financially.

And with that, we close the loop on all 3 essential layers of AI Engineering.

Let’s do one final recap.

• Serving: Leverage vLLM and PagedAttention to maximize GPU memory efficiency.

• Compute & Routing: Balance traffic and route queries dynamically to control your costs.

• Data: Build real-time, high-throughput pipelines to continuously feed your vector stores.

Building with AI doesn’t mean you have to master all three layers on day one. Start at the Application Layer, perfect your evaluation, and only descend down the stack when your product requirements explicitly demand it!

Across all three layers we covered in this series (Application, Model, and Infrastructure), which one are you currently spending most of your engineering effort on?

Drop a message in the comments or reply to this email, I read and reply to all!

Be safe,
Marcos.