LLM Gateways, a Software Engineering Perspective

Explore LLM Gateways like LiteLLM, examining engineering concepts like load balancing, moderation, and abstraction for production inference deployments.

Overview

Taking a deep look into LLM Gateways (LiteLLM or similar), and the software engineering concepts behind the scene.

We will be looking into the features of an LLM Gateway, from loadbalancing, moderation, abstractive approach and more.

Tech stack