FreeLLMAPI is a self-hosted proxy that aggregates the free tiers of 28 LLM providers (339 model endpoints, roughly 4 billion tokens per month) behind a single OpenAI-compatible /v1 endpoint. It handles routing, automatic failover when a provider hits quota, and per-key usage tracking to keep you under each free-tier cap.
The project hit 17K stars in days because it solves a practical problem: LLM costs kill side projects and experimentation. If you can stack free tiers intelligently, you can run agents, prototypes, and hobby tools without a credit card.
Architecture
FreeLLMAPI sits between your agent code and 28+ upstream providers. When a request arrives at /v1/chat/completions, the router:
- Parses the requested model name.
- Looks up which providers offer that model (or a compatible fallback).
- Checks per-provider quota state in memory.
- Calls the first available provider.
- If that provider returns a rate-limit error, moves to the next provider in the fallback chain.
- Increments the usage counter for the provider that succeeded.
API keys for all 28 providers are stored encrypted at rest using AES-256-GCM. The proxy decrypts them on startup and holds them in memory. The threat model assumes you trust the host where the proxy runs; there is no HSM or secret-manager integration yet.
The model catalog updates via a signed feed from freellmapi.co. The proxy polls the feed, verifies the signature, and merges new models, quota changes, and compatibility fixes without restarting. This means you get new free tiers as providers launch them, without a git pull or Docker rebuild.
Routing and Failover
The router uses a priority list per model. For example, if you request gpt-4o-mini, the router might try:
- OpenRouter (free tier: 200 requests/day)
- Groq (free tier: 14,400 requests/day, 30 req/min)
- Together AI (free tier: $25 credit)
If OpenRouter returns HTTP 429, the router immediately retries with Groq. If Groq also rate-limits, it falls to Together AI. The client sees a single response, not three retries.
What happens mid-stream? If a provider rate-limits after sending the first chunk of a streaming response, the proxy surfaces the error to the client. It does not attempt to resume the stream from a different provider because the partial response has already been sent. Non-streaming requests retry transparently.
Quota tracking is in-memory and resets when the proxy restarts. Each provider has a token budget (configured in the catalog). The proxy increments a counter after every successful request. When the counter exceeds the budget, that provider is skipped until the next reset window (daily or monthly, depending on the provider’s terms). This is a soft limit; the proxy does not enforce it at the provider level, so you can still hit rate limits if multiple proxy instances share the same API key.
State Management
| Component | Storage | Persistence | Reset Trigger |
|---|---|---|---|
| API keys | Encrypted file (AES-256-GCM) | Survives restart | Manual rotation |
| Quota counters | In-memory map | Lost on restart | Daily/monthly cron or restart |
| Model catalog | JSON file + signed feed | Survives restart | Feed update (polled every 6h) |
| Request logs | Optional stdout/file | Configurable | Log rotation policy |
The lack of persistent quota state means restarting the proxy resets all counters to zero. If you run multiple instances behind a load balancer, each instance tracks quota independently, so you can exceed free-tier limits. The recommended deployment is a single instance per API key set.
Security Boundaries
Encryption at rest: API keys are encrypted with a master key derived from an environment variable (FREELLM_MASTER_KEY). If an attacker reads the encrypted key file without the master key, they get ciphertext. If they read the environment or memory, they get plaintext keys.
No credential isolation: All 28 provider keys live in one process. A memory-disclosure bug (buffer overflow, side channel) exposes every key. The proxy does not sandbox provider calls or use separate processes per provider.
No audit log: The proxy logs request counts and errors but does not log which user (if you add auth) called which model with what prompt. If you need compliance-grade audit trails, you will need to add middleware.
No built-in auth: The /v1 endpoint is unauthenticated by default. Deploy behind a reverse proxy (Caddy, nginx) with HTTP Basic Auth or OAuth if you expose it to the internet.
Deployment Shape
Docker:
docker run -d \
-p 3000:3000 \
-e FREELLM_MASTER_KEY=your-secret-key \
-v $(pwd)/keys.enc:/app/keys.enc \
ghcr.io/tashfeenahmed/freellmapi:latestBare metal:
git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi
npm install
npm run build
FREELLM_MASTER_KEY=your-secret-key npm startThe proxy listens on port 3000 by default. Point your agent code at http://localhost:3000/v1 instead of https://api.openai.com/v1.
Observability: The proxy exposes basic metrics at /metrics (Prometheus format): request count, error rate, per-provider quota usage. No distributed tracing yet. If you run this in production, add a sidecar like Grafana Agent or OTEL Collector.
Drop-In Replacement
Because the proxy implements the OpenAI /v1 spec, you can swap it into existing agent code with a one-line change:
# Before
client = OpenAI(api_key="sk-...")
# After
client = OpenAI(
api_key="not-used", # proxy ignores this
base_url="http://localhost:3000/v1"
)The proxy translates your request into the upstream provider’s format (OpenRouter, Groq, Together AI all have slight variations) and translates the response back into OpenAI’s schema. This works for chat completions, embeddings, and image generation. Audio endpoints are in the catalog but not yet wired up.
Failure Modes
Single point of failure: If the proxy crashes, all your agents lose LLM access. Run it under a process supervisor (systemd, Docker restart policy, Kubernetes liveness probe).
Quota exhaustion: If you burn through all 28 free tiers in one day, the proxy returns HTTP 429 with no fallback. You will need to wait for the next reset window or add a paid provider to the catalog.
Provider API changes: If a provider changes their API schema (new required field, deprecated endpoint), the proxy will fail until the catalog is updated. The signed feed mitigates this, but there is still a window between the provider’s change and the feed update.
Latency stacking: If the first provider in the fallback chain is slow or times out, the proxy waits for the timeout before trying the next provider. This can add seconds to your request. The default timeout is 30 seconds per provider.
Memory leaks: Quota counters grow unbounded if you never restart. A long-running proxy with high traffic will eventually OOM. Add a cron job to restart daily or implement counter expiration.
Technical Verdict
FreeLLMAPI is a clever hack for cost-constrained experimentation. The router logic is straightforward, the failover works, and the OpenAI-compatible interface means you can drop it into existing code. The signed feed is a smart way to keep the catalog fresh without manual updates.
Use it if:
- You are prototyping or building side projects where LLM cost is a blocker.
- You control the host and can secure the master key in the environment.
- You can tolerate occasional rate-limit errors and multi-second failover latency.
- You want to experiment with multiple models without managing 28 sets of API keys.
- You are comfortable with in-memory quota state that resets on restart.
Avoid it if:
- You need high availability or SLA guarantees (no persistent state, single point of failure).
- You run multi-tenant systems (no per-user quota isolation or credential sandboxing).
- You handle sensitive user data or need compliance-grade audit logs.
- You require sub-second response times (failover can add 30+ seconds per retry).
- You plan to run multiple instances behind a load balancer (quota tracking breaks).
Deploy this on a trusted host, not a shared server. If you need HA, observability, or compliance, build on top of this or use a commercial LLM gateway.