Once the gateway could actually route requests to Azure, the next problem was enforcement. Nothing was stopping a virtual key from making a thousand requests a second or blowing through however much budget I mentally allocated to a side project. This chapter covers rate limiting, budget checks, and cost estimation, plus a real bug in this exact code that I found and fixed while writing this series (the fix is already in the repo, commit 58b252b).
Redis for anything that can be approximate and fast
Requests-per-minute and tokens-per-minute limits live in Redis, using fixed sixty-second windows keyed by virtual key id:
const WINDOW_SECONDS = 60;
async assertRpm(virtualKeyId: string, limitRpm: number | null): Promise<void> {
if (!limitRpm) return;
const key = this.windowKey('rpm', virtualKeyId);
const count = await this.redis.client.incr(key);
if (count === 1) {
await this.redis.client.expire(key, WINDOW_SECONDS);
}
if (count > limitRpm) {
throw new HttpException('Rate limit exceeded: requests per minute', HttpStatus.TOO_MANY_REQUESTS);
}
}Tokens-per-minute is trickier, because I don’t know how many tokens a request will use until Azure has already answered it. So assertTpm checks the count from the previous window before the call happens (true pre-call enforcement isn’t actually possible here), and recordTokenUsage adds the real count after the response comes back, which is what actually gets enforced on the next request in that window:
async recordTokenUsage(virtualKeyId: string, tokens: number): Promise<void> {
if (tokens <= 0) return;
const key = this.windowKey('tpm', virtualKeyId);
const count = await this.redis.client.incrby(key, tokens);
if (count === tokens) {
await this.redis.client.expire(key, WINDOW_SECONDS);
}
}I’m fine with this being slightly eventually-consistent. A burst of concurrent requests at the very start of a window could theoretically slip past the token check before the running total catches up. For a personal gateway protecting Azure credits, that’s an acceptable tradeoff for not needing distributed locks around every request.
Postgres for anything involving money
Budget enforcement is different. I made a deliberate choice not to track spend in Redis at all, even though it would have been faster. Money needs one source of truth, and I didn’t want a world where Redis and Postgres could disagree about how much a key has spent because of a missed write or a Redis restart. BudgetService sums directly from UsageLog in Postgres, scoped to the budget period:
async assertWithinBudget(virtualKeyId: string, maxBudget: number | null, budgetPeriod: string): Promise<void> {
if (!maxBudget) return;
const periodStart = this.periodStart(budgetPeriod);
const result = await this.prisma.usageLog.aggregate({
_sum: { estimatedCost: true },
where: {
virtualKeyId,
...(periodStart ? { createdAt: { gte: periodStart } } : {}),
},
});
const spent = result._sum.estimatedCost?.toNumber() ?? 0;
if (spent >= maxBudget) {
throw new ForbiddenException('Virtual key budget exceeded');
}
}DAILY and MONTHLY periods compute a UTC boundary, TOTAL has no lower bound at all, meaning it sums everything the key has ever spent. During testing I set a budget of 0.00001 on a key to try to trigger the limit quickly, and it never triggered. That confused me for a minute until I remembered maxBudget is a Decimal(12, 4) column, and 0.00001 rounds down to 0.0000, which is falsy in the if (!maxBudget) return; check. Not a bug, just a reminder that test values need to respect the actual column precision, not an abstract idea of “a very small number.”
GatewayService.enforceLimits runs all three checks, in a specific order, before the Azure call happens, and that order is non-negotiable:
private async enforceLimits(virtualKey: AuthenticatedVirtualKey): Promise<void> {
await this.rateLimiter.assertRpm(virtualKey.id, virtualKey.rateLimitRpm);
await this.rateLimiter.assertTpm(virtualKey.id, virtualKey.rateLimitTpm);
await this.budgetService.assertWithinBudget(virtualKey.id, virtualKey.maxBudget, virtualKey.budgetPeriod);
}Rate limits first because they’re cheapest to check (Redis), budget last because it’s the most expensive (a Postgres aggregate). All of them happen before a single byte goes to Azure, since there’s no point spending real Azure quota on a request that’s going to get rejected anyway.
A race I found by actually reading the ordering carefully
Early on, chatCompletions sent the response back to the client and then, separately, awaited the usage log write. That’s backwards for the non-streaming case: a very fast second request from the same key could read a stale budget total, because the first request’s usage row hadn’t committed to Postgres yet. I fixed it by awaiting finalizeUsage before writing the response for the non-streaming path. Streaming didn’t have this problem, since res.end() naturally happens only after the whole stream, and therefore the usage recording, has already completed.
The bug: comparing text to uuid
Here’s the one I want to walk through in detail, because it’s a good example of a bug that only shows up under a specific, easy-to-miss condition. UsageService.getSummary powers both the dashboard spend chart and the per-key detail page, and it accepts an optional virtualKeyId to scope the query to one key. The original raw SQL looked like this:
WHERE ${virtualKeyId ? Prisma.sql`virtual_key_id = ${virtualKeyId}::uuid` : Prisma.sql`TRUE`}That ::uuid cast looks reasonable if you’re used to Postgres columns actually being the uuid type. Mine aren’t. Every id in this schema, including virtual_key_id on usage_logs, is a Prisma String mapped to a plain Postgres TEXT column (that’s what @id @default(uuid()) on a String field produces, not a native uuid column). So this query was comparing a text column against a value explicitly cast to uuid, and Postgres doesn’t have an = operator defined between those two types without an explicit cast on both sides. The result was a hard failure:
Error: P1000...
No operator matches the given name and argument types. You might need to add explicit type casts.Which surfaced as a plain 500 on GET /api/usage/keys/:id, which meant the entire virtual key detail page in the dashboard just showed an error alert instead of the spend chart. I found this by actually tailing the API’s logs while reproducing the failure in the browser, rather than guessing from the frontend error message alone, since the frontend only knew “the request failed,” not why. The fix is one line, dropping the unnecessary cast entirely:
WHERE ${virtualKeyId ? Prisma.sql`virtual_key_id = ${virtualKeyId}` : Prisma.sql`TRUE`}Comparing text to text, which is what the column actually is. It’s a small fix, but it’s a good reminder that raw SQL doesn’t get the type safety Prisma’s query builder gives you for free, and that “id” in a schema doesn’t automatically mean “Postgres uuid type” just because the value looks like one.
Next: switching over to the frontend, and the Angular side of this project, starting with a zoneless flag that I thought I’d already turned on.