Chapter 6 of 13

The Gateway: Making Azure AI Foundry Look Like OpenAI

This is the module the entire project exists to support. Everything before this (auth, crypto, virtual keys) was infrastructure to make this part safe to expose. Everything after it (usage tracking, the dashboard) is infrastructure to make this part observable and manageable. The gateway itself is @Controller('v1'), deliberately matching OpenAI’s URL shape so any client SDK that already speaks OpenAI’s API just needs a different base URL and a different key.

A second, separate auth track

The gateway controller is marked @Public() at the class level, which sounds backwards until you remember the global JwtAuthGuard from the auth chapter. Gateway routes need to bypass that guard entirely, they’re not authenticated with an admin JWT, they’re authenticated with a virtual key. So @Public() opts out of the JWT guard, and @UseGuards(VirtualKeyGuard) opts back into a completely separate one:

@ApiTags('gateway')
@Public()
@UseGuards(VirtualKeyGuard)
@Controller('v1')
export class GatewayController {
  constructor(private readonly gatewayService: GatewayService) {}

  @Get('models')
  listModels(@CurrentVirtualKey() virtualKey: AuthenticatedVirtualKey) {
    return this.gatewayService.listModels(virtualKey);
  }

  @Post('chat/completions')
  chatCompletions(
    @CurrentVirtualKey() virtualKey: AuthenticatedVirtualKey,
    @Body() body: Record<string, unknown>,
    @Res() res: ExpressResponse,
  ): Promise<void> {
    return this.gatewayService.chatCompletions(virtualKey, body, res);
  }
}

VirtualKeyGuard hashes the bearer token, looks it up, checks status and expiry, and attaches a plain object to the request with everything downstream code needs, including the allowed deployment ids as a Set for cheap membership checks:

request.virtualKey = {
  id: virtualKey.id,
  allowedDeploymentIds: new Set(
    virtualKey.allowedModels.map((e) => e.deploymentId),
  ),
  maxBudget: virtualKey.maxBudget ? virtualKey.maxBudget.toNumber() : null,
  budgetPeriod: virtualKey.budgetPeriod,
  rateLimitRpm: virtualKey.rateLimitRpm,
  rateLimitTpm: virtualKey.rateLimitTpm,
};

That’s fetched once by the guard, not re-queried by every downstream check, since it would otherwise mean a database round trip for rate limiting, another for budget checking, and another for model resolution, all on every single request.

Real request bodies don’t fit a strict DTO

I went back and forth on this one. My instinct from building the admin API was: every request body gets a class-validator DTO with whitelist: true, so unknown fields get stripped and required fields get enforced. I tried that here first, and it broke almost immediately, because a real OpenAI chat completion body has dozens of optional fields (temperature, top_p, tools, response_format, logit_bias, and more that get added over time). A strict whitelist DTO would silently drop any field it doesn’t explicitly know about, which quietly breaks any client relying on a feature I hadn’t modeled yet.

The DTOs (ChatCompletionRequestDto, EmbeddingsRequestDto) still exist, but only for Swagger documentation. The actual request handling types the body as Record<string, unknown>, skips Nest’s DTO pipeline entirely, and does the minimum manual check that actually matters:

private assertChatCompletionShape(body: Record<string, unknown>): void {
  if (typeof body.model !== 'string' || !body.model) {
    throw new BadRequestException('Request body must include a "model" field');
  }
  if (!Array.isArray(body.messages) || body.messages.length === 0) {
    throw new BadRequestException('Request body must include a non-empty "messages" array');
  }
}

That’s it. Everything else passes straight through to Azure. It’s less strict than I’d normally want an API to be, but “stay compatible with whatever the OpenAI SDK sends” won out over “validate everything,” because the whole value proposition of this gateway is that existing code doesn’t need to change.

Talking to Azure

AzureOpenAiClient builds Azure’s specific URL shape ({endpoint}/openai/deployments/{deploymentName}/{path}?api-version=...) and sends the API key as a header, not a bearer token, which is how Azure OpenAI and Azure AI Foundry both expect it:

function buildUrl(deployment: DeploymentCredentials, path: string): string {
  const endpoint = deployment.endpoint.replace(/\/+$/, '');
  return `${endpoint}/openai/deployments/${deployment.deploymentName}/${path}?api-version=${deployment.apiVersion}`;
}

It’s a plain fetch call, no SDK wrapper. I didn’t see a reason to pull in the Azure OpenAI SDK for two request shapes I was already hand-modeling.

Fallback on retryable failures

ModelDeployment.fallbackForId, modeled back in the schema chapter, gets used for real here. If the primary deployment responds with a retryable status, the gateway looks up a deployment whose fallbackForId points at the primary and retries against that instead:

const RETRYABLE_STATUS_CODES = new Set([429, 500, 502, 503, 504]);

private async callWithFallback(primary, call) {
  const primaryCredentials = await this.providersService.getDecryptedCredentials(primary.id);
  const response = await call(primaryCredentials);

  if (response.ok || !RETRYABLE_STATUS_CODES.has(response.status)) {
    return response;
  }

  const fallback = await this.prisma.modelDeployment.findFirst({
    where: { fallbackForId: primary.id, isActive: true },
  });

  if (!fallback) return response;

  const fallbackCredentials = await this.providersService.getDecryptedCredentials(fallback.id);
  return call(fallbackCredentials);
}

I tested this against a temporary local mock Azure server that I deliberately made return 429 on the primary deployment, and confirmed the retry hit the fallback and returned a real successful response, for both plain JSON and streaming SSE. That mock server never got committed, it existed purely as a way to prove the retry logic without spending real Azure quota on inducing failures on purpose.

Streaming without buffering the whole response

Chat completions can stream. The upstream Azure response body is a ReadableStream, and it gets piped straight to the Express response as chunks arrive, not buffered and re-sent:

const reader = upstream.body.getReader();
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  res.write(Buffer.from(value));

  tail += decoder.decode(value, { stream: true });
  const lines = tail.split('\n');
  tail = lines.pop() ?? '';
  for (const line of lines) {
    const parsed = this.tryParseUsageChunk(line);
    if (parsed) usage = parsed;
  }
}
res.end();

The interesting part is that it’s doing two things with the same bytes at once: forwarding them to the client immediately for real-time streaming, and also parsing each SSE line looking for a usage chunk, so token counts can get logged once the stream ends. To make that usage chunk actually show up, chat completion requests automatically get stream_options: { include_usage: true } injected if the client didn’t already ask for it, since that’s what makes Azure emit the final usage payload as part of the stream instead of leaving it out entirely.

One bug worth mentioning: ProvidersService.create originally let an unhandled Postgres unique-constraint violation bubble up as a raw 500 whenever I tried to register a deployment alias that already existed. I noticed it while testing duplicate aliases on purpose, and added a small utility, rethrowAsConflict, that catches Prisma’s P2002 error code and turns it into a proper ConflictException (409) instead. Small fix, but it’s the difference between an API that tells you what went wrong and one that just says “something broke.”

Next: usage tracking, budgets, and rate limiting, which is also where I’ll walk through a genuinely embarrassing Postgres bug that made it all the way to a deployed environment before I caught it.