Prepare for Production

A short checklist before you put the SDK in front of users.

Handle compatibility errors at startup

The client validates the Engine's contract version and tokenizer identity on first use and raises EngineCompatibilityError when the deployment and the SDK release do not match. Surface this clearly — it means "upgrade the SDK or the Engine", not "retry". SDK 0.4.1 accepts Engine contracts in the range >=1.2.0 <2.0.0; the rules are documented in Compatibility.

Pre-release Engine contracts are rejected unless you opt in explicitly (allowPrereleaseContract / allow_prerelease_contract).

Route deprecation warnings

The Engine announces upcoming breaking changes as deprecation notices. The SDK surfaces each notice once — by default through console.warn (TypeScript) or the warnings module (Python). In production, route them to your logging system:

TypeScript

const client = new InveraClient({
  baseUrl,
  warn: (message) => logger.warn({ source: "invera" }, message),
});

Python

client = InveraClient(
    base_url,
    warn=lambda message: logger.warning("invera: %s", message),
)

Timeouts and cancellation

• TypeScript — pass an AbortSignal per request (stream({ messages, signal })); abort it to stop the stream.
• Python — configure timeout_seconds on the client (default 60).

Node.js and Docker deployment

The default TypeScript tokenizer loads @huggingface/transformers and its native onnxruntime-node dependency when the first completion starts. The currently supported Linux baseline is glibc. Use a glibc-based image such as node:22-bookworm-slim for both build and runtime stages.

node:*-alpine uses musl and is not supported by the default tokenizer's current ONNX Runtime binaries. Only use Alpine if your application supplies and validates a compatible musl build of the complete native runtime.

The tokenizer package is loaded through a computed dynamic import. Framework bundlers and output tracers cannot always discover that import, so a successful server build does not prove the tokenizer is present. The production image must explicitly retain @huggingface/transformers, onnxruntime-node, and their transitive runtime dependencies alongside the server output.

npm application

This pattern builds the server, removes development-only dependencies, and copies the complete production dependency tree into the runtime stage. Replace .output and the command with your framework's server output when necessary.

FROM node:22-bookworm-slim AS build
WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci

COPY . .
RUN npm run build
RUN npm prune --omit=dev

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production

COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/.output ./.output

CMD ["node", ".output/server/index.mjs"]

Do not copy only the files reported by framework output tracing unless you have also configured the framework to include the tokenizer's full native dependency tree.

pnpm monorepo

In a pnpm workspace, pnpm deploy can produce an isolated production dependency tree. This example is useful when the application output is already self-contained except for the SDK's dynamically loaded tokenizer dependencies:

FROM node:22-bookworm-slim AS build
WORKDIR /app

RUN corepack enable
COPY . .
RUN pnpm install --frozen-lockfile
RUN pnpm --filter <server-package> build
RUN pnpm --filter @layerwise/invera-sdk deploy --prod --legacy /runtime-sdk

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production

COPY --from=build /runtime-sdk/node_modules ./node_modules
COPY --from=build /app/<server-package>/.output ./.output

CMD ["node", ".output/server/index.mjs"]

If the runtime server has other external production dependencies, deploy the server package itself instead, or copy its complete pruned production tree as well. Verify the final image by starting a real completion, because model discovery alone does not initialize the tokenizer.

Hugging Face downloads and cache

On its first load, the tokenizer fetches the release-pinned tokenizer and model metadata assets from Hugging Face. Production therefore needs outbound HTTPS access, or those exact revision-pinned assets must already be available in a filesystem cache. Transformers.js uses ./.cache by default, relative to the process working directory; in the examples above that is /app/.cache.

For containers that may restart or scale out, pre-populate that cache in the image or mount it as a persistent, writable volume. A read-only or ephemeral filesystem without outbound access will fail when the first completion tries to initialize the tokenizer.

Troubleshooting

Symptom	Cause	Fix
Install @huggingface/transformers... or a module-not-found error	The bundler or output tracer omitted the computed dynamic import and its dependency tree.	Copy or deploy the complete production dependencies, including @huggingface/transformers, onnxruntime-node, and transitive dependencies.
ld-linux-x86-64.so.2 is missing or libonnxruntime.so cannot load	The image is Alpine/musl, but the installed ONNX Runtime native library expects glibc.	Rebuild both stages from node:22-bookworm-slim, or supply a tested musl-compatible native runtime.
Asset download or cache errors on the first completion	The tokenizer assets are not cached and the runtime cannot reach Hugging Face, or its cache is not writable.	Allow outbound access or provide a writable, pre-populated cache containing the pinned revision.

Show what you are connected to

client.models.current() returns the model id, profile, Engine build, and context window as a stable shape — use it for diagnostics pages and support bundles instead of the raw discovery document.

Treat evidence scores as signals

Grounding scores quantify how strongly generated words trace back to context — they are not factual verification. Calibrate thresholds against your own data before hiding or blocking content on them, and prefer surfacing requires_review to a human over silently discarding responses. If you need semantic support/contradiction judgements, plug in a verifier.

Pin and upgrade deliberately

Pin the SDK version in your lockfile or requirements and read the Compatibility rules before upgrading either the SDK or the Engine deployment. Documentation for previous SDK versions is available from the version dropdown in the navigation bar.