The problem#
Every model deployment seems to start the same way: a hand-written Flask endpoint, no versioning, no record of what changed and no idea how it performs once it's live.
The approach#
InferX turns deployment into an upload. You upload a scikit-learn model and get a versioned prediction endpoint, API keys, rate limits, webhooks and a dashboard for its traffic, without writing any serving code.
- 35+ REST endpoints for deployment, versioning and real-time inference, behind a modular service layer designed from HLD and LLD up.
- Auth and abuse protection from the start: JWT and OAuth 2.0 login, hashed API keys, per-model sharing, Argon2 password hashing, HMAC-signed webhooks and Redis-backed rate limits on every route.
Architecture#
A Next.js dashboard and direct API clients both talk to one FastAPI app. The app keeps users, model metadata and prediction logs in PostgreSQL through SQLAlchemy, stores model files on S3 (or local disk in development), and uses Redis for two things: cached predictions and rate-limit counters.
Anything the caller doesn't need to wait for, like writing the prediction log, caching the result and firing webhooks, runs as a FastAPI background task after the response is sent.
Hover or tap any part to trace what it talks to. 16 parts, 20 connections. Swipe sideways to see it all.
- 1A client POSTs input to /predictions/{model_id}, either from the dashboard with a JWT or from an app with an X-API-Key.
- 2The middleware stack logs and times the request, then get_current_user resolves the caller from the token or the hashed key in PostgreSQL.
- 3The route's rate limit is checked in Redis with a sliding window; over the limit returns 429.
- 4The model record is loaded from PostgreSQL, the input is hashed, and Redis is checked for a cached prediction. A hit returns immediately.
- 5On a miss the model loader reads the file through the storage service (or its in-memory cache), runs predict and returns the result.
- 6After the response, background tasks write the result to Redis with a one hour TTL and log the prediction to PostgreSQL.
- 7Secondary path: the same tasks call trigger_webhooks, which POSTs an HMAC-SHA256 signed event to each registered URL and retries on failure.
Making inference fast#
A prediction request builds a cache key from the model id, version and a SHA-256 hash of the sorted input JSON. A hit returns straight from Redis. A miss loads the model through an in-process LRU of loaded models, runs predict, and queues the cache write, the database log and the webhooks for after the response.
def _generate_cache_key(self, model_id: str, input_data: Any, version: Optional[int] = None) -> str:
# Key format: pred:{model_id}:{version}:{input_hash}
input_json = json.dumps(input_data, sort_keys=True, default=str)
input_hash = hashlib.sha256(input_json.encode()).hexdigest()[:16]
version_str = str(version) if version else "latest"
return f"pred:{model_id}:{version_str}:{input_hash}"Including the version in the key means a new model version never serves a stale answer, and sort_keys means {"a":1,"b":2} and {"b":2,"a":1} share an entry. Cached entries expire after an hour, and a model's entries can be invalidated with one call.
Shipping it#
- Docker multi-stage builds (base, dependencies, development, production) cut the image size by 65%.
- 77% test coverage from 39+ pytest suites, with Pydantic-validated request and response schemas, and a GitHub Actions workflow that runs them against real PostgreSQL and Redis service containers.
What I learned#
The model is the smallest part of a model deployment. Auth, versioning, caching and tests are what make it something other people can use.