Your data science team trained a model that works on their laptops. Now a product manager wants it live in the app by the end of the quarter, and the two sides cannot agree on how to ship it. That gap, between a model that runs in a notebook and one that serves real users, is where most AI features stall.
To deploy AI models with Docker, you package the model, its runtime, and every dependency into a single container image, expose it behind an inference API, and run that image on a container platform such as Kubernetes or a managed container service. Docker makes the model reproducible and portable, so it behaves the same on a laptop, in staging, and in production, regardless of the host’s Python or CUDA version.
This guide covers how that works step by step, what it costs in engineer time, when a container is the wrong tool, and the specific mistakes we see teams make the first time they put a model into production.
What does it mean to deploy an AI model with Docker?
A Docker container is a lightweight, isolated environment that bundles your model together with everything it needs to run: the Python interpreter, the ML framework such as PyTorch, TensorFlow, or ONNX Runtime, the system libraries, and the model weights themselves. You describe that environment once in a text file called a Dockerfile, build it into an image, and run copies of that image as containers.
Deploying an AI model with Docker means turning your trained model into one of these images and running it as a service that other systems call. Instead of asking every server to have the right Python version and CUDA drivers installed, you ship a self-contained artifact that already has them. The host only needs Docker.
The result is an inference service: a small API that takes an input (a text string, an image, a row of features), runs it through the model, and returns a prediction. Everything the model needs travels inside the image.
Why deploy AI models with Docker instead of a bare server?
Four reasons come up on almost every project.
Reproducibility. An ML model is sensitive to library versions. A model trained against PyTorch 2.1 can behave differently, or fail to load, under 2.3. The image pins every version, so the model you tested is the model that runs.
Dependency isolation. AI workloads pull in heavy, conflicting dependencies: CUDA, cuDNN, specific NumPy builds. Containers keep each model’s stack separate, so two models with different requirements run on the same host without fighting.
Parity across environments. The classic “it works on my machine” problem disappears. The same image runs on a developer laptop, a staging cluster, and production, which makes bugs reproducible and rollbacks instant.
Scaling and recovery. When traffic climbs, the platform starts more copies of the image. When a container crashes, it is replaced in seconds from the same known-good artifact.
How do you deploy an AI model with Docker, step by step?
The path from a trained model to a running service is six steps. The diagram below shows the whole sequence, and the sections after it explain the parts that trip teams up.
1. Package the model. Export the trained weights to a portable format such as a .pt or .onnx file and write a small inference script that loads the weights and exposes a predict function.
2. Write the Dockerfile. Choose a base image. For CPU inference, a slim Python image keeps the size down. For GPU inference, start from an official NVIDIA CUDA base image so the drivers line up. Install only the libraries the model needs at inference time, not the full training stack.
3. Build the image. Run the build, which executes the Dockerfile and bakes the model, runtime, and dependencies into one versioned artifact.
4. Push to a registry. Store the image in a private container registry such as Amazon ECR, Google Artifact Registry, or Azure Container Registry that your production platform can pull from.
5. Serve behind an API. Wrap the model in a web server such as FastAPI or TorchServe so other services can send requests and receive predictions over HTTP.
6. Monitor and scale. Track latency, error rates, and, for GPU workloads, memory. Add replicas as traffic grows and roll back to the previous image tag if a new model misbehaves.
# CPU inference image for a PyTorch model served with FastAPI
FROM python:3.11-slim
WORKDIR /app
# Install only inference dependencies, not the training stack
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy the model weights and the inference code
COPY model/ ./model/
COPY app.py .
# One request in, one prediction out
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]For a GPU model, the only meaningful change is the base image. Swap python:3.11-slim for an nvidia/cuda image and install the GPU build of your framework. The application code stays the same.
CPU or GPU containers: which does your model need?
Not every model needs a GPU in production, and GPUs are the single largest cost in most AI deployments. The rule of thumb is to match the container to the model’s real latency budget, not to the hardware you trained on.
Best for
Smaller models, text classification, tabular models, low request volume
Large language models, image or video models, high throughput
Base image
python:slim, about 150 MB
nvidia/cuda, 1 to 3 GB
Cost
Low, runs on standard compute
High, needs GPU instances billed by the hour
Setup effort
Minimal
NVIDIA Container Toolkit plus driver matching
Typical fit
Models where CPU latency is acceptable
When CPU inference is too slow for the user
Docker vs serverless vs managed model endpoints
Docker is not the only way to ship a model, and it is not always the cheapest. Here is how the three common options compare, and the diagram after the table turns it into a quick decision.
Control
Full
Medium
Low
Ops effort
Highest
Medium
Lowest
Cost model
Pay for running instances
Pay per request, scales to zero
Pay per call or per hour
Cold starts
None once running
Can be seconds on large images
Rare
Best when
Steady traffic, custom or GPU models
Bursty or occasional traffic
Using a hosted foundation model
When is Docker the wrong choice for AI deployment?
Containers are the default for custom models, but they are not free, and there are cases where reaching for Docker first is a mistake.
You are only calling a hosted model. If your feature is a wrapper around a hosted foundation-model API, there is no model to containerize. Call the API from your existing backend and skip the extra layer.
Traffic is rare and unpredictable. A model that serves a handful of requests a day does not justify an always-on container and its idle cost. A serverless container or a managed endpoint that scales to zero is usually cheaper, as long as you can tolerate a cold start.
The image is enormous and latency is critical. GPU images with full CUDA stacks can run several gigabytes. If a cold pull adds seconds you cannot afford, you need warm pools and pre-pulled images, which adds cost and complexity.
Nobody owns the platform. Docker plus Kubernetes is real infrastructure. If you have no one to own scaling, security patching, and cost control, a managed service will serve you better until you do. Being honest about this up front saves a lot of money.
How Redwerk deploys AI models for mid-market teams
Most teams that come to us are not missing the model. They have a working prototype and a stalled path to production, usually because the people who trained the model are not the people who run infrastructure. That gap is exactly what we fill.
Redwerk fields engineers who already know the specific stack, whether that is PyTorch on CUDA, ONNX Runtime, or a Python inference service behind Kubernetes, so we are productive in days rather than spending weeks ramping on your tooling. On one AI development engagement we took over an AI-powered optimization platform mid-build and carried it through to a production launch, handling the containerization, the inference API, and the monitoring the original team had not reached yet.
We also do this without demanding a finished spec. If you know the outcome you need but not the exact architecture, we work it out with you and keep you in the loop the whole way, which matters when the people approving the work are not the ones reading the Dockerfile.
If you are adding a model to an existing product, our guide on adding AI to an existing SaaS covers the product side, and if your stack is .NET, see .NET Core vs .NET Framework for Docker containers for the runtime choice. When you want a senior team that already knows how to ship models to production, here is how Redwerk builds and deploys AI into SaaS products.
FAQ: deploying AI models with Docker
How long does it take to containerize an AI model?
For a single model with a clear inference path, a production-ready containerized service usually takes a senior engineer two to four weeks, including the API, the registry setup, and basic monitoring. A rough proof of concept can run in a day or two, but the gap between “runs in a container” and “safe in production” is where most of the time goes.
Do I need Kubernetes to run AI models in Docker?
No. Kubernetes is worth it when you run many models or need automatic scaling and self-healing. To start, a single container on a managed container service such as AWS App Runner, Google Cloud Run, or Azure Container Apps is simpler and often enough. You can move to Kubernetes later without changing the image.
How big is a typical AI model Docker image?
CPU images built on a slim Python base are often 200 MB to 1 GB. GPU images that include the CUDA toolkit commonly run 3 to 8 GB, and the model weights add on top of that. Multi-stage builds and slim base images are the main levers for keeping the size down.
Can I run GPU inference inside a Docker container?
Yes. With the NVIDIA Container Toolkit installed on the host, containers can access the GPU directly. You start from an NVIDIA CUDA base image, install the GPU build of your framework, and run the container with GPU access enabled. The model code does not change between CPU and GPU.
See how Redwerk took over core development of an AI optimization platform and carried it through to a successful product launch
