How to Deploy and Run AI Models on Intel Gaudi Accelerators

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

To deploy an AI model on an Intel Gaudi accelerator, first match the Gaudi2 or Gaudi3 hardware, host driver, firmware, SynapseAI release, operating system, Python, framework, and container image in Intel’s support matrix. Then verify the node with hl-smi, make sure the habanalabs-container-runtime is registered with Docker, and start the matching Intel Gaudi image with the Habana runtime. PyTorch and TensorFlow workloads run through the Gaudi software stack; distributed training, DeepSpeed, vLLM, SGLang, and quantized inference have separate documented paths. You can use a compatible server or hosted capacity such as Intel Tiber AI Cloud or an Amazon EC2 Gaudi instance, but region, quota, release, and networking availability must be checked before you build the deployment.

What does deploying a model on Intel Gaudi involve?

Gaudi deployment is a version-controlled hardware and software integration task rather than a matter of installing a generic GPU package. The accelerator, Linux kernel, host driver, firmware, SynapseAI software, framework build, Python version, and container image must be a supported combination. A container that starts successfully with an incompatible driver can still fail later when it imports PyTorch, initializes a device, compiles a graph, or launches a distributed job.

#	Preview	Product	Price
1		AI Accelerator Kit, M.2 HAT Plus +13TOPS Module | For Edge Inference	$460.06	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Intel’s Gaudi documentation covers Gaudi2 and Gaudi3, model migration, PyTorch training and inference, TensorFlow, distributed training, DeepSpeed, vLLM, SGLang, and quantized FP8 and INT4 workflows. Treat the release-specific support matrix as the authority for exact versions. Documentation pages and examples change with each SynapseAI release, so an old command or image tag should not be copied without checking its release context.

• Hardware: Gaudi2 or Gaudi3 devices in a supported server, virtual machine, or cloud instance.
• Host software: Intel Gaudi driver and firmware compatible with the selected SynapseAI release.
• Execution layer: Docker with the Habana container runtime, or a supported Kubernetes or Slurm integration.
• Workload software: The framework, plugin, model code, and Python version listed for that release.
• Operations: Device visibility, networking, shared memory, storage, and permissions configured for the workload.

There is no single current performance number that represents every model, batch size, precision, and parallelism choice. Benchmark the exact workload after the functional deployment is stable.

#1 Best Overall

AI Accelerator Kit, M.2 HAT Plus +13TOPS Module | For Edge Inference

• Performance: Embedded single-board computer equipped with a quad-core 64-bit processor and supporting the Linux operating system; suitable for edge computing, the Internet of Things (IoT), and other control applications
• Specifications: The development board offers multiple configuration options, featuring LPDDR4 memory and eMMC flash storage, allowing users to select the configuration that best suits their needs
• Design: The industrial AI module features a compact design with low power consumption and supports AI acceleration, making it suitable for deep learning and machine vision
• Reliability: The motherboard supports a wide temperature range, ensuring long-term, continuous, stable, and reliable operation in industrial environments
• Applications: Widely used in embedded development, smart gateways, AI vision, and industrial automation

$460.06

Buy on Amazon

How do I choose Gaudi2, Gaudi3, or hosted capacity?

Choose the accelerator generation and access model that your support matrix, capacity plan, and software release can actually support. Gaudi2 and Gaudi3 are not interchangeable labels for the same environment: use the image and driver path for the generation installed in the node.

Gaudi2 versus Gaudi3 in a deployment decision

For hosted access, Intel’s setup guidance names Intel Tiber AI Cloud and Amazon EC2 DL1 as access paths. Confirm the target region, account quota, instance or VM shape, networking, attached storage, maintenance policy, and preinstalled software before committing. Availability and pricing can vary by region and date, and a provider may expose only specific Gaudi software releases.

When is a self-managed server appropriate?

Self-managed capacity makes sense when you need persistent local storage, custom networking, predictable placement, or control over kernel and orchestration settings. It also makes you responsible for driver and firmware lifecycle, container-runtime registration, host security, cooling and power, and replacement capacity. A hosted node is faster to start when the provider supplies a supported driver image, but you still need to verify that its release matches your model stack.

Which operating systems, Python versions, and tools are supported?

The support matrix, not a generic PyTorch compatibility list, determines the valid combination. Intel’s displayed Gaudi3 configuration for Intel Gaudi software 1.24.1 lists the following values:

These values are not permission to mix any Python 3.10–3.12 environment with any image. The matrix ties each value to a particular operating system and software release. Record the matrix row you selected as part of the deployment manifest so another operator can reproduce it.

What does SynapseAI provide?

SynapseAI is Intel’s software stack for Gaudi devices. Intel describes its optimized framework containers as including the required dependencies and a complete SynapseAI installation, with integration and validation against recent TensorFlow and PyTorch versions. Using that container path avoids silently substituting a CPU-only wheel or a generic accelerator build. If you build from source, you must reproduce the equivalent compiler, plugin, library, and runtime versions yourself.

How do I verify a Gaudi node before starting a container?

Verify the host first. Device discovery and runtime registration failures are much easier to diagnose before model packages and checkpoints are involved.

1. Confirm that the machine is a Gaudi node. On the host, run hl-smi. If the command is unavailable or shows no devices, check that you are connected to the intended server or virtual machine and that the host driver is installed.
2. Read the installed driver. Run hl-smi | grep Driver. Compare the reported driver with the support-matrix row for the SynapseAI release and image you selected.
3. Check permissions. The account launching Docker must be allowed to access the device files and the container engine. A cloud instance can have a driver installed while a restricted user still cannot open the device.
4. Check the Habana container runtime. Confirm that habanalabs-container-runtime is installed and registered with Docker. If it is missing, install the package and configure the container service according to Intel’s release instructions, then restart or reload the container engine as required.
5. Record firmware and host details. Keep the driver output, operating-system release, Docker version, runtime configuration, and selected image tag with the deployment record.

A healthy hl-smi result proves that the host can see the accelerator; it does not prove that a particular PyTorch, vLLM, or model package is compatible. The image and driver still have to match the matrix.

How do I start an Intel Gaudi Docker container?

Use an image tag selected from the support matrix and pass the Habana runtime explicitly. Intel’s example includes device visibility, an MPI shared-copy setting, elevated scheduling capability, host networking, and host IPC.

1. Select the image. Replace the variable in the commands below with the complete image reference and tag listed for your Gaudi generation, SynapseAI release, operating system, and framework. Do not reuse an old tag merely because it appears in a blog post.
2. Pull the image. Run docker pull {{GAUDI_IMAGE_FROM_SUPPORT_MATRIX}} using the registry and tag specified by Intel for that release.
3. Start an interactive shell. Use the following pattern:

docker run --runtime=habana 
  -e HABANA_VISIBLE_DEVICES=all 
  -e OMPI_MCA_btl_vader_single_copy_mechanism=none 
  --cap-add=sys_nice 
  --net=host 
  --ipc=host 
  {{GAUDI_IMAGE_FROM_SUPPORT_MATRIX}} bash

4. Check visibility inside the container. Run hl-smi again and import the framework supplied by the image. A device list from the host but not from the container usually indicates runtime registration, permissions, or an image mismatch.
5. Mount code and data deliberately. Add read-only mounts for source and checkpoints where possible, and a writable mount for logs and outputs. Ensure the container user can read the checkpoint files without making the entire host filesystem writable.

--net=host and --ipc=host simplify high-performance communication and distributed jobs, but they also broaden the container’s access to host networking and shared memory. Use the security policy appropriate to your environment and remove flags only after confirming that the workload’s communication requirements still work.

How do I run PyTorch or TensorFlow training on Gaudi?

Start with Intel’s framework container and its Gaudi examples, then adapt the model code. The container supplies the Gaudi integration; your training script still needs the framework-specific device selection, data pipeline, checkpointing, and distributed configuration expected by the release.

PyTorch workflow

1. Launch the PyTorch image selected from the matrix and verify that the Gaudi device is visible inside the container.
2. Use the Gaudi-enabled PyTorch package and the model example documented for your SynapseAI release. Do not install an unrelated upstream wheel over the container’s tested package set.
3. Run a small, deterministic smoke test: load one batch, execute a forward and backward pass, and save a checkpoint. This isolates device and operator issues before a long job.
4. Scale batch size, sequence length, precision, and data-loader workers one variable at a time. Record those settings with throughput and memory observations.
5. For multi-device training, use the release’s documented distributed launcher and environment variables. Confirm host networking, IPC, device visibility, and rendezvous settings before increasing node count.

TensorFlow workflow

TensorFlow 2 is listed among the supported frameworks in Intel’s Gaudi2 white paper, while current release documentation determines the supported TensorFlow build for a particular Gaudi2 or Gaudi3 image. Use the supplied TensorFlow container and examples rather than assuming that a package from the public Python index contains the Gaudi integration. Test one device, then validate mirrored or distributed execution with the release-specific instructions.

DeepSpeed and distributed training

Intel documents DeepSpeed and distributed-training paths for Gaudi. They require more than setting a device count: the launcher, process placement, hostnames, ports, shared memory, and collective-communication settings must agree on every node. Begin with the documented single-node or smallest multi-node example, then change topology only after the baseline completes and produces a valid checkpoint.

How can I serve a model with vLLM on Gaudi?

Use the vLLM Hardware Plugin for Intel Gaudi and a release-matched installation route. Intel’s quickstart documents Docker Compose, a Dockerfile path, and source installation, with ready-to-run images, custom runtime parameters, benchmarking guidance, and validated model families including LLaMA, Mistral, and Qwen. Model and feature support can change between plugin tags, so check the tag that matches your installed Gaudi software.

Docker Compose route

The Compose route is convenient when the plugin documentation provides a complete service definition for your release. Copy the release-matched Compose configuration, set the model identifier and checkpoint volume, expose the serving port, and preserve the Habana runtime and device-visibility settings. Validate the image tag before starting; a generic vLLM image may not include the Gaudi plugin.

Dockerfile route

Use the documented Gaudi base image and Dockerfile instructions when you need to add tokenizer files, an internal model package, monitoring libraries, or organization-specific startup logic. Keep the base image, Python version, plugin version, and framework packages aligned with the support matrix. Build once, record the resulting image digest, and promote that immutable image between environments.

Source-install route

A source installation is appropriate when you need a plugin change that is not in a ready-made image or when your build system requires fully controlled dependencies. Follow the plugin’s supported Linux, Python, and Gaudi combination exactly. Pin the source revision and every resulting package, then run the plugin’s smoke test before loading a large checkpoint.

Model and serving checks

• Confirm that the model family and requested features are listed for the plugin release you selected.
• Start with a small prompt and short output to verify tokenizer, weights, device initialization, and HTTP or OpenAI-compatible API behavior.
• Set sequence length, batch limits, precision, and memory-related parameters explicitly rather than relying on defaults from a generic vLLM guide.
• Benchmark representative prompts after warm-up. Record latency distribution, throughput, concurrent requests, context length, and precision; a single tokens-per-second figure is not portable across workloads.

Should I use Docker, Kubernetes, or Slurm?

Docker is the simplest path for a single node and the reference container examples. Kubernetes or Slurm becomes useful when you need scheduling, quotas, multi-node placement, or queue management, but each layer adds configuration that must remain compatible with the Gaudi runtime.

The displayed Gaudi3 software 1.24.1 matrix includes Kubernetes 1.34, 1.35, and 1.36. That does not make every Kubernetes version valid for every SynapseAI release, so upgrade the cluster only after checking the matrix row for the image you will run.

How do I troubleshoot common Gaudi deployment failures?

hl-smi shows no devices

• Confirm that the machine is actually a Gaudi node and that the instance or VM was allocated with accelerators.
• Check the host driver and firmware, then inspect permissions for the user and device files.
• Run hl-smi outside the container before debugging the model or framework.

Docker reports “Unknown runtime specified habana”

The container engine does not know the Habana runtime. Install and register habanalabs-container-runtime, configure Docker as required by the Intel release instructions, restart or reload the service, and retry with --runtime=habana. Confirm that the runtime name in Docker’s configuration exactly matches the command.

The image starts but Python packages or operators do not match

Compare the host driver, firmware, SynapseAI release, operating system, Python version, framework build, and image tag against one support-matrix row. Remove ad-hoc package upgrades that replaced tested components, then recreate the container from the correct image. If you built an image yourself, compare its dependency lockfile with the release’s base image.

A distributed job hangs or exits during initialization

Check host networking, IPC size, device visibility, hostname resolution, rendezvous ports, rank assignments, and firewall rules on every node. Use the smallest documented distributed example first. A process that can see one device locally may still fail when collective communication cannot reach its peers.

vLLM fails before serving requests

Use the vLLM Gaudi plugin release and container or source-install route that matches the installed Gaudi software. Generic vLLM instructions can omit the Gaudi plugin or select unsupported Python and framework versions. Reduce the model and context settings for the first launch, inspect plugin logs, and then restore production parameters one at a time.

Training runs out of memory or slows unexpectedly

Check sequence length, micro-batch size, gradient accumulation, activation checkpointing, precision, and data-loader behavior before changing the hardware. For serving, inspect concurrent requests, KV-cache limits, context length, and quantization support for the plugin tag. Compare only workloads with the same model revision, prompt mix, precision, and latency target.

What should be in a reproducible Gaudi deployment record?

Keep a machine-readable record alongside the model and container so that a replacement node can be validated quickly.

• Gaudi generation, device count, server or cloud instance type, and region.
• Operating-system edition and point release.
• Host driver, firmware, Docker, and habanalabs-container-runtime versions.
• SynapseAI release, image reference and digest, Python version, and framework build.
• Model revision, tokenizer revision, checkpoint location, precision, maximum sequence length, and serving or training parameters.
• Launcher configuration, node list, network and port settings, mounts, and security exceptions.
• Smoke-test output, benchmark methodology, logs, and the exact command used to start the workload.

This record separates an application regression from a host change. It also makes it possible to roll back an image or SynapseAI release without guessing which component changed.

What is the safest deployment sequence?

1. Select access: choose Gaudi2 or Gaudi3 on a compatible server, Intel Tiber AI Cloud, or an Amazon EC2 Gaudi instance, and verify region, quota, storage, networking, and provider release.
2. Select a matrix row: pin hardware generation, SynapseAI, operating system, Python, Docker or Kubernetes, framework, and image.
3. Validate the host: run hl-smi | grep Driver, verify permissions and firmware, and register the Habana runtime.
4. Run a smoke container: use the matching image with --runtime=habana, device visibility, host IPC, and the documented networking settings.
5. Run one device: load a small model or batch and save a test output or checkpoint.
6. Scale deliberately: add devices, nodes, longer contexts, larger batches, or production concurrency only after the baseline succeeds.
7. Freeze and observe: record image digests and versions, collect logs and device telemetry, and benchmark the workload that matters to your users.

Frequently Asked Questions

Can I move a Gaudi checkpoint between Gaudi2 and Gaudi3?

Checkpoint files are generally an application and framework concern, but optimizer state, fused operators, precision choices, and custom kernels can make a restart release-dependent. Test loading and resuming in the target image, and keep a framework-level export or conversion path when you need portability.

Do Gaudi containers require internet access at runtime?

Not necessarily. Once the image, model weights, tokenizer, and Python dependencies are available locally, an inference service can run without outbound access. Air-gapped deployments should mirror the approved image and artifacts, verify checksums, and disable startup code that attempts to download missing files.

How should I expose a Gaudi inference service securely?

Put the service behind your normal API gateway or private network, restrict inbound ports, and avoid exposing host networking directly to untrusted clients. Apply authentication, request limits, logging, and separate read-only model mounts even when the container example uses host networking for performance.

What should I benchmark before choosing a cloud region?

Measure end-to-end latency and throughput with the actual model, context-length distribution, concurrency, precision, cold-start behavior, and storage location. Include data-transfer time and quota limits; accelerator availability alone does not determine the service’s usable capacity.

Quick Recap

Bestseller No. 1

AI Accelerator Kit, M.2 HAT Plus +13TOPS Module | For Edge Inference

$460.06

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AI acceleratorsGaudi2Gaudi3Intel GaudiMLOpsPyTorchTensorFlowvLLM