dagster-slurm

What dagster-slurm is

dagster-slurm connects Dagster assets with Slurm compute, allowing assets to run locally during development and on a Slurm cluster in production. Dagster dependencies and lineage can link data ingestion, cluster computation, and downstream aggregation or publishing. In Slurm mode, the software transfers the current checkout payload and Pixi-locked environment, submits jobs, and returns job status and logs to Dagster. The default is one Slurm job per asset. Dagster can display Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours for asset runs. Local and Slurm are the stable execution modes. A Ray launcher supports distributed execution, GPU scheduling, and optional reuse of allocations within a run; heterogeneous Slurm jobs are experimental, and allocation reuse currently focuses on Ray assets. Spark integration is marked as work in progress. Cluster connections use SSH, with key-based authentication, jump hosts, and configurable host-key checking. Runtime environment variables can keep connection details and credentials outside the repository. Jobs use Pixi environments, and production expects a pre-deployed environment bundle on the cluster. The open-source software is free and self-hosted, with Docker Compose available for testing cluster workflows.

Who it suits

dagster-slurm suits Dagster users who develop on a laptop and need cluster execution for compute-heavy assets, such as model training or hyper-parameter sweeps. It is also for teams that want Slurm job status and resource details visible alongside Dagster asset runs.

Limits

The free Open-source software plan requires Dagster and Slurm; cluster execution also needs a Slurm cluster. Production mode expects a pre-deployed environment bundle, while Spark integration is work in progress and heterogeneous Slurm jobs are experimental.

Getting started

1. Visit the dagster-slurm website and follow its documentation.
2. Use the free Open-source software plan; Dagster and Slurm are required.
3. Set up a local environment following the project's Python 3.12 development conventions.
4. Run assets locally without SSH or Slurm, or use the provided Docker Compose Slurm stack to test cluster workflows.
5. For production cluster execution, configure SSH access and deploy the required Pixi environment bundle on the cluster.

Verdict

Choose dagster-slurm if your Dagster assets need local development and Slurm cluster execution with run details surfaced in Dagster. Look elsewhere if you need a finished Spark integration or stable heterogeneous Slurm jobs.