Pulsar is an open-source, self-hosted tool for monitoring machine learning models in production. It tracks model drift, concept drift, and model performance, using training data as the baseline for its metrics. An SDK and a containerised ingestion service bring model inputs and outputs into a time series database, with support for historical and real-time data. Its current database integration is InfluxDB, and the product is built to handle large volumes of ingested data without degraded performance. A scheduler checks drift and can trigger standard or customised alerts when metrics reach set thresholds. Pulsar includes Grafana to display drift measures over selected time ranges and user-defined features. Drift measures include Kullback-Leibler divergence, Kolmogorov-Smirnov, Levene, and Wasserstein tests, along with other statistical tests. Performance measures include accuracy, recall, F1 score, Brier score, mean squared error, and R2 score. The homepage displays deployment integration logos for Flask, FastAPI, TensorFlow Extended, Seldon, AWS, Azure, Google Cloud, and Grafana. Pulsar is described as a community edition. Its help center offers a quick-start guide, guides for creating and managing Pulses, contact support, expert help, and a user community. Rocket Science Development is identified as its maker.
Who it is for
Pulsar suits software engineers, data scientists, and machine learning engineers who need to monitor models after deployment. It is a fit for teams that can use a self-hosted deployment and want drift and model performance metrics, threshold alerts, and a Grafana dashboard.
What is good
- Free, open-source community edition.
- Supports historical and real-time model input and output data.
- Tracks drift and performance against training data.
- Can alert when performance metrics reach thresholds.
- Includes a Grafana dashboard for user-defined features and time ranges.
- Offers an SDK and containerised ingestion service.
What to know first
- Self-hosted deployment requires integration into a user's machine.
- The current time series database integration is InfluxDB.
Verdict
Choose Pulsar if you need a free, open-source option for monitoring production model drift and performance, with alerts and Grafana visualisation. It is best suited to teams able to deploy and integrate a self-hosted service; its listed current database integration is InfluxDB.
Get started with Pulsar
- Visit the Pulsar website.
- Choose the self-hosted deployment route.
- Use the SDK or containerised data ingestion service to integrate Pulsar into your machine.
- Connect model inputs and outputs to its InfluxDB integration.
- Use the help center's quick-start guide to begin.
Questions about Pulsar
Is Pulsar free?
Yes. It is described as a free, open-source community edition.
Where can Pulsar run?
Its listed deployment option is self-hosted, with an SDK and a containerised data ingestion service.
Which database does it integrate with?
Its current time series database integration is InfluxDB.
What does Pulsar monitor?
It monitors model drift, concept drift, and model performance, using training data as a baseline for metrics.
What metrics can it track?
Listed drift measures include Kullback-Leibler divergence, Kolmogorov-Smirnov, Levene, and Wasserstein tests. Performance measures include accuracy, recall, F1 score, Brier score, mean squared error, and R2 score.
Who makes Pulsar?
Rocket Science Development is identified as Pulsar.ml's maker.
Compared on model monitoring software
- Drift monitoring
- Yes
- Model performance metrics
- Yes
- Deployment options
- self-hosted
