NannyML monitors deployed machine-learning models and estimates their performance when ground-truth labels are delayed or unavailable. It detects concept drift and multivariate and univariate data drift, then relates alerts to changes in model performance. Data-quality checks and ranked drift alerts can help teams investigate features associated with performance issues. Users can connect model performance to monetary or other business outcomes. NannyML Cloud offers webhooks for triggering retraining actions and an SDK to automate monitoring-data ingestion. It is available through Azure and AWS marketplaces as SaaS or a managed application; in the managed option, the application and required infrastructure are provisioned within the customer's cloud subscription. For its Azure managed application, NannyML describes encrypted traffic and disks, while noting automated backups were still being implemented and advising customers to safeguard model inputs and outputs elsewhere. The self-managed open-source library is free and installs with pip or conda. Cloud UI uploads of local datasets are limited to files smaller than 100 MB. Listed plans include Starter at 399.00 USD per month and Scale at 999.00 USD per month, each with a 30-day free trial; Enterprise pricing is on request.
Who it is for
It suits teams monitoring deployed machine-learning models, especially when labels arrive late or are unavailable. The self-managed library may suit users seeking a free option, while Cloud plans offer managed deployment choices.
What is good
- Estimates performance without timely ground-truth labels
- Tracks concept and data drift
- Connects performance to business outcomes
- Cloud offers webhooks and automated ingestion SDK
- Free self-managed open-source library
What to know first
- Direct Cloud UI uploads must be under 100 MB
- Azure managed-application automated backups were still in progress
- Starter lists a two-model limit
Verdict
NannyML combines drift monitoring with estimates of model performance and business impact. Check the deployment and backup details, as well as plan limits, against your monitoring requirements.
NannyML plans and pricing
All plansCompared on model monitoring software
- Free plan
- Yes
- Paid from
- $99/mo
- Drift monitoring
- Yes
- Model performance metrics
- Yes
- Data quality checks
- Yes
- Alert channels
- email, Slack, webhook
- Included model limit
- 2 models
