Put context in the Prometheus alert rule first. Use labels for short, stable, structured values that Alertmanager may use for routing or grouping; use annotations for human-facing explanations such as summaries, descriptions, dashboards, and runbook links. Use an Alertmanager webhook to deliver that already-processed context to an HTTP system that lacks a native integration, or to transform it for a receiver’s contract. A webhook is normally a transport and integration decision, not a second authoring location for alert context.
Alertmanager receives both labels and annotations, groups and routes alerts, deduplicates notifications, and places the resulting data in its notification payload. Author reusable meaning in the rule, then choose the receiver—native integration or webhook—that can present or transform it correctly.
As an Amazon Associate I earn from qualifying purchases.
What you are deciding
There are two separate questions:
- Where should the context be authored? In the Prometheus alert rule, as labels and annotations.
- How should the processed alert be delivered? Through an Alertmanager receiver, such as a built-in integration or the generic webhook receiver.
Keeping those questions separate prevents a common design mistake: putting all useful text into a webhook template even though the same context belongs to the alert itself. Rule-level context is available to every receiver and to every later notification. Webhook-specific formatting should be reserved for the external system’s contract.
Labels and annotations have different jobs
Prometheus alerting rules can define additional labels and annotations. Both can contain templates, but they carry different kinds of information.
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
| Field | Best use | Operational effect | Typical examples |
|---|---|---|---|
| Labels | Compact, structured metadata | Available to Alertmanager for route matching, grouping, inhibition, and notification data | severity: critical, team: payments, service: checkout, environment: production |
| Annotations | Longer, explanatory information for people or downstream messages | Included in notifications but normally not used as routing or grouping identity | summary, description, dashboard URL, runbook URL, impact explanation |
A useful test is whether a value should identify an alert or explain it. If changing the value should change which route or group receives the alert, it is a candidate label. If it is prose that helps an operator understand or repair the condition, it belongs in an annotation.
Why long prose does not belong in labels
Labels are part of an alert’s identity. Values that change frequently, contain paragraphs, or include volatile measurements can create many distinct label sets. That makes grouping and deduplication less predictable and can increase the amount of metadata Alertmanager must handle. Keep labels short and stable; put detailed text in annotations.
Why annotations are not a routing substitute
Annotations are intended for additional information. Do not rely on a changing description, current metric value, or runbook sentence to select a route. Put the routing key—such as team, severity, or service—in a label and let the annotation explain the incident.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How context moves through the alerting pipeline
- Prometheus evaluates the alert expression.
- When the expression is firing, Prometheus evaluates templates in the rule’s label and annotation fields and creates an alert object.
- Prometheus sends that object, including its labels and annotations, to Alertmanager.
- Alertmanager applies grouping, routing, inhibition, deduplication, and receiver timing.
- The selected receiver renders a notification. A webhook receives a structured HTTP payload containing group-level and per-alert data.
This order matters. A label or annotation is created during Prometheus rule evaluation; a notification template is rendered later by Alertmanager. A value that is available in one stage is not automatically available under the same variable name in the other.
Author reusable context in the Prometheus rule
Start with the rule because every receiver can then use the same metadata. The following pattern keeps routing fields short and explanatory fields readable:
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
groups:
- name: checkout.rules
rules:
- alert: CheckoutErrorRateHigh
expr: rate(http_requests_total{service='checkout',status=~'5..'}[5m]) > 0.05
for: 10m
labels:
severity: critical
team: payments
service: checkout
environment: production
annotations:
summary: Checkout is returning too many server errors
description: Error rate for {{ $labels.instance }} is {{ printf "%.2f" $value }} per second.
runbook: https://runbooks.example.invalid/checkout-error-rateThe label names and values in this example are deliberately compact and suitable for matching. The annotations carry sentences and an operational link. Prometheus supports templates in both fields. Rule templates can use alert labels and the evaluated value, so a description can include the affected instance or the value that triggered the condition.
Use templates sparingly in the rule. Prometheus evaluates alert-field templates during every rule iteration for each firing alert. For complicated presentation, link to a console or runbook rather than constructing a large block of text in the rule.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Keep identity stable
- Use bounded values such as a fixed severity vocabulary, a service name, an environment, and an owning team.
- Do not put request IDs, timestamps, full error messages, or unbounded user input in labels.
- Use a label only when Alertmanager or another consumer needs to match, group, or filter on it.
- Put current measurements and explanatory prose in annotations.
How labels affect routing and grouping
Alertmanager uses alert labels when it evaluates routes and builds groups. A route can match a label such as team or severity; grouping can combine alerts that share the configured grouping labels. Consequently, adding, removing, or changing a label can alter the destination, the group composition, and the deduplication identity.
Annotations normally travel with the alert but do not define its routing identity. If two alerts have the same grouping labels but different descriptions, Alertmanager can still place them in one notification group; the receiver may see the differing annotations on the individual alerts.
Designing labels for predictable groups
- Define a small vocabulary and document allowed values.
- Use the same label key for the same concept across rules. For example, do not alternate between
owner,team, andresponsible_groupfor one routing dimension. - Do not encode a unique incident number or changing metric value in a label.
- Review route and grouping behavior after adding a new label; a seemingly harmless label can split notifications into many groups.
When a webhook is the right choice
The generic webhook receiver is appropriate when the destination does not have a native Alertmanager integration or when another system needs the grouped alert data over HTTP. Prometheus documentation describes the purpose directly: “For notification mechanisms not natively supported by the Alertmanager, the webhook receiver allows for integration.”
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
Use a webhook when an external incident, automation, ticketing, or internal service needs to receive Alertmanager’s structured result. Do not use it merely because you need a description; annotations already provide that description to native receivers and to the webhook payload.
| Decision axis | Native rule labels and annotations | Alertmanager webhook |
|---|---|---|
| Main role | Author context with the alert rule | Deliver grouped alert data to an HTTP endpoint and optionally transform it |
| Evaluation stage | Prometheus rule evaluation | Alertmanager notification rendering and delivery |
| Routing and grouping | Labels can participate in route matching and grouping | Receives the groups after Alertmanager has processed them |
| Typical content | Severity, team, service, summary, description, runbook link | Data required by an external receiver’s API or incident schema |
| Operational burden | Maintain stable labels and useful annotations | Maintain endpoint availability, authentication, HTTP settings, payload validity, and downstream handling |
What the standard webhook payload contains
The standard webhook body represents Alertmanager’s view of a notification group. It includes group-level fields and an array of individual alerts. The exact receiver contract should be checked against the Alertmanager version you operate, because the latest documentation is mutable.
| Payload area | Purpose |
|---|---|
groupLabels |
Labels that identify the notification group |
commonLabels |
Labels shared by all alerts in the group |
commonAnnotations |
Annotations shared by all alerts in the group |
alerts |
Per-alert objects, each with its own labels and annotations and its firing or resolved state |
| Group and delivery metadata | Information such as the receiver name, notification status, external URL, and payload version |
A simplified body can look like this:
{
"version": "4",
"status": "firing",
"receiver": "operations-webhook",
"groupLabels": {"alertname": "CheckoutErrorRateHigh", "team": "payments"},
"commonLabels": {"severity": "critical", "service": "checkout"},
"commonAnnotations": {"summary": "Checkout is returning too many server errors"},
"alerts": [
{
"status": "firing",
"labels": {"instance": "checkout-01", "severity": "critical"},
"annotations": {"description": "Error rate is above the configured threshold"}
}
]
}Do not assume that a value in commonAnnotations is present for every group. It is common only when the annotation exists and has the same value across the alerts in that group. For alert-specific text, read the annotation on each object in alerts.
Group fields versus per-alert fields
A receiver that creates one incident per notification may use group fields for the title and iterate over alerts for details. A receiver that creates one record per alert should use each alert’s labels and annotations instead of treating common fields as complete. Designing this distinction explicitly prevents lost context when one group contains alerts with different instances, descriptions, or runbook links.
Configure a webhook without moving context out of the rule
Define the receiver in Alertmanager and point it at the endpoint’s URL or a file containing that URL. A file avoids putting the endpoint directly in the main configuration and can simplify secret-management workflows:
Recommended Free Tools
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
receivers:
- name: operations-webhook
webhook_configs:
- url_file: /etc/alertmanager/webhook-url
send_resolved: true
max_alerts: 0Attach the receiver to a route that matches the labels you designed in the rule. Keep the route selection in labels and leave the human-readable text in annotations. Enable resolved notifications only if the downstream system understands a resolved state; otherwise it may create confusing follow-up records.
Endpoint responsibilities
- Accept the HTTP method and content type emitted by Alertmanager.
- Authenticate and authorize requests using the HTTP settings supported by your Alertmanager configuration and your receiver.
- Handle retries or duplicate deliveries safely. Alertmanager deduplicates notifications according to its grouping and timing rules, but a receiver should still be idempotent.
- Return an appropriate success response only after the payload is accepted or durably queued.
- Log validation failures without logging credentials or unnecessarily sensitive annotation content.
Use native receiver templates before writing a custom webhook payload
When a built-in receiver can express the desired message, render the annotations with Alertmanager’s notification templates. Those templates expose group data, common annotations, and each alert’s annotations. This keeps the payload format maintained by the integration and avoids making your own JSON contract.
A notification template can iterate over alerts and print each alert’s explanatory fields:
{{ range .Alerts }}
{{ .Labels.alertname }}: {{ .Annotations.summary }}
{{ .Annotations.description }}
{{ end }}The exact fields available depend on the notification template context, not on the Prometheus rule context. Treat the two stages as separate APIs even though both use Go-style template syntax.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePrometheus rule templates and Alertmanager templates are different contexts
In a Prometheus rule, expressions such as {{ $labels.instance }} and {{ $value }} refer to the alert evaluation. In an Alertmanager notification template, you normally access the notification data through objects such as .Alerts, .CommonLabels, and .CommonAnnotations. Copying an expression from one file into the other can produce an empty value or a template error.
| Need | Prometheus rule field | Alertmanager notification |
|---|---|---|
| Include the evaluated alert value | Use the rule template variable available during evaluation, such as $value |
Use a value carried in the alert data; do not expect the rule variable to exist |
| Read a label | Use the rule’s alert-label template context | Read the label from the alert or common-label object in the notification context |
| Print a description | Set the rule’s annotation | Render the annotation from the alert or common-annotation object |
Keep rule templates lightweight and put presentation logic in the receiver template where possible. This reduces repeated work during rule evaluation and lets different receivers format the same annotations for their own channels.
Custom webhook payloads are an advanced option
Alertmanager can render a custom webhook payload when the standard structure does not match the receiver’s contract. This transfers responsibility to you: Alertmanager does not validate that the rendered result is valid JSON or that it conforms to the receiver’s schema.
Checks required for a custom payload
- Define the receiver’s required fields, types, nesting, and handling of firing and resolved alerts.
- Escape annotation text correctly for JSON strings. Quotes, backslashes, and line breaks in an operator-written description can otherwise invalidate the body.
- Test empty and missing annotations, multiple alerts in one group, mixed firing and resolved states, and non-ASCII text.
- Validate the rendered body against the receiver schema before enabling production delivery.
- Version the contract so a receiver change does not silently discard alert context.
Prefer the standard webhook structure unless the external system explicitly requires a different shape. Every custom field is another compatibility obligation during Alertmanager upgrades and rule changes.
Free tools Windows power users keep installed
One-click scans. No signup required.
Common failure modes and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Alerts reach the wrong team | A routing label is missing, misspelled, or assigned a value outside the route matcher | Inspect the alert’s labels in Prometheus and align the label key and allowed values with the Alertmanager route |
| One incident contains unrelated alerts | Grouping labels are too broad | Review the configured grouping dimensions and add only stable labels that should define a notification group |
| Notifications are split into many small groups | A volatile or high-cardinality value was added as a label | Move the value to an annotation and remove it from routing and grouping identity |
| The webhook has a title but no detailed explanation | The receiver reads only common annotations or ignores per-alert annotations | Read each object’s annotations inside the alerts array and handle common fields as optional conveniences |
| A template prints nothing | A Prometheus expression was used in an Alertmanager template, or the annotation is absent | Use the correct stage’s data model and provide a fallback for missing annotations |
| The endpoint rejects the request as malformed JSON | A custom payload did not escape rendered text or produced an unexpected value type | Validate the rendered body with representative alert text and simplify the payload or return to the standard structure |
| Resolved alerts create duplicate incidents | The receiver does not distinguish firing from resolved |
Configure send_resolved deliberately and make downstream state transitions idempotent |
A practical decision framework
- Need routing, grouping, inhibition, or filtering? Add a short, stable label to the rule.
- Need a sentence, current value, dashboard, or runbook? Add an annotation to the rule.
- Does a built-in receiver support the destination? Use it and render the annotations with its notification template.
- Is the destination unsupported or does it require an HTTP contract? Use the generic webhook receiver.
- Does the destination accept the standard payload? Keep the standard structure.
- Does it require a different schema? Use a custom payload only after validating JSON, edge cases, authentication, and duplicate handling.
Bottom line
Native labels and annotations are the source of truth for alert context: labels identify and classify the alert, while annotations explain it. Alertmanager then groups and routes that metadata. A webhook is the delivery and transformation layer for systems that need the processed alert data over HTTP. Author context once in the rule, keep labels stable, keep prose in annotations, and customize the webhook body only when the receiver’s contract genuinely requires it.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
