Hardware FixRecommendedDevice not working? Your driver may be the problemCheck updates for common hardware issues.Fix DriversOctober DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan Now×
Skip to content
World desk11 min

How to Centralize Docker Logs with Fluentd and Elasticsearch

Learn the complete Docker stdout/stderr to Fluentd to Elasticsearch pipeline, including Compose configuration, Kubernetes DaemonSets, tag routing, buffering, security, and failure diagnosis.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Centralize Docker logs by sending each container’s stdout and stderr through Docker’s Fluentd logging driver to Fluentd, then buffering, routing, and writing the events with Fluentd’s Elasticsearch output plugin. Elasticsearch stores the resulting indexes, while Kibana can provide optional searching and dashboards.

The reliable sequence is container output → Docker Fluentd driver → Fluentd forward input → tag-based routing and buffers → Elasticsearch output → Elasticsearch indexes → optional Kibana. Docker tags identify the source, so they also determine which Fluentd <match> rule receives an event.

As an Amazon Associate I earn from qualifying purchases.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

How the Docker–Fluentd–Elasticsearch pipeline works

A container does not normally write directly to Elasticsearch. Docker captures the process’s stdout and stderr, and the Fluentd logging driver forwards those records to a Fluentd listener. Fluentd receives the records with its forward input, applies routing rules based on the tag, buffers them, and sends them through the fluent-plugin-elasticsearch output. Elasticsearch creates the destination indexes; Kibana is an optional interface for querying and visualizing them.

What each component does

Component Responsibility
Docker container Writes application messages to stdout and stderr.
Docker Fluentd logging driver Hands those records to Fluentd and attaches a tag such as httpd.access.
Fluentd forward input Listens for the driver’s events, normally on TCP/UDP port 24224.
Fluentd routing and buffers Matches tags, queues records during transient failures, and controls delivery timing.
Elasticsearch output plugin Converts Fluentd events into writes to Elasticsearch.
Elasticsearch Stores log indexes that can be searched and retained according to your policy.
Kibana Optionally explores the indexes and presents searches or visualizations.

The Fluentd project describes this combination as an EFK pipeline: “By combining these three tools EFK (Elasticsearch + Fluentd + Kibana) we get a scalable, flexible, easy to use log collection and analytics pipeline.”

#1 Best Overall
Sale
TOZO A1 Wireless Earbuds Bluetooth 5.3 Light Weight in Ear Headphones | IPX5 Waterproof,2 Mics for AI Calls,Immersive Premium Sound Bass Headset with Charging Case,32 EQ via App Black
  • [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
  • [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
  • [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
  • [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
  • [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.

Plan versions, connectivity, and security before configuring

The official Compose example uses Elasticsearch 8.17.1, Elasticsearch Ruby gem 8.19.0, and fluent-plugin-elasticsearch 5.4.3. These are example pins, not performance measurements or universal compatibility requirements. Check the plugin’s supported Elasticsearch client versions against the Elasticsearch version you intend to deploy before fixing your own image tags.

Item Value shown in the Compose example How to use it
Elasticsearch image 8.17.1 Example image pin only; review it before deployment.
Elasticsearch Ruby gem 8.19.0 Example client dependency installed in the Fluentd image.
fluent-plugin-elasticsearch 5.4.3 Example output plugin version; verify compatibility with your client and server.
Fluentd configuration path /fluentd/etc/fluent.conf Default path used by the Fluentd image; mount your configuration there.

The Compose design has four services: an application, Fluentd, Elasticsearch, and Kibana. Fluentd publishes port 24224 for the Docker driver and waits for Elasticsearch to pass a health check before attempting delivery. The application must be able to reach the Fluentd address you put in fluentd-address; test that path from the application container rather than assuming that a host name or localhost has the same meaning in every Docker networking arrangement.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The sample disables Elasticsearch security with xpack.security.enabled=false. That setting is explicitly a testing convenience. A production deployment needs authentication, encrypted transport or HTTPS as appropriate, access control, and durable storage and retention settings that match the sensitivity and volume of your logs.

Build the Docker Compose pipeline

1. Define the application’s Fluentd logging driver

Attach the driver to each service whose stdout and stderr should be centralized. The official representative service is:

services:
  web:
    image: httpd
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

fluentd-address identifies the TCP/UDP listener and tag supplies the routing key. The sample uses localhost:24224; retain that only when it is reachable in your runtime. If Fluentd is on another container, host, or network, replace it with the reachable address and verify the port from the application’s network namespace.

Rank #2
Sale
Smart Watch for Men Women Compatible with iPhone Samsung Android Phone 1.69 inch Full Touch Screen IP68 Waterproof Bluetooh Fitness Tracker Heart Rate/Sleep Monitor (Black)
  • 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
  • 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
  • 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
  • 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
  • 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.

Use a deliberate tag convention instead of one opaque value for every service. A tag such as httpd.access can be matched specifically, while a hierarchy such as docker.web.access can support broader rules. Docker also supports templates including docker.{{.ID}} and docker.{{.Name}}. Put the template inside a <code> or configuration file exactly as required so the braces are not interpreted by your article tooling.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

2. Expose Fluentd’s forward input

Fluentd must listen for the events emitted by Docker’s driver:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

Binding to 0.0.0.0 allows the listener to accept connections on the interfaces exposed by the Fluentd service. Publish or route port 24224 according to your Compose network, firewall, and container placement. The driver and the listener must agree on the port and reachable address.

3. Route events to Elasticsearch and stdout

Install the Elasticsearch output plugin in the Fluentd image, mount the configuration at /fluentd/etc/fluent.conf, and add a match rule. This is the representative pattern from the Compose example:

<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    <buffer>
      flush_interval 1s
    </buffer>
  </store>
  <store>
    @type stdout
  </store>
</match>

The <match *.**> pattern catches the tagged events in this example. The copy directive sends each event to two stores: Elasticsearch for persistence and stdout for live diagnostics. Keeping the stdout store while bringing up the pipeline lets you distinguish “Docker never reached Fluentd” from “Fluentd received the event but could not write Elasticsearch.” Remove or redirect that diagnostic output when it is no longer useful.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #3
New Bee Guitar Cable 10ft Electric Instrument Bass AMP Cord
  • [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
  • [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
  • [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
  • [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
  • [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by

logstash_format true, the fluentd prefix, and the %Y%m%d date format produce date-oriented index names such as fluentd-20261015 for an event on October 15, 2026. The exact index date follows the event and plugin behavior; set index lifecycle and retention policies separately rather than treating daily names as a retention policy.

4. Add readiness and start the services in the right order

Make the Fluentd service depend on an Elasticsearch health check. The health check prevents the collector from racing an Elasticsearch node that has started its process but is not ready to accept writes. Fluentd’s buffer then provides protection against short interruptions after startup, but it is not a substitute for durable capacity planning.

  1. Start Elasticsearch with the security posture appropriate to the environment. If you use the sample’s security-disabled mode, restrict it to an isolated test setup.
  2. Wait for the health check to report readiness.
  3. Start Fluentd with the mounted configuration and confirm that its forward listener is bound to port 24224.
  4. Start the application containers with the Fluentd logging driver enabled.
  5. Generate a known test request or message, watch Fluentd’s stdout copy, and then search the resulting Elasticsearch index through your normal Elasticsearch or Kibana interface.

Use tags as the routing and ownership model

A tag is more than a label. It is the key Fluentd uses to select a <match> rule, so a mismatch can make a healthy connection appear to lose logs. Decide whether your first segment identifies the platform, team, or environment, and keep the pattern stable across Compose projects and clusters.

Fixed tags for simple Compose projects

A fixed value such as httpd.access is easy to read and works well when one service has one log stream. You can add separate tags for error output, workers, or other services and then write targeted match rules instead of sending every event through one broad rule.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Docker templates for container identity

When containers are created and replaced frequently, templates such as docker.{{.ID}} and docker.{{.Name}} preserve container identity in the tag. Ensure your Fluentd match patterns account for the resulting hierarchy. A rule written only for httpd.access will not match a tag beginning with docker..

Rank #4
BSIMB 15.6" Digital Family Calendar & Chore Chart, No Subscription Fees
  • Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
  • Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
  • All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
  • Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
  • Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.

Sub-second timestamps

The logging driver supports fluentd-address and tag, and Fluentd v1 also supports optional sub-second precision. Older implementations did not support sub-second precision, so confirm the behavior of the Docker and Fluentd versions you actually run before depending on millisecond or finer ordering.

Make delivery resilient with buffers and observability

The sample sets a one-second buffer flush interval. That is a configuration example, not a universal tuning value: a shorter interval reduces waiting time but increases write frequency, while a longer interval can improve batching at the cost of latency. Select a buffer type, capacity, retry policy, and storage location that can absorb the Elasticsearch outage you are prepared to tolerate.

  • Keep the Elasticsearch output inside a buffer so temporary connection failures do not immediately discard events.
  • Monitor buffer growth, retry activity, and Fluentd process health; a full buffer is an early warning that the backend or network is falling behind.
  • Use the stdout copy during diagnosis to prove that Fluentd is receiving and routing records.
  • Keep the Elasticsearch readiness check and the Fluentd retry behavior aligned with the time it takes your backend to recover.
  • Decide explicitly how to handle records that cannot be delivered after retries: retain them on durable storage, alert an operator, or follow the loss policy appropriate to the application.

Docker’s logging driver is the handoff point. If the driver cannot connect, records may remain unavailable before Fluentd’s own buffers can help. If Fluentd receives the event but Elasticsearch is unavailable, Fluentd’s output buffer and retry settings are the relevant controls.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Kubernetes: run Fluentd as a node-level DaemonSet

In Kubernetes, the common Fluentd pattern is a DaemonSet so one collector runs on each node and can collect that node’s local container logs. The documented image is quay.io/fluent/fluentd-kubernetes-daemonset. This changes the deployment scope from a four-service Compose project to a cluster-wide, node-level collection layer.

Configure the Elasticsearch destination with environment variables

The Kubernetes DaemonSet image exposes these settings:

Variable Purpose and documented default
FLUENT_ELASTICSEARCH_HOST Hostname of the Elasticsearch service or endpoint. Set it to the destination used by your cluster.
FLUENT_ELASTICSEARCH_PORT Elasticsearch port; the documented default is 9200.
FLUENT_ELASTICSEARCH_SSL_VERIFY Whether TLS certificate verification is enabled; the documented default is true.
FLUENT_ELASTICSEARCH_SSL_VERSION TLS protocol version; the documented default is TLSv1_2.

The defaults assume a Kubernetes service named elasticsearch-logging. Change the host when your Service has another name, lives in another namespace, or is exposed through a different endpoint. Keep certificate verification enabled unless you have a narrowly defined, controlled reason to change it, and provide the trust material required by your cluster’s certificate chain.

Operational differences from Compose

A DaemonSet automatically places collectors on nodes, so scaling collection follows node placement rather than the number of application services in one Compose project. Plan service discovery, secret management, rolling upgrades, and node-local log access as cluster operations. The Elasticsearch destination, authentication, TLS, index naming, buffering, and retention decisions still require the same review as the Compose design.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Secure and maintain the Elasticsearch side

Disabling Elasticsearch security makes an isolated demonstration easier, but it removes the controls that protect logs from unauthorized reading or tampering. For production, define who can write logs, who can search them, how Fluentd authenticates, and how traffic is encrypted. Store credentials and certificates through the secret mechanism appropriate to your deployment rather than placing them in a public Compose file or image.

Persistence and retention deserve separate decisions. Elasticsearch storage must survive the failure you care about, and log indexes should have a deletion or rollover policy that matches disk capacity, legal requirements, and the value of historical records. Daily index names from logstash_format help organization; they do not automatically provide backups, rollover, or deletion.

Compose versus Kubernetes, and self-hosted versus managed Elasticsearch

Choose the deployment model based on scope and operational ownership rather than on the log format. Compose is straightforward for a single host or a small, deliberately bounded environment. Kubernetes is better aligned with node-level collection across a cluster, but it adds DaemonSet lifecycle, service discovery, secret, and upgrade responsibilities.

Decision axis Docker Compose Kubernetes DaemonSet
Deployment scope Services defined in one Compose application or host arrangement. One Fluentd collector per eligible node across the cluster.
Collection placement Explicit Fluentd service reachable by the application containers. Node-level collection using the Kubernetes DaemonSet pattern.
Destination configuration Values in the Fluentd configuration and Compose service definitions. Environment variables such as FLUENT_ELASTICSEARCH_HOST and port/TLS settings.
Secrets and upgrades Managed with the Compose deployment workflow. Managed with Kubernetes secrets, rollout, and cluster upgrade workflows.

Self-hosting Elasticsearch gives you control over topology, storage, retention, and upgrades, but your team owns capacity, patching, availability, authentication, and recovery. A managed Elasticsearch backend shifts much of that infrastructure operation to the provider while leaving you responsible for Fluentd’s plugin compatibility, credentials, network path, index policy, and total log volume. Compare operational ownership, security and TLS controls, persistence and retention, scaling, plugin/server compatibility, and total cost before choosing.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Troubleshoot from the first missing hop

No records appear in Fluentd stdout

  • Confirm the container uses driver: fluentd, not the default Docker driver.
  • Check that fluentd-address points to the address reachable from the application container and that port 24224 is exposed and listening.
  • Verify Fluentd is bound to 0.0.0.0 on the expected port.
  • Generate a fresh stdout or stderr message after the configuration change; old records are not necessarily replayed by the driver.

Fluentd receives events, but the Elasticsearch index stays empty

  • Watch the stdout store to confirm the event reaches the <match> rule.
  • Compare the event tag with the rule pattern. A fixed tag such as httpd.access must match the pattern you wrote; template-generated tags need corresponding patterns.
  • Check the Elasticsearch host and port, backend health, plugin installation, and client/server version compatibility.
  • Inspect buffer and retry messages. A growing buffer indicates that Fluentd cannot complete backend writes.

Writes fail after enabling TLS or security

  • Confirm the endpoint, port, credentials, and certificate trust chain.
  • Keep FLUENT_ELASTICSEARCH_SSL_VERIFY=true unless a controlled exception is documented.
  • For Kubernetes, verify the configured TLS version, whose documented default is TLSv1_2, is accepted by the endpoint.

Logs arrive late or in bursts

Review the buffer flush interval, batch behavior, backend health, and network latency. The sample’s one-second interval is only a starting point. Do not reduce buffering solely to hide a slow or unavailable Elasticsearch service; fix the capacity or connectivity problem and set an outage policy.

Production readiness checklist

  • Every required container has the Fluentd logging driver enabled.
  • The driver address is reachable from the container network namespace.
  • Fluentd’s forward input listens on the intended port and interface.
  • Tag naming is documented, stable, and covered by matching rules.
  • The Elasticsearch output plugin and Elasticsearch client/server versions have been checked together.
  • Buffers, retries, storage, and alerts cover the expected backend outage window.
  • Elasticsearch authentication, TLS, authorization, persistence, and retention are enabled for production.
  • A health check prevents startup races, and a test event has been traced from Docker to an Elasticsearch index.
  • Kubernetes deployments use the DaemonSet image and destination variables appropriate to the cluster’s service name and certificate setup.
  • Kibana access is restricted to the people and roles that should be able to inspect application logs.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Outdated Drivers Are Slowing You DownFree scan - exact matches

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.