Free tools Windows power users keep installed
One-click scans. No signup required.
To install Apache Kafka on a Linux VPS, install a supported Java runtime, download and verify a pinned Kafka binary, initialize its KRaft metadata storage, start the broker, and validate it with a test topic. A default broker is appropriate only for a private test: Apache Kafka disables security by default, so a public VPS must use deliberate listeners, authentication, encryption, authorization, and firewall rules before clients can connect.
Need a server to try this on? Antihost offers Linux VPS plans with NVMe storage, full root access and a 14-day free trial.
This guide uses the official binary workflow and a single-node KRaft setup suitable for development or a controlled test server. Production availability requires multiple hosts, a redundant controller quorum, persistent storage, and capacity planning based on your workload.
As an Amazon Associate I earn from qualifying purchases.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat you need before installing Kafka
A supported 64-bit Linux VPS
Use a supported 64-bit Linux distribution with persistent storage. Kafka writes both message data and metadata to disk, so an ephemeral VPS disk is unsuitable. Give the server a DNS name that resolves to the address clients will use; this becomes important when configuring advertised.listeners.
- A non-root service account named
kafka(or an equivalent dedicated account). - Persistent disk space for Kafka logs, with monitoring for utilization and I/O latency.
- A host firewall that allows only the client, administration, broker, and controller traffic you actually need.
- Administrative access for installing packages, creating directories, and configuring a service manager.
Java 17 or Java 21
Current Kafka 4.1 documentation fully supports Java 17 and Java 21 and recommends the newest long-term-support release, Java 21 at the time of writing. The current quick-start example for Kafka 4.3.0 requires Java 17 or newer. Install Java from your Linux distribution or a trusted OpenJDK build, then verify the runtime before touching Kafka.
java -versionOn an Ubuntu-like distribution, the package name is commonly openjdk-21-jre-headless; package names differ on other distributions. If you choose Java 17 instead, use the distribution’s Java 17 runtime and confirm that the selected Kafka release supports it.
Choose KRaft for a new installation
Kafka’s KRaft mode stores cluster metadata in Kafka’s own quorum instead of requiring ZooKeeper. It is the preferred workflow for a new installation. ZooKeeper instructions describe an older architecture and should not be selected for a fresh deployment unless you have a specific compatibility reason and a documented migration plan.
| Topology | Appropriate use | Important limitation |
|---|---|---|
| Combined broker and controller | Development, labs, and a single-VPS test | The same process handles client data and controller metadata; it is not a highly available design. |
| Dedicated controllers and brokers | Critical or larger deployments | Requires multiple nodes, private controller networking, and a planned quorum. |
| ZooKeeper-based cluster | Existing installations that still depend on that architecture | It is the older workflow, not the recommended starting point for new clusters. |
For controller availability, Kafka’s KRaft guidance uses a quorum of 2N + 1 controllers to tolerate N concurrent controller failures. A single VPS cannot provide host-level fault tolerance, regardless of how Kafka is configured.
Install Java and prepare the service account
- Update the operating system and install a supported Java 17 or Java 21 runtime with your distribution’s package manager.
- Create a system user and group for Kafka rather than running the broker as root.
- Create a stable installation directory such as
/opt/kafkaand a separate persistent data directory if your storage layout requires one. - Make the
kafkaaccount the owner of the installation, data, and log directories. - Record the server’s hostname and the DNS name that remote clients will resolve.
Do not copy a broker’s data directory between unrelated clusters. KRaft formats the metadata and log directories with a cluster identity; reusing a formatted directory with a different identity can prevent the broker from starting or produce an inconsistent node.
Download and extract a pinned Kafka release
Obtain the binary archive from the official Apache Kafka release channel and verify its checksum or signature according to that release’s instructions. Pin one version in your procedure so the archive name, configuration layout, and operational commands remain reproducible. The current quick-start example uses Kafka 4.3.0 with the Scala 2.13 distribution.
tar -xzf kafka_2.13-4.3.0.tgz
cd kafka_2.13-4.3.0Move or symlink the extracted directory to a stable path such as /opt/kafka after you have verified the archive. A stable path lets a systemd unit and monitoring configuration survive a controlled version upgrade; test upgrades on a separate node before changing a production broker.
Initialize KRaft storage
The bundled storage utility creates a cluster identifier and formats the directories described by the release’s KRaft configuration. For a one-node development broker, the standalone option creates the local controller configuration needed by the quick-start flow.
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.propertiesThe exact configuration path can vary between Kafka releases. Use the KRaft configuration shipped in the archive you downloaded instead of blindly copying a path from another version. Save the generated cluster ID in your deployment records. Do not format an existing data directory again unless you intentionally plan to destroy and recreate that cluster.
The --standalone command is a convenient single-node setup. A multi-node deployment needs node-specific broker and controller settings, a planned controller.quorum.voters list, and separate persistent directories; do not clone a standalone configuration onto every server.
Configure listeners before allowing remote clients
A listener is the network endpoint Kafka accepts connections on. Kafka supports PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. Listener names, security protocols, advertised addresses, and KRaft controller settings must agree.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Keep controller traffic private
The controller listener is for the KRaft quorum, not application clients. Bind it to a private interface or an internal network and never expose it to client networks or the public Internet. In a combined development node, the controller can be local; in a production topology, controller-to-controller traffic should remain on a protected network.
Advertise an address clients can actually reach
Clients use the bootstrap address only to obtain metadata. Kafka then returns the addresses in advertised.listeners, and clients connect to those addresses. If the broker advertises localhost, a private hostname, or an unroutable address, local tests can pass while remote clients fail.
For a local-only quick start, the supplied example uses localhost:9092. For remote access, advertise a DNS name that resolves from every client network and maps to a firewall-approved client listener. Do not advertise the controller endpoint.
Map listener names to security protocols
When you use named listeners, keep listener.security.protocol.map, controller.listener.names, and the inter-broker listener settings consistent. A conceptual combined-node arrangement has one client listener and one private controller listener:
Recommended Free Tools
process.roles=broker,controller
node.id=1
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://kafka.example.net:9092
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXTUse the release’s generated configuration as the authoritative baseline. The port shown for the controller above is an example; choose a private port that is allowed between the nodes in your quorum. For a secured deployment, replace the client and, where appropriate, inter-broker protocol with SSL or SASL_SSL and add the corresponding certificate, truststore, and SASL settings.
Start Kafka and make it persistent
Foreground start for the first validation
From the Kafka installation directory, start the broker in the foreground:
bin/kafka-server-start.sh config/server.propertiesKeep this first run attached to the terminal so startup errors are visible. Confirm that the process remains running and that the log shows the expected listeners and KRaft role. Stop it cleanly after the initial test if you are going to create the service unit.
Run the broker with systemd
On a VPS, systemd is a practical packaging choice: it starts Kafka after boot, restarts it after an unexpected failure, and runs it under the dedicated account. The unit below is an example; adjust paths, Java options, resource limits, and log handling to your distribution.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →[Unit]
Description=Apache Kafka broker
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=kafka
Group=kafka
WorkingDirectory=/opt/kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.targetSave the unit under your systemd configuration directory, reload systemd, enable it at boot, and start it. Then inspect the service status and journal. Keep Kafka’s own log directory on persistent storage and set retention and rotation policies so diagnostic files cannot fill the VPS disk.
Create and inspect a test topic
With the broker running and the client listener reachable, create a topic using the same bootstrap address that your client will use:
bin/kafka-topics.sh --create
--topic quickstart-events
--bootstrap-server localhost:9092Inspect its metadata, including partition leadership:
bin/kafka-topics.sh --describe
--topic quickstart-events
--bootstrap-server localhost:9092For a remote test, replace localhost:9092 with the advertised client address and provide the client properties required by your security protocol. A topic command that reaches the bootstrap socket but fails during metadata lookup usually points to an incorrect advertised address, DNS, firewall rule, or TLS/SASL setting.
Produce and consume one record
Run a producer and consumer from the Kafka installation, using matching client security properties when authentication or TLS is enabled:
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092Type a test line into the producer and verify that the consumer receives it. Stop both console programs with Ctrl-C after the test. This confirms more than process startup: it exercises topic metadata, partition leadership, writes, reads, and the selected client listener.
Secure a public VPS before opening the firewall
Apache Kafka states: Security is off by default. A freshly-installed Apache Kafka cluster accepts unauthenticated PLAINTEXT connections on every listener and applies no authorization. Treat an unmodified installation as a closed-network test system, not an Internet-facing service.
Use encryption and authentication
- Use
SSLfor TLS-encrypted connections with certificate validation, orSASL_SSLwhen you also need SASL authentication. - Use separate certificates and trust relationships appropriate to your clients, brokers, and controllers.
- Keep inter-broker traffic protected when brokers communicate across hosts or untrusted network segments.
- Distribute client trust material and credentials through a protected secret-management process, not a world-readable configuration file.
SASL_PLAINTEXT authenticates clients but does not encrypt the connection. Use it only where the network itself is trusted and the exposure is deliberate; TLS-backed protocols are the safer choice for traffic crossing a public or shared network.
Enable authorization and least privilege
Authentication answers who is connecting; authorization controls what that identity can do. Create separate identities for producers, consumers, and administrators, then grant only the topic, group, cluster, and transactional permissions each identity needs. Do not give application clients administrative access simply to make an initial test pass.
Apply firewall boundaries
Allow the configured client listener only from the application networks that need it. Permit controller and inter-broker listeners solely between the relevant private addresses. Restrict administration, SSH, metrics, and service-management access to your operations network or a bastion host. A firewall rule that opens an entire Kafka port to the Internet defeats listener-level security planning.
Protect data at rest
Kafka’s network security settings do not encrypt message logs stored on disk. If the VPS disk or snapshots could be accessed by an unauthorized party, use filesystem, block-device, or message-level encryption appropriate to your threat model. Protect backups and log archives with the same care as the live data.
Size the VPS from the workload
There is no universal production VPS size for Kafka. Required CPU, memory, storage, and network capacity depend on message rate, message size, retention period, replication factor, partition count, consumer behavior, and disk I/O.
| Planning factor | Why it changes capacity | What to measure |
|---|---|---|
| Ingress and egress rate | Determines network bandwidth, CPU, and write throughput. | Sustained and peak bytes per second, including protocol overhead. |
| Retention | Longer retention requires more usable disk and headroom for cleanup. | Daily data volume multiplied by retention, plus operational reserve. |
| Replication factor | Replicas multiply storage and network writes and require multiple brokers. | Replica count and recovery bandwidth during failures. |
| Partition count | Partitions increase metadata, file handles, scheduling work, and recovery time. | Partitions per topic and total partitions per broker. |
| Controller metadata | KRaft controllers need memory and disk for the metadata log. | Metadata growth and controller quorum health. |
Kafka’s KRaft operations guidance gives a typical baseline of 5 GB of main memory and 5 GB of disk for a controller metadata log. That figure is not a complete broker recommendation: a broker carrying retained application data needs additional disk, memory, I/O capacity, and network headroom. Monitor actual utilization and leave room for segment rolling, reassignments, and recovery.
Validate the installation systematically
- Run
java -versionand confirm Java 17 or 21 is the runtime used by the service account. - Check the process or systemd status and review broker logs for listener, storage, and KRaft errors.
- Use the host’s socket tools to confirm Kafka is bound only to the intended local and private addresses.
- Run
kafka-topics.sh --describeand verify that the test topic has a leader and expected metadata. - Produce and consume a test record with the same security properties used by the real application.
- From a separate client network, test DNS resolution, TCP reachability, TLS trust, authentication, and authorization independently.
- Check disk utilization, open-file limits, network throughput, and service restart behavior before relying on the VPS.
Troubleshoot common failures
The broker exits during startup
- Verify that the service is using the intended Java binary and that it satisfies the Kafka release requirement.
- Check the configuration path in
ExecStart; a systemd unit can point to a different file than your interactive shell. - Review storage errors and permissions. The
kafkaaccount must be able to read configuration and write every formatted log directory. - If KRaft reports a cluster-ID or formatting problem, stop and inspect the data directory before formatting anything again. Reformatting destroys the existing cluster state.
Local commands work but remote clients cannot connect
- Inspect
advertised.listenersfirst. Replace loopback, container-only, or private names with a DNS name reachable from the client network. - Confirm that the firewall allows the configured client listener from the client’s source address.
- Check that the DNS record resolves to the correct VPS address and that no reverse proxy or NAT rule changes the port.
- Make sure the client trusts the broker certificate and uses the same SASL mechanism and credentials configured by the broker.
TLS or SASL connections fail
- Verify that the listener name maps to the intended protocol in
listener.security.protocol.map. - Check certificate names, truststores, keystores, passwords, and file permissions under the Kafka service account.
- Confirm that clients use the security protocol assigned to the listener they are contacting; a plaintext client cannot negotiate an SSL listener.
- For KRaft, ensure controller and inter-broker listener mappings remain consistent and that the controller endpoint is not being used as a client bootstrap address.
The topic has no usable leader
Inspect the topic description and broker logs. A one-node development cluster cannot maintain a replica after that node is down. In a multi-broker cluster, check broker membership, replication settings, disk health, and network reachability between brokers and controllers.
When a single VPS is the wrong architecture
A single VPS is useful for learning, integration testing, and low-risk workloads, but it is one failure domain. Host failure, disk loss, maintenance, or a network outage takes down both the broker and, in a combined setup, the controller. Critical deployments should separate broker and controller roles, use a redundant controller quorum, place replicas on independent hosts, and define backup, restore, monitoring, and upgrade procedures.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteBefore moving beyond a test installation, document the advertised addresses, listener security protocols, certificate rotation process, ACLs, retention policy, disk-alert thresholds, and the steps for replacing a failed node. Those operational details determine whether the cluster remains recoverable when the VPS itself is unavailable.
Quick Recap
Installation checklist
- Java 17 or 21 installed and verified.
- Dedicated non-root Kafka account and persistent, correctly owned data directories.
- Pinned Kafka binary downloaded from the official release channel and verified.
- KRaft cluster ID generated once and storage formatted with the matching configuration.
- Client, inter-broker, and controller listeners intentionally separated and mapped.
advertised.listenersresolves and routes correctly from every client network.- Authentication, TLS, authorization, and firewall restrictions applied before public exposure.
- Test topic created, described, produced to, and consumed from using the intended client settings.
- Systemd restart behavior, logs, disk monitoring, and recovery procedures tested.
- Capacity and availability reviewed against message rate, retention, replication, partitions, and failure tolerance.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
