kafka
This is an old revision of the document!
Table of Contents
Kafka
Kafka (kRaft) in a VM
Install Java with SDKman ( curl -s “https://get.sdkman.io” | bash ) and “sdk install java 25.0.3-tem” (if there's a newer version and this page is not updated, use “sdk list java” to get the list). Install Maven with “sdk install maven 3.9.16” and Gradle with “sdk install gradle 9.5.1”.
Extract the Kafka 4.2.0 tarball.
Configure kRaft
# config/kraft/server.properties process.roles=broker,controller node.id=1 # Change for every server controller.quorum.voters=1@a.xyz:9093,2@b.xyz:9093,3@c.xyz:9093 #Default recommended is 6 (so each node gets 2 partitions) num.partitions=3 #At least 2 nodes must be in-sync; change existing lines offsets.topic.replication.factor=3 transaction.state.log.replication.factor=3 transaction.state.log.min.isr=2
#reinitialize the log storage after changing the default partition no. KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)" echo $KAFKA_CLUSTER_ID #re-use the UUID of the 1st server's storage on the 2nd & 3rd servers when formatting storage ./bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c ./config/kraft/server.properties sudo touch /var/log/mykafka.log && sudo chown ubuntu. /var/log/mykafka.log # Create a service file, this is for testing nohup ./bin/kafka-server-start.sh ./config/kraft/server.properties > /var/log/mykafka.log 2>&1 &
Kafka in Docker
# In Docker Compose
services:
kafka:
image: apache/kafka:latest
container_name: kafka
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
kafka.1779193384.txt.gz · Last modified: by reddy
