Kafka with K8s

개요

• Kubernetes에 kafka를 설치하여 동작 테스트

minikube를 활용할 것임

minikube & Docker

• 설치 및 실행은 이전 포스트를 참고.
• 테스트 버전
  • minikube v1.25.2
  • kubernetes, kubectl v1.23.3

$ minikube start

–nodes 옵션으로 node 갯수를 조절 가능. 하지만 내 맥북에선 사양탓인지 안되어서 하나의 node로 테스트

Deploy Kafaka With Kubernetes

• 참고 사이트

namespace 생성

apiVersion: v1
kind: Namespace
metadata:
  name: "kafka"
  labels:
    name: "kafka"

Deploying Zookeeper

• port 는 2181

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
  namespace: kafka
spec:
  type: NodePort
  ports:
    - name: zookeeper-port
      port: 2181
      nodePort: 30181
      targetPort: 2181
  selector:
    app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
  namespace: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - image: wurstmeister/zookeeper
          imagePullPolicy: IfNotPresent
          name: zookeeper
          ports:
            - containerPort: 2181

• Cluster-IP 확인이 필요

  NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
  zookeeper-service   NodePort    10.107.170.58   <none>        2181:30181/TCP   10h
  

Deploying a Kafka Broker

• KAFKA_ZOOKEEPER_CONNECT에 위에서 확인한 Cluster-IP 10.107.170.58를 사용

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-broker
  name: kafka-service
  namespace: kafka
spec:
  ports:
  - port: 9092
  selector:
    app: kafka-broker
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka-broker
  name: kafka-broker
  namespace: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      hostname: kafka-broker
      containers:
      - env:
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 10.107.170.58:2181
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://:9092
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://kafka-broker:9092
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka-broker
        ports:
        - containerPort: 9092

hosts 수정 필요

• 헤당 예시는 kafka-broker로 host를 등록해야만 동작.

  127.0.0.1 kafka-broker
  

Testing

• 모든 파드가 떠있는지 확인한다.

% kubectl get all -n kafka
NAME                                READY   STATUS    RESTARTS      AGE
pod/kafka-broker-5cbdf4c4fb-8cb4j   1/1     Running   1 (14m ago)   10h
pod/zookeeper-55b668879d-qzgfs      1/1     Running   1 (14m ago)   10h

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/kafka-service       ClusterIP   10.110.57.129   <none>        9092/TCP         10h
service/zookeeper-service   NodePort    10.107.170.58   <none>        2181:30181/TCP   10h

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafka-broker   1/1     1            1           10h
deployment.apps/zookeeper      1/1     1            1           10h

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/kafka-broker-5cbdf4c4fb   1         1         1       10h
replicaset.apps/zookeeper-55b668879d      1         1         1       10h

• 서비스를 port-forward 하여 테스트 가능하도록

  % kubectl port-forward svc/kafka-service 9092 -n kafka
  

• 테스트를 위해 kcat 설치
  • Mac OS는 brew install kcat

kcat Command Test

• Producer

  % echo "hello world" | kcat -P -b 127.0.0.1:9092 -t test
  

• Consumer

  % kcat -C -b localhost:9092 -t test
  hello world
  % Reached end of topic test [0] at offset 1
  

다음 목표

• python 예제로 producer, consumer 만들어 볼 예정

마무리

Kafka에 대해 정확히 모르고 예시만 일단 따라해봄.. 그래서 hosts에 kafka-broker hostname 등록 안하고 싶었는데 아직 방법 확인 못함. 공부가 필요하다..