빅데이터 분석을 위한 준비 Kafka + Python 연결

Kafka + Python 연결 테스트

#1 Start zookeeper & kafka

[root@zepp zookeeper-3.4.14]# vi conf/zoo.cfg

[root@zepp zookeeper-3.4.14]# bin/zkServer.sh start

ZooKeeper JMX enabled by default

Using config: /home/min/zookeeper-3.4.14/bin/../conf/zoo.cfg

Starting zookeeper … STARTED

[root@zepp kafka_2.12-2.2.0]# bin/kafka-server-start.sh config/server.properties

[2019-04-19 18:12:03,572] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)

[2019-04-19 18:12:04,104] INFO starting (kafka.server.KafkaServer)

[2019-04-19 18:12:04,105] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)

[2019-04-19 18:12:04,128] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)

#2 create topic

[root@zepp kafka_2.12-2.2.0]# ./bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic suicides

Created topic suicides.

[root@zepp kafka_2.12-2.2.0]# bin/kafka-topics.sh –list –zookeeper localhost

suicides

#3 python용 kafka 설치

Collecting kafka

  Downloading https://files.pythonhosted.org/packages/21/71/73286e748ac5045b6a669c2fe44b03ac4c5d3d2af9291c4c6fc76438a9a9/kafka-1.3.5-py2.py3-none-any.whl (207kB)

    100% |████████████████████████████████| 215kB 12.2MB/s

Installing collected packages: kafka

Successfully installed kafka-1.3.5

(base) [min@zepp ~]$

#4 verify message

[root@zepp kafka_2.12-2.2.0]# bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic suicides

#5  python source

#6 Result