ProcMail › dom › OpenSourceSW › (번역)PleaseStopCallingDatabasesCPOrAP › 5일차-XMLWebServicesWithJava › CopyingOffsetsOfAConsumerGroupToAnotherConsumerGroupInKafka0.10WithPythonScript
Copying offsets of consumer group grp1 to grp2 (Kafka 0.10)
cf. Kafka 1.1: copy offsets of a consumer group to another consumer group
The following python scripts are available from github
- stop all consumers of grp1 group
- check status of grp1 (source group)
~/programs/kafka_2.11-0.10.1.1$ bin/kafka-consumer-groups.sh --describe --group grp1 --bootstrap-server localhost:9092 Consumer group `grp1` is rebalancing.
- check status of grp2 (target group)
~/programs/kafka_2.11-0.10.1.1$ bin/kafka-consumer-groups.sh --describe --group grp2 --bootstrap-server localhost:9092 Consumer group `grp2` does not exist.
- run poll-without-commit.py script to check status of consumer groups (DO NOT break this time)
$ python3 ~/bin/poll-without-commit.py grp1 grp2
- check status of groups in a new terminal
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp1 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER grp1 test1 0 26 31 5 py-1_/127.0.0.1 grp1 test1 1 29 33 4 py-1_/127.0.0.1 grp1 test1 2 28 33 5 py-1_/127.0.0.1 ~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER grp2 test1 0 unknown 31 unknown py-1_/127.0.0.1 grp2 test1 1 unknown 33 unknown py-1_/127.0.0.1 grp2 test1 2 unknown 33 unknown py-1_/127.0.0.1
- STOP above script(poll-without-commit.py) with Ctrl+C and WAIT for grp2 to go into REBALANCING state.
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2 Consumer group `grp2` is rebalancing.
- copy offsets from grp1 to grp2 with copy-offsets.py
$ python3 ~/bin/copy-offsets.py grp1 grp2 partition: 0 beginning: 0 committed: 26 end: 31 position: 26 partition: 1 beginning: 0 committed: 29 end: 33 position: 29 partition: 2 beginning: 0 committed: 28 end: 33 position: 28 partition: 0 beginning: 0 committed: None end: 31 position: 26 partition: 1 beginning: 0 committed: None end: 33 position: 29 partition: 2 beginning: 0 committed: None end: 33 position: 28
- run poll-without-commit.py script to check status of consumer groups (DO NOT break this time)
$ python3 ~/bin/poll-without-commit.py grp1 grp2
- check result of copying
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp1 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER grp1 test1 0 26 31 5 py-1_/127.0.0.1 grp1 test1 1 29 33 4 py-1_/127.0.0.1 grp1 test1 2 28 33 5 py-1_/127.0.0.1 ~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER grp2 test1 0 26 31 5 py-1_/127.0.0.1 grp2 test1 1 29 33 4 py-1_/127.0.0.1 grp2 test1 2 28 33 5 py-1_/127.0.0.1 ~/programs/kafka-p1$
- Ctrl+C above script poll-without-commit.py