Copying offsets of consumer group grp1 to grp2 (Kafka 0.10)


The following python scripts are available from [https]github



  • stop all consumers of grp1 group

  • check status of grp1 (source group)
~/programs/kafka_2.11-0.10.1.1$ bin/kafka-consumer-groups.sh --describe --group grp1 --bootstrap-server localhost:9092
Consumer group `grp1` is rebalancing.

  • check status of grp2 (target group)
~/programs/kafka_2.11-0.10.1.1$ bin/kafka-consumer-groups.sh --describe --group grp2 --bootstrap-server localhost:9092
Consumer group `grp2` does not exist.

  • run poll-without-commit.py script to check status of consumer groups (DO NOT break this time)
$ python3 ~/bin/poll-without-commit.py grp1 grp2

  • check status of groups in a new terminal
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp1
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
grp1                           test1                          0          26              31              5               py-1_/127.0.0.1
grp1                           test1                          1          29              33              4               py-1_/127.0.0.1
grp1                           test1                          2          28              33              5               py-1_/127.0.0.1

~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
grp2                           test1                          0          unknown         31              unknown         py-1_/127.0.0.1
grp2                           test1                          1          unknown         33              unknown         py-1_/127.0.0.1
grp2                           test1                          2          unknown         33              unknown         py-1_/127.0.0.1

  • STOP above script(poll-without-commit.py) with Ctrl+C and WAIT for grp2 to go into REBALANCING state.
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2
Consumer group `grp2` is rebalancing.

  • copy offsets from grp1 to grp2 with copy-offsets.py
$ python3 ~/bin/copy-offsets.py grp1 grp2
partition: 0 beginning: 0 committed: 26 end: 31 position: 26
partition: 1 beginning: 0 committed: 29 end: 33 position: 29
partition: 2 beginning: 0 committed: 28 end: 33 position: 28
partition: 0 beginning: 0 committed: None end: 31 position: 26
partition: 1 beginning: 0 committed: None end: 33 position: 29
partition: 2 beginning: 0 committed: None end: 33 position: 28

  • run poll-without-commit.py script to check status of consumer groups (DO NOT break this time)
$ python3 ~/bin/poll-without-commit.py grp1 grp2

  • check result of copying
~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp1
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
grp1                           test1                          0          26              31              5               py-1_/127.0.0.1
grp1                           test1                          1          29              33              4               py-1_/127.0.0.1
grp1                           test1                          2          28              33              5               py-1_/127.0.0.1

~/programs/kafka-p1$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group grp2
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
grp2                           test1                          0          26              31              5               py-1_/127.0.0.1
grp2                           test1                          1          29              33              4               py-1_/127.0.0.1
grp2                           test1                          2          28              33              5               py-1_/127.0.0.1
~/programs/kafka-p1$ 

  • Ctrl+C above script poll-without-commit.py
Valid XHTML 1.0! Valid CSS! powered by MoniWiki
last modified 2019-02-24 13:51:45
Processing time 0.0070 sec