java编程
搜索
分类

kafka消费不到数据的排查过程

作者：香山上的麻雀1008 时间：2023-11-14 04:35:30　

kafka消费不到数据的排查

集群上新安装并启动了3个kafka Broker，代码打包上传至集群，运行后发现一直消费不到数据，

本地idea中debug后发现，程序一直阻塞在如下程序中，陷入了死循环。

/**
    * Block until the coordinator for this group is known and is ready to receive requests.
    * 等待直到我们和服务端的GroupCoordinator取得连接
    */
   public void ensureCoordinatorReady() {
       while (coordinatorUnknown()) {//无法获取GroupCoordinator
           RequestFuture<Void> future = sendGroupCoordinatorRequest();//发送请求
           client.poll(future);//同步等待异步调用的结果
           if (future.failed()) {
               if (future.isRetriable())
                   client.awaitMetadataUpdate();
               else
                   throw future.exception();
           } else if (coordinator != null && client.connectionFailed(coordinator)) {
               // we found the coordinator, but the connection has failed, so mark
               // it dead and backoff before retrying discovery
               coordinatorDead();
               time.sleep(retryBackoffMs);//等待一段时间，然后重试
           }

}
   }

流程大概说就是

  • consumer会从集群中选取一个broker作为coordinator

  • 然后group中的consumer会向coordinator发请求申请成为consumergroup中的leader

  • 最后有1个consumer会成为consumerLeader ,其他consumer成为follower

  • consumerLeader做分区分配任务,同步给coordinator

  • consumerFollower从coordinator同步分区分配数据

问题出现在第一步，意思就是说Consumer和服务端的GroupCoordinator无法取得连接，所以程序一直在等待状态。

看了下__consumer_offsets 这个topic情况,50个分区全在broker id为152的broker上

bin/kafka-topics.sh --describe --zookeeper localhost:2182 --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:1    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets    Partition: 0    Leader: 152    Replicas: 152   Isr:152
    Topic: __consumer_offsets    Partition: 1    Leader: 152    Replicas: 152   Isr:152
    Topic: __consumer_offsets    Partition: 2    Leader: 152    Replicas: 152   Isr:152
    Topic: __consumer_offsets    Partition: 3    Leader: 152   
......

但是集群上并没有broker id为152的节点，想到该集群kafka节点曾经添加删除过节点，初步断定152是之前的kafka节点，后来该节点去掉后又加入新的节点但是zookeeper中的数据并没有更新。

所以就关闭broker，进入zookeeper客户端，将brokers节点下的topics节点下的__consumer_offsets删除，然后重启broker，注意，此时zookeeper上__consumer_offsets还并没有生成,要开启消费者之后才会生成.

然后再观察__consumer_offsets,分区已经均匀分布在三个broker上面了

 bin/kafka-topics.sh --zookeeper localhost:2182 --describe --topic __consumer_offsets
Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:3    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets    Partition: 0    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 1    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 2    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 3    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 4    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 5    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 6    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 7    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 8    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 9    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 10    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 11    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 12    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 13    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 14    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 15    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 16    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 17    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 18    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 19    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 20    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 21    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 22    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 23    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 24    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 25    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 26    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 27    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 28    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 29    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 30    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 31    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 32    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 33    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 34    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 35    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 36    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 37    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 38    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 39    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 40    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 41    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 42    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 43    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 44    Leader: 422    Replicas: 422,420,421    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 45    Leader: 420    Replicas: 420,422,421    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 46    Leader: 421    Replicas: 421,420,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 47    Leader: 422    Replicas: 422,421,420    Isr: 422,420,421
    Topic: __consumer_offsets    Partition: 48    Leader: 420    Replicas: 420,421,422    Isr: 420,422,421
    Topic: __consumer_offsets    Partition: 49    Leader: 421    Replicas: 421,422,420    Isr: 422,420,421

这个时候重启程序，发现已经可以正常消费了，问题解决。

参考资料：

  • https://stackoverflow.com/questions/42362911/kafka-high-level-consumer-error-code-15

来源：https://chengyanan.blog.csdn.net/article/details/100771092

标签：kafka,消费,数据
0
投稿

猜你喜欢

  • Spring Boot 开发环境热部署详细教程

    2023-07-28 01:50:02

  • 配置Android SDK

    2023-12-05 09:57:29

  • C# KeyUp事件中MessageBox的回车（Enter）键回调问题解决方案

    2021-06-25 06:24:06

  • Spring Boot实现热部署的实例方法

    2023-10-24 08:12:50

  • 11月编程语言排行榜 Python逆袭C#上升到第4

    2023-04-27 09:51:17

  • 浅谈Unity脚本生命周期与执行顺序

    2022-05-22 00:47:34

  • C# 解析 Excel 并且生成 Csv 文件代码分析

    2021-11-09 18:04:04

  • 面试Spring中的bean线程是否安全及原因

    2021-06-11 18:25:38

  • C#实现DVD借出归还管理系统

    2021-07-02 06:55:38

  • Java字符串编码知识点详解介绍

    2023-10-16 09:41:04

  • 详解Spring-Boot中如何使用多线程处理任务

    2022-05-14 13:10:40

  • Unity实现倒计时组件

    2021-06-18 01:06:44

  • Android MTU 值修改的实例详解

    2021-09-20 02:27:07

  • Android中Edittext设置输入条件

    2023-03-11 23:05:08

  • Java基础之异常处理详解

    2021-07-14 03:03:49

  • IDEA启动tomcat控制台中文乱码问题的解决方法(100％有效)

    2021-06-25 10:45:23

  • C语言 socketpair用法案例讲解

    2022-07-24 12:41:48

  • Java spring webmvc如何实现控制反转

    2023-03-12 06:35:56

  • SpringBoot2零基础到精通之映射与常用注解请求处理

    2022-06-11 15:41:51

  • Java中StringUtils与CollectionUtils和ObjectUtil概念讲解

    2023-11-29 07:45:38
    • asp之家 软件编程 m.aspxhome.com