定制型网站设计,有全部公司的网站,域名购买后还要解析吗,心理医生免费咨询文章目录 1.1.监控配置1.2.监控工具1.3.性能指标系统相关指标GC相关指标JVM相关指标Topic相关指标Broker相关指标 1.4.性能指标说明1.5.重要指标说明 1.1.监控配置
开启JMX服务端口#xff1a;kafka基本分为broker、producer、consumer三个子项#xff0c;每一项的启动都需要… 文章目录 1.1.监控配置1.2.监控工具1.3.性能指标系统相关指标GC相关指标JVM相关指标Topic相关指标Broker相关指标 1.4.性能指标说明1.5.重要指标说明 1.1.监控配置
开启JMX服务端口kafka基本分为broker、producer、consumer三个子项每一项的启动都需要用到 $KAFKA_HOME/bin/kafka-run-class.sh 脚本在该脚本中存在以下语句
if ...
KAFKA_JMX_OPTS-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticatefalse -Dcom.sun.management.jmxremote.sslfalsefi
if ...KAFKA_JMX_OPTS$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port$JMX_PORT
Fi 在启动kafka的过程中只要指定 JMX_PORT 的值即可对broker、producer、consumer进行监控。目前有两种方法在$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/bin/kafka-console-consumer.sh $KAFKA_HOME/bin/kafka-console producer.sh三个脚本中分别添加 $JMX_PORTXXXX 语句但是只适用于使用console方式对topic进行使用的情况。 修改$KAFKA_HOME/bin/kafka-run-class.sh 脚本中的上述语句使其端口随机变化可以通过 ps -ef |grep kafka 命令来获取随机的端口号来进行监控。
1.2.监控工具
Prometheus监控Kafka
如可以采用docker部署
kafka-exporter
docker run -ti -d --rm -p 9308:9308 danielqsj/kafka-exporter --kafka.server192.168.0.4:9092监控项名称阈值说明使用的公式Kafka的Brokers在线1m !1严重count(kafka_server_replicamanager_leadercount{job~$job})Kafka集群中副本处于同步失败或失效状态的分区数0严重sum(kafka_topic_partition_under_replicated_partition{topic~$topic, namespace~$kubernetes_namespace})Kafka集群中控制器的数量!1严重sum(kafka_controller_kafkacontroller_activecontrollercount{job~$job})Kafka离线分区数0严重sum(kafka_controller_kafkacontroller_offlinepartitionscount{job~$job})Kafka每秒入网络流量150中度avg_over_time(kafka_server_BrokerTopicMetrics_OneMinuteRate{nameBytesInPerSec,topic}[1m]) / 1024 /1024Kafka请求处理程序线程空闲的平均时间百分比 0.3中度avg_over_time(kafka_server_KafkaRequestHandlerPool_OneMinuteRate{nameRequestHandlerAvgIdlePercent,}[1m]) 2. 在prometheus.yml出添加kafka配置 - job_name: kafka_exporterstatic_configs:- targets: [$node1:9308]重启prometheus加载。在promethues的管理界面可以查看状态 然后配置grafana来展示图表效果。告警监控项如下表供参考
时间百分比rcent,}[1m])Kafka请求处理程序线程空闲的平均时间百分比 0.3中度avg_over_time(kafka_server_KafkaRequestHandlerPool_OneMinuteRate{name“RequestHandlerAvgIdlePercent”,}[1m])Kafka网络处理器线程空闲的平均时间百分比 0.3中度avg_over_time(kafka_network_SocketServer_Value{name“NetworkProcessorAvgIdlePercent”,}[1m])Kafka已建立的连接数 3000中度 5000严重sum(avg_over_time(kafka_server_socket_server_metrics_connection_count{listener“PLAINTEXT”,}[1m])) by (instance,app)Kafka每秒新建连接数 100中度 200 严重sum(avg_over_time(kafka_server_socket_server_metrics_connection_creation_rate[1m])) by (instance)Kafka请求在请求队列中等待的时间5000中度avg_over_time(kafka_network_RequestMetrics_999thPercentile{name“RequestQueueTimeMs”,request“Produce”,}[1m])Kafka_leader处理请求的时间5000中度avg_over_time(kafka_network_RequestMetrics_999thPercentile{name“LocalTimeMs”,request“Produce”,}[1m])Kafka请求等待follower的时间1000中度avg_over_time(kafka_network_RequestMetrics_999thPercentile{name“RemoteTimeMs”,request“Produce”,}[1m])Kafka请求在响应队列中等待的时间1000中度avg_over_time(kafka_network_RequestMetrics_999thPercentile{name“ResponseQueueTimeMs”,request“Produce”,}[1m])Kafka发送响应的时间1000中度avg_over_time(kafka_network_RequestMetrics_999thPercentile{name“ResponseSendTimeMs”,request“Produce”,}[1m])Kafka汇总传入消息速率 200000中度avg_over_time(kafka_server_BrokerTopicMetrics_OneMinuteRate{name“MessagesInPerSec”,topic“”}[1m])kafka消费滞后告警1000sum(kafka_consumergroup_lag{topic!“sop_free_study_fix-student_wechat_detail”}) by (consumergroup, topic) 1000kafka-exporter停止 1kafka_exporter_build_infokafka server停止1kafka_brokerskafka监控topic实时生产速率 0sum(irate(kafka_topic_partition_current_offset{topic !~ __consumer_offsetsKafka消费者端分区偏移量5m 0sum(delta(kafka_consumergroup_current_offset[5m])/5) by (consumergroup, topic)Kafka消费者组的当前主题分区偏移汇总sum(delta(kafka_consumergroup_current_offset_sum[5m])/5) by (consumergroup, topic)Kafka某个消费组消费延迟5m 100000中度sum(kafka_consumergroup_lag) by (consumergroup,partition,topic)Kafka某个消费者组在某个主题分区的近似滞后情况汇总sum(kafka_consumergroup_lag_sum) by (consumergroup,partition,topic)某个消费组成员kafka_consumergroup_members{instance“$instance”}Kafka分区的位移量汇总sum(kafka_topic_partition_current_offset) by (partition,topic)Kafka分区的同步副本数1m 0 中度sum(kafka_topic_partition_in_sync_replica)Kafka旧主题分区偏移sum(kafka_topic_partition_oldest_offset{topic~“$topic”}) by (partition,topic)Kafka主题分区的副本数1m 3中度sum(kafka_topic_partition_replicas{topic~“$topic”})Kafka主题分区复制不足的分区数sum(kafka_topic_partition_under_replicated_partition{topic~“$topic”})Kafka 总分区数5m 1000中度sum(kafka_topic_partitions) by(topic)
1.3.性能指标
系统相关指标
系统信息收集 java.lang:typeOperatingSystemThread信息收集 java.lang:typeThreading获取mmaped和direct空间通过BufferPoolMXBean获取used、capacity、count
GC相关指标
Young GC java.lang:typeGarbageCollector,nameG1 Young GenerationOld GC java.lang:typeGarbageCollector,nameG1 Old Generation
JVM相关指标
通过MemoryMXBean获取JVM相关信息HeapMemoryUsage和NonHeapMemoryUsage通过MemoryPoolMXBean获取其他JVM内存空间指标例如Metaspace、Codespace等
Topic相关指标
Topic消息入站速率Byte kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec,topic topicTopic消息出站速率Byte kafka.server:typeBrokerTopicMetrics,nameBytesOutPerSec,topic topicTopic请求被拒速率 kafka.server:typeBrokerTopicMetrics,nameBytesRejectedPerSec,topic topicTopic失败拉去请求速率 kafka.server:typeBrokerTopicMetrics,nameFailedFetchRequestsPerSec,topic topic;Topic发送请求失败速率 kafka.server:typeBrokerTopicMetrics,nameFailedProduceRequestsPerSec,topic topicTopic消息入站速率message kafka.server:typeBrokerTopicMetrics,nameMessagesInPerSec,topic topic
Broker相关指标
Log flush rate and time kafka.log:typeLogFlushStats,nameLogFlushRateAndTimeMs同步失效的副本数 kafka.server:typeReplicaManager,nameUnderReplicatedPartitions消息入站速率消息数 kafka.server:typeBrokerTopicMetrics,nameMessagesInPerSec消息入站速率Byte kafka.server:typeBrokerTopicMetrics,nameBytesInPerSec消息出站速率Byte kafka.server:typeBrokerTopicMetrics,nameBytesOutPerSec请求被拒速率 kafka.server:typeBrokerTopicMetrics,nameBytesRejectedPerSec失败拉去请求速率 kafka.server:typeBrokerTopicMetrics,nameFailedFetchRequestsPerSec发送请求失败速率 kafka.server:typeBrokerTopicMetrics,nameFailedProduceRequestsPerSecLeader副本数 kafka.server:typeReplicaManager,nameLeaderCountPartition数量 kafka.server:typeReplicaManager,namePartitionCount下线Partition数量 kafka.controller:typeKafkaController,nameOfflinePartitionsCountBroker网络处理线程空闲率 kafka.server:typeKafkaRequestHandlerPool,nameRequestHandlerAvgIdlePercentLeader选举比率 kafka.controller:typeControllerStats,nameLeaderElectionRateAndTimeMsUnclean Leader选举比率 kafka.controller:typeControllerStats,nameUncleanLeaderElectionsPerSecController存活数量 kafka.controller:typeKafkaController,nameActiveControllerCount请求速率 kafka.network:typeRequestMetrics,nameRequestsPerSec,requestProduceConsumer拉取速率 kafka.network:typeRequestMetrics,nameRequestsPerSec,requestFetchConsumerFollower拉去速率 kafka.network:typeRequestMetrics,nameRequestsPerSec,requestFetchFollowerRequest total time kafka.network:typeRequestMetrics,nameTotalTimeMs,requestProduceConsumer fetch total time kafka.network:typeRequestMetrics,nameTotalTimeMs,requestFetchConsumerFollower fetch total time kafka.network:typeRequestMetrics,nameTotalTimeMs,requestFetchFollowerTime the follower fetch request waits in the request queue kafka.network:typeRequestMetrics,nameRequestQueueTimeMs,requestFetchFollowerTime the Consumer fetch request waits in the request queue kafka.network:typeRequestMetrics,nameRequestQueueTimeMs,requestFetchConsumerTime the Produce fetch request waits in the request queue kafka.network:typeRequestMetrics,nameRequestQueueTimeMs,requestProduceBroker I/O工作处理线程空闲率 kafka.network:typeSocketServer,nameNetworkProcessorAvgIdlePercentISR变化速率 kafka.server:typeReplicaManager,nameIsrShrinksPerSec
1.4.性能指标说明
指标单位具体含义kafka.broker_offset offsetsbroker上当前消息的偏移量offsetkafka.consumer.bytes_inbytes/secondconsumer 字节率bytes in ratekafka.consumer.delayed_requestsrequests延迟的 consumer 请求数kafka.consumer.expires_per_secondevictions/second延迟 consumer 的请求到期expiration速率kafka.consumer.fetch_raterequestsconsumer 向 broker 发送提取请求fetch requests的最低速率kafka.consumer.kafka_commitswrites/second面向 Kafka 的 offset commits 速率kafka.consumer.max_lagoffsets最大消费滞后consumer lagkafka.consumer.messages_inmessages/secondconsumer 消息消费consumption的速率kafka.consumer.zookeeper_commitswrites/second面向 ZooKeeper 的 offset commits 速率kafka.consumer_lagoffsetsconsumer 和 broker 之间的消息滞后lagkafka.consumer_offsetoffsetsconsumer 的当前消息偏移量current message offsetkafka.expires_secevictions/second延迟生产者delayed producer的请求到期request expiration速率kafka.follower.expires_per_secondevictions/second关注者followers的请求到期request expiration速率kafka.log.flush_rateflushes/second日志刷新速率kafka.messages_inmessages传入incoming信息速率kafka.net.bytes_inbytes/second传入incoming字节速率kafka.net.bytes_outbytes/second传出outgoing字节速率kafka.net.bytes_rejectedbytes/second被拒绝rejected的字节速率kafka.producer.bytes_outbytes/secondproducer 字节输出速率kafka.producer.delayed_requestsrequests延迟的 producer 请求数kafka.producer.expires_per_secondsevictions/secondproducer 请求到期率kafka.producer.io_waitnanosecondsProducer I/O 等待时间kafka.producer.message_ratemessages/secondProducer 消息速率kafka.producer.request_latency_avgmillisecondsProducer 平均请求延迟kafka.producer.request_raterequests/secondproducer 每秒钟的请求数kafka.producer.response_rateresponses/secondproducer 每秒钟的响应数kafka.replication.isr_expandsnodes/second副本加入 ISR 池的速率kafka.replication.isr_shrinksnodes/second副本离开 ISR 池的速率kafka.replication.leader_electionsevents/second领导选举Leader election频率kafka.replication.unclean_leader_electionsevents/secondUnclean 的领导选举Leader election频率kafka.replication.under_replicated_partitions未使用的分区数kafka.request.fetch.failedrequests客户端获取请求fetch request失败次数kafka.request.fetch.failed_per_secondrequests/second每秒钟的客户端获取请求fetch request失败率kafka.request.fetch.time.99percentilerequests/second获取请求fetch request时间的第 99 百分位的值kafka.request.fetch.time.avgrequests/second获取请求fetch request时间的平均值kafka.request.handler.avg.idle.pctfractions请求处理程序线程request handler threads的平均空闲时间占比kafka.request.metadata.time.99percentilemilliseconds元数据metadata请求时间的第 99 百分位的值kafka.request.metadata.time.avgmilliseconds元数据metadata请求时间的的平均值kafka.request.offsets.time.99percentilemillisecondsoffset 请求时间的第 99 百分位的值kafka.request.offsets.time.avgmillisecondsoffset 请求时间的平均值kafka.request.produce.failedrequests失败的产品请求produce requests数kafka.request.produce.failed_per_secondrequests/second每秒钟的产品请求produce requests失败率kafka.request.produce.time.99percentilerequests/second产品请求produce requests时间的第 99 百分位的值kafka.request.produce.time.avgrequests/second产品请求produce requests平均时间kafka.request.update_metadata.time.99percentilemilliseconds更新元数据请求update metadata requests时间的第 99 百分位的值kafka.request.update_metadata.time.avgmilliseconds更新元数据请求update metadata requests时间的平均值
1.5.重要指标说明
参照kafka-manager管理工具 1.
kafka.replication.under_replicated_partitions
Under Replicated Partitions: 在一个运行健康的集群中处于同步状态的副本数ISR应该与总副本数简称AR:Assigned Repllicas完全相等如果分区的副本远远落后于leader那这个follower将被ISR池删除随之而来的是IsrShrinksPerSec(可理解为isr的缩水情况后面会讲)的增加。由于kafka的高可用性必须通过副本来满足所有有必要重点关注这个指标让它长期处于大于0的状态。 2. Brokers Spread broker使用率如kafka集群9个broker某topic有7个partition则broker spread: 7 / 9 77% 3. Brokers Leader Skew leader partition是否存在倾斜如kafka集群9个broker某topic14个partition则正常每个broker有2个leader partition。若其中一个broker有0个leader partition一个有4个leader partition则broker leader skew: (4 - 2) / 14 14% 由于kafka所有读写都在leader上进行 broker leader skew会导致不同broker的读写负载不均衡配置参数 auto.leader.rebalance.enabletrue 可以使kafka每5min自动做一次leader的rebalance消除这个问题。 4. Lag 表示consumer的消费能力计算公式为Lag LogSize - Consumer OffsetKafka Manager从zk获取LogSize从kafka __consumer_offsets topic读取Offset。两步操作存在一个时间gap因此吞吐很大的topic上会出现LogSize Offset 的情况。导致Lag负数。
