metricbeat+kafka+hbase

一.Consumer

实例:http://blog.51cto.com/1054054

metricbeat+kafka:

Kafka笔记整理(一)

metricbeat对接kafka

Kafka module | Metricbeat Reference [6.3] | Elastic

kafka+hbase:

kafka数据导入hbase

Kafka 接受数据并消费到hbase数据库

kafkaToHbase

hbase提取kafka中的数据存储

Kafka-HBase integration

二.以上只是简单的消费者,以下添加了stream:

code:https://github.com/gongel/metricbeat-kafka-hbase/tree/master/Proj/src/main/java/cn/gongel

用到的是stream中的foreach:http://kafka.apache.org/11/documentation/streams/developer-guide/dsl-api#stateless-transformations

三.Q&A

1.KafkaStream时间戳问题CreateTime = -1引起的程序中断

2.基于Kubernetes在AWS上部署Kafka时遇到的一些问题

3.https://blog.csdn.net/gel1234/article/details/81224139

4.Q:client/metadata found some partitions to be leaderless

A:kafka停掉了。删除kafka-logs(或者data)目录下的文件,会导致kafka停止

四.扩展

Kafka源码分析-Content Table

 

0

kafka配置

配置参考:https://blog.csdn.net/M_SIGNALs/article/details/53201595

错误一:

[2018-07-19 09:42:21,913] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 2 doesn't match stored broker.id 1 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
	at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:670)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:209)
	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
	at kafka.Kafka$.main(Kafka.scala:92)
	at kafka.Kafka.main(Kafka.scala)
[2018-07-19 09:42:21,916] INFO shutting down (kafka.server.KafkaServer)
[2018-07-19 09:42:21,918] WARN  (kafka.utils.CoreUtils$)
java.lang.NullPointerException
	at kafka.server.KafkaServer$$anonfun$shutdown$5.apply$mcV$sp(KafkaServer.scala:569)
	at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:85)
	at kafka.server.KafkaServer.shutdown(KafkaServer.scala:569)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:326)
	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
	at kafka.Kafka$.main(Kafka.scala:92)
	at kafka.Kafka.main(Kafka.scala)

是由于从master复制到slave节点时,log有数据没删掉。

Kafka集群partitions/replicas默认分配解析

Kafka运行机制与各组件详解

常用命令kafka

0