本文共 1314 字，大约阅读时间需要 4 分钟。

整体架构拓扑图 

Agent1 错误（exec source 不是avro source）

一号机配置文件

监听/opt/flume/flume/data/data.log作为flume线生产者

a1.sources = r1a1.sinks = k1a1.channels = c1a1.sources.r1.type = execa1.sources.r1.command = tail -F /opt/flume/flume/data/data.loga1.sources.r1.shell = /bin/sh -ca1.sinks.k1.type = avroa1.sinks.k1.hostname = hadoop2a1.sinks.k1.port = 44444a1.channels.c1.type = memorya1.sources.r1.channels = c1a1.sinks.k1.channel = c1

二号机配置文件

消费一号机sink信息，并输出给kafka

[root@hadoop2 conf]# more avro-memory-kafka.conf a1.sources = r1a1.sinks = k1a1.channels = c1a1.sources.r1.type = avroa1.sources.r1.bind = hadoop2a1.sources.r1.port = 44444a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.brokerList = hadoop:9092a1.sinks.k1.topic = zza1.sinks.k1.batchSize = 5a1.sinks.k1.requiredAcks = 1a1.channels.c1.type = memorya1.sources.r1.channels = c1a1.sinks.k1.channel = c1

二号机启用Flume

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console

一号机启用Flume

flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console

一号机启用Kafka消费消息

kafka-console-consumer.sh --bootstrap-server hadoop:9092 --from-beginning  --topic zz

一号机向/opt/flume/flume/data/data.log中追加信息，并查看一号机Kafka消费端打印信息

转载地址：http://vtazi.baihongyu.com/