I have been trying to ingest data from kafka source to flume. 17:01:50,066 INFO .utils.AppInfoParser: Kafka commitId : unknown 17:01:49,952 INFO : Logging to 4jLoggerAdapter() via 4jLog Key.deserializer = class .serialization.StringDeserializer serializer = class .serialization.ByteArrayDeserializer 17:01:49,941 INFO .consumer.ConsumerConfig: ConsumerConfig values: 17:01:49,854 INFO .Application: Starting new configuration. 17:01:49,847 INFO .AbstractConfigurationProvider: Channel channel1 connected to 17:01:49,835 INFO .DefaultSinkFactory: Creating instance of sink: sink1, type: hdfs 17:01:49,799 INFO .DefaultSourceFactory: Creating instance of source source1, type .kafka.KafkaSource 17:01:49,798 INFO .AbstractConfigurationProvider: Created channel channel1 17:01:49,790 INFO .DefaultChannelFactory: Creating instance of channel channel1 type memory 17:01:49,783 INFO .AbstractConfigurationProvider: Creating channels 17:01:49,783 INFO .FlumeConfiguration: Post-validation flume configuration contains configuration for agents: 17:01:49,766 INFO .FlumeConfiguration: Added sinks: sink1 Agent: tier1 17:01:49,766 INFO .FlumeConfiguration: Processing:sink1 17:01:49,765 INFO .FlumeConfiguration: Processing:sink1 17:01:49,760 INFO .PollingPropertiesFileConfigurationProvider: Reloading configuration file:/run/cloudera-scm-agent/process/2727-flume-AGENT/nf 17:01:49,742 INFO .PollingPropertiesFileConfigurationProvider: Configuration provider starting The result indicated that the flume agent could not get any data from Kafka. If the flume agent could fetch message from the topic, then the output of the above command should be empty. $ kafka-console-consumer -zookeeper bda03:2182,bda04:2182,bda05:2182 -topic test -consumer-property group.id=flume Then, I tried to retreive message from topic "test" via command line (with consumer group flume) However, there was nothing put into HDFS. fileType = DataStreamĪfter starting the flume agent, there was no error message in flume and kafka log file. I created topic "test" in kafka, and would like to configure flume to act as consumer to fetch data from this topic and save it to HDFS.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |