How to implement Custom Sink for a Embedded Flume Agent?

Question

I am building a spring boot standalone application that needs to consume messages from a remote server and write them to a Syslog Server . I am using Flume embedded agent to reliably log messages. But I am getting following error.

Caused by: org.apache.flume.FlumeException: Component type of com.security.flume.sink.SyslogSink is not in allowed types of [AVRO] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.checkAllowed(EmbeddedAgentConfiguration.java:308) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.validate(EmbeddedAgentConfiguration.java:182) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgentConfiguration.configure(EmbeddedAgentConfiguration.java:199) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgent.doConfigure(EmbeddedAgent.java:151) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0] at org.apache.flume.agent.embedded.EmbeddedAgent.configure(EmbeddedAgent.java:98) ~[flume-ng-embedded-agent-1.9.0.jar:1.9.0]

As per the documentation Embedded Agent only supports Avro Sink . Does it mean we can not even write a custom sink?

I am fairly new to Flume. I really appreciate your help on this issue.

Thanks!

Answer 1

this means that for embedded agent you just can use AvroSink as a custom sink. However, there are some techniques you can use to collect data from an external source using custom components:

• Create a custom client that is capable of establishing communication with Flume source that exists in prior, like AvroSources or SyslogTcpSource .

• Create a custom flume source having direct communication with a preexisting client through an IPC or RPC protocol.

I hope this answer is not too late and can help you with something. Thanks for Reading!!