New to Scala and apache flink, why does my map function run correctly REPL but fail in Flink

Question

I'm trying (unsuccessfully) to run a simple hello world type program in Apache Flink. The code takes a message from Apache Kafka adds a "." after each letter and prints the new string to stdout. The code correctly gets the message from Kafka, but the map function to add the "." fails. I've tried the function at the REPL prompt and the scala code works correctly there. Scala code:

scala> input = "hello"
input: String = hello
scala> val output = input.flatMap(value => value + ".")
output: String = h.e.l.l.o

flink program: flink code cut off line reads

val messageStream = env.addSource(new FlinkKafkaConsumer09("CL", new SimpleStringSchema, properties))

I cant figure out where I'm going wrong, I've tried the apache documentation to no avail. any help you could give me would be well received.

Answer 1

First of all I would recommend studying some basics about functional programming and operations like map/flatMap/reduce etc.

All of those mentioned functions apply to collections. In your scala example as @pedrofuria pointed out you apply the flatMap function to String which is collection of char s

In the flink example messageStream can be abstracted as a collection of strings, so to perform the operation you described you ought to do sth like:

val stream = messageStream.map(str => str.mkString("."))

I used the mkString instead of flatMap from your example because the former rather than

hello (as you wrote)

it produces

hello

But once more really start with basics of functional programming.

Answer 2

thanks for the help, i figured out the problem, sort of. Although the flatMap function works at the scala prompt, it doesn't work in Flink proper, as Flink requires FlatMap be passed a new FlatMapFunction with an override. Still unsure why it works at the scala prompt in flink, but the code now compiles and runs as expected.

Answer 3

Because flatMap in scala prompt is not the same with flatMap in your flink program.

flatMap in your scala prompt is just a function in scala.

The flink flatMap can be applied while input likes below:

val input = benv.fromElements(
            "To be, or not to be,--that is the question:--",
            "Whether 'tis nobler in the mind to suffer",
            "The slings and arrows of outrageous fortune",
            "Or to take arms against a sea of troubles,")
val counts = input
            .flatMap { _.toLowerCase.split("\\W+") }
            .map { (_, 1) }.groupBy(0).sum(1)

See: scala-shell