Developing and deploy jobs in Apache Flink

Question

we started to develop some jobs with Flink. Our current dev / deployment process looked like this: 1. develop code in local IDE and compile 2. upload jar-file to server (via UI) 3. register new job

However it turns out that the generate jar-file is ~70MB and upload process takes a few minutes. What is the best way to speed up development (eg using a on-server ide?)

Answer 1

One solution is to use a version control system and after committing and pushing your changes, you might build the jar on the server itself. You could write a simple script for this.

The other solution, which would take time and effort is to set up a CI CD Pipeline which would automate the entire process and much manual effort would be minimised.

Also, try not to use fat jar to minimise the jar size if you have to scp your jar to the cluster.

Answer 2

First off, uploading 70 mb shouldn't take minutes nowadays. You might want to check your network configurations. Of course, if your internet connection is not good, you can't help it.

In general, I'd try to avoid cluster test runs as much as possible. It's slow and hard to debug. It should only ever be used for performance tests or right before releasing into production.

All logic should be unit tested. The complete job should be integration tested and ideally you'd also have an end-to-end test. I recommend to use a docker-based approach for external systems and use things like test containers for Kafka, such that you are able to run all tests from your IDE.

Going onto the test cluster should then be a rare thing. If you find any issue that has not been covered by your automatic tests, you need to add a new test case and solve it locally, such that there is a high probability that it will be solved on the test cluster.

edit (addressing comment):

If it's much easier for you to write a Flink job to generate data, then it sounds like a viable option. I'm just fearing that you would also need to test that job...

It rather sounds like you want to have an end-2-end setup where you run several Flink jobs in succession and compare the final results. That's a common setup for complex pipelines consisting of several Flink jobs. However, it's rather hard to maintain and may have unclear ownership status if it involves components from several teams. I like to rather solve it by having tons of metrics (how many records are produced, how many invalid records are filtered in each pipeline...) and having specific validation jobs that just assess the quality of (intermediate) output (potentially involving humans).

So from my experience, either you can test the logic in some local job setup or it's so big that you spend much more time in setting and maintaining the test setup than actually writing production code. In the latter case, I'd rather trust and invest in the monitoring and quality assurance capabilities of (pre-)production that you need to have anyways.

If you really just want to test one Flink job with another, you can also run the Flink job in testcontainers, so technically it's not an alternative but an addition.