504 gateway timeout in the browser when fetching the data from oracle database

Question

the title of the questions says every thing. this is happening in the cloud foundry environment. the application structure is like this there are two databases one is Mysql and other one is Oracle both are configured and its datasources are created well.

The ultimate goal is to fetch the data present in the oracle database into mysql database using Springboot application(ETL). It takes approx 30 mins to run the select query in the oracle DB as there are joins applied on tables with huge data around 10000 records will be fetched after the select query execution.

its working very well when i run the same in the local environment in eclipse IDE. but the problem comes up in cloud environment.

When i run the rest link in the browser in cloud env. It runs for around 10 to 15 mins and throws 504 httpstatus and what i observed is that in the background the data is fetched in the application. I can see that in the logs which is kept. but in the browser it throws the exception.

I would like to show my code if its mine but actually its a company work thats why i cannot show it..

could someone please help me out with this 504 error ?

Answer 1

Based on limited information that you have shared, it looks like you are invoking a REST API to run the ETL job which transfers the data from one database to another which is a time consuming/long running job. According to me, this is a good candidate for an asynchronous operation instead of a synchronous one. With as much details as you have given, I would advise that your REST API should trigger a new thread which takes up the job of running the ETL job in the background, and immediately return success in the response if the thread is successfully triggered. You can then have another operation on your UI that would show the status of the ETL job which would have to be maintained somewhere in the database and picked up from there using a service and shown on the UI.

NOTE: Gateway timeout is inevitable for such long running operations if done synchronously, unless you increase the timeout of the respective service, which I wouldn't suggest doing.