在 ETL 项目中进行自动化端到端测试的最佳方法是什么？

Question

We have an ETL Project wherein Data from one UI application flows down in a series of applications(Transformation/Logic).

The overall motto is to see the data of 'UI 1' in 'UI 2'. 'Spring Java Server 1' calls the API of UI1 Server and the data gets flow into different servers and persisted in the database. A script also persists some data in the database. 'UI 2 with Backend server' gets the data from database and show it in the UI.

Now the question, I am looking for the best possible way to perform E2E testing. We'll start by triggering 'Spring Java Server 1' which calls APIs of 'UI 1 with BackendServer 1' and at the end using selenium see the data in 'UI 2'.

I am thinking about creating a separate Java Application with UI just for testing which triggers the 'Server 1' and with selenium verifies the UI2 But the problem is with Script. After the database, we have to manually run script and then wait for it to complete. How we can automate manually running the script with some kind of state Management?

Please suggest more on the architecture of E2E or how can i enhance this E2E setup. And what tools/framework i can use.

Answer 1

With regards to automating the execution of the script, look at PowerShell. It is a powerful tool for automating repetitive and time-consuming tasks, such as executing scripts automatically instead of doing it manually. You also create a custom PowerShell script that connects to SQL Server to run commands. If you're looking for ETL automation tools to track the data from the source, through the ETL processing until it loads to the target, there are quite a few options available that you will need to evaluate to see which one is right for you. Look at dbt, RightData, QuerySurge, Datagaps ETL Validator, SSISTester, TestBench, DbFit, or AnyDbTest.