We are planning to migrate our existing flow in Apache NiFi to Informatica PowerCenter. Could you guys please suggest whether it is feasible or not. P ...
We have a pipeline to extract embeddings (feature vectors) from images stored in Cloud Storage bucket and insert into a BigQuery table. We're consist ...
Using ETL tool to run syncs from SQL Server to MySQL, often encounter errors like this: Exception during SQL execution: com.mysql.cj.jdbc.exceptions. ...
We are repointing a large number (>1000) DataStage jobs from one database to another. As part of this, we will need to make the same changes to a s ...
I have a map and the keys are strings. If the key contains the word "kg" I want to multiply the value by 2.2 and then replace "kg" with "lb" in the ke ...
I'm trying out Airflow with a basic ETL, I've put the .py in dags folder and hit up my docker composer, looking at the logs in Airflow it says "FileNo ...
Implementing scd2 in dbt using aws redshift. How do I define conditional natural keys? unique_id = ['crm_id', 'curr_recrd_flg', 'actve_flg'] I want ...
I'm working on an ETL pipeline in Azure Synapse. In the previous version I used an Array set as a parameter of the pipeline and it contained JSON obj ...
I'm currently trying to create txt files from all tables in the dbo schema I have like 200s-300s tables there, so it would takes up too much times to ...
I am building a DWH based on data I am collecting from an ERP API. currently, I am fetching the data from the API based on an incremental mechanism I ...
enter image description here I have tried traditional approach of using Agg (Group By: ID, Store Name) and Max(Each Object) columns separately. Then ...
Is it possible to run simple select statements dynamically via ADF? I want to copy tables from source to sink (sql to sql) using ADF. I am using the ...
I want to create tables in a Postgres database using Python's SQLAlchemy package and insert data from a dataframe into them. I also want to assign for ...
I have a stored procedure that takes values from an ADF pipeline to write watermark values in a SQL Server table. The new watermark value essentially ...
I am new to DBT and I am facing a design challenge. The thing is I have 20+ data integrations. Each integration is feeding data into one Postgres DB. ...
Working on a side-project which aggregates data from various websites, sanitizes the input data, then stores it in postgres. Currently, I have to imp ...
I would like to merge data from different data sources (ERP system, Excel files) with the ADF and make it available in an AzureSQLDB for further analy ...
If columnA in table-a and not in table-b direct map table-a.columnnA to target table-c If columnA in both table-a and table-b AND if table-a.columnB ...
I've been asked to migrate on-premises Python ETL scripts that live on a syslog box over to AWS. These scripts run as cron-jobs and output logs that a ...
I am unable to run newly created AWS Glue Crawler. I followed IAM Role guide at https://docs.aws.amazon.com/glue/latest/dg/create-an-iam-role.html?icm ...