How to add a data quality check utility in awe glue.?
Question
如何在 aws 胶水中添加仅检查数据质量（如空值、正确的数据类型等）的作业
2 answers
solution1
1 2021-05-11 15:47:30
you can check pyDeequ library too. It enables you to implement python code to DQ checks.
solution2
-1 2019-09-10 17:22:26
You can write pyspark glue job for quality check. You can also use AWS Deequ library to do it but it is in scala.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Data ingestion with Kafka and Hadoop - how to avoid data duplication that can result from quality check failure?
How do you handle data validation in the model using ColdFusion Model-Glue?
Quality Control: Check file names in a folder against filenames in a table
Define Data Quality Rules for Big Data
How to UPIVOT all columns in a table and aggregate into Data Quality/ Validation Metrics? SQL SNOWFLAKE
How to check for correct data input in C
How to check data present or not in html id
How to check that the data is 'null' in django view?
How to add validate data search
How to create Validation utility for variable validation in spring boot or java
Related Tags