将多个数据库查询的结果合并到一个优雅的表中的逻辑或过程是什么？

Question

Basically, I have three tables that contain all the data that I want, but I am having to do some crazy JOIN and WHERE statements that are not working for me. I have finally resorted to using temporary tables, but I was wondering if there is a more long term solution.

The Situation: We Pull large amounts of data via SOAP to our database, we have no control on how the data is organized, combined together, labeled and etc., we need to split it up the best we can so that it can eventually become useful to us.

What I am asking is how do the pro's or etc. "Prep" data so that it can either be finally inserted into a usefully table via other tables quickly, and how does it stay updated with the flow of new data coming in? What is the terminology? What should I research?

Thanks in advance!

Answer 1

The terminology I use for that is for preparing the data and getting ready for insertion is "staging" the data. We typically insert/update rows into temporary staging tables.

We massage and tweak the data in the staging tables, assigning foreign keys, fixing malformed data, splitting big multiuse fields into individual fields, and so on, to get the data cleaned up BEFORE the rows are inserted into the actual target tables.

(I don't know that this is a standard terminology, others may refer to it differently.)

---

FOLLOWUP

To improve query performance for complex data, we sometimes store pre-joined and pre-calculated results. Basically, we populate a table with "query ready" results, to make for much simpler queries of historic data. The big drawback to this is that we now have redundant data, which can become "out-of-sync" with the operational data. We use a scheduled (nightly) process to re-populate these tables.

(I'm unsure of the industry standard jargon for these types of tables.)

In terms of researching this, these approaches are going to be described in articles/white papers on "data warehouse" and "data mart". And that's almost always described as " ETL " The three major steps: Extract - Transform - Load . There's also a lot of noise in the industry press about "data mining" and "big data".