通过API导入大数据

Question

I am trying to develop a system in which I will sync my database with 3-Party database by provided API. The API has format in which we can provide From-Date and To-Date

Problems

1. There is no API which gives me only modified records.
2. The data is too large (1000 records/day average)
3. Need a scheduler so all the records are updated automatically
4. I also need to keep track of modified records(which is the biggest problem as I can't get them by modified date)

Note : As per the previous requirement i Have already developed the system in which i can Specify the From-Date and To-Date the record get updated (its completed with the GUI no ajax was uses). and even if I request 1 day records the system get time out error.

NOTE 2 : I really should no say but the client is too strict (DUMB);( he just need the solution nothing else will do

Answer 1

Assuming that the data doesn't need to be "fresh" can you not write a process to run hourly / nightly fetching that days worth of data and processing it into your DB?

Obviously this would only work if you're sure previous records are not updated?

Does the API provide batches?

Why did you you choose a web client with Ajax to process this data? Would a windows / console application be better suited?

Answer 2

If the data is too big to retrieve by any given query, you're just going to have to do it by ID. Figure out a good size (100 records? 250?), and just spin through every record in the system by groups of that size.

You didn't say if you're pulling down data, pushing up data, or both. If you're only pulling it down, then that's the best you can do, and it will get slower and slower as more records are added. If you're only pushing it, then you can track a "pushed date". If it's both, how do you resolve conflicts?