I am trying to develop a system in which I will sync my database with 3-Party database by provided API. The API has format in which we can provide From-Date and To-Date
Problems
Note : As per the previous requirement i Have already developed the system in which i can Specify the From-Date and To-Date the record get updated (its completed with the GUI no ajax was uses). and even if I request 1 day records the system get time out error.
NOTE 2 : I really should no say but the client is too strict (DUMB);( he just need the solution nothing else will do
Assuming that the data doesn't need to be "fresh" can you not write a process to run hourly / nightly fetching that days worth of data and processing it into your DB?
Obviously this would only work if you're sure previous records are not updated?
Does the API provide batches?
Why did you you choose a web client with Ajax to process this data? Would a windows / console application be better suited?
If the data is too big to retrieve by any given query, you're just going to have to do it by ID. Figure out a good size (100 records? 250?), and just spin through every record in the system by groups of that size.
You didn't say if you're pulling down data, pushing up data, or both. If you're only pulling it down, then that's the best you can do, and it will get slower and slower as more records are added. If you're only pushing it, then you can track a "pushed date". If it's both, how do you resolve conflicts?
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.