javascript：如何最有效地管理两个文件之间的数据导入

Question

I have two API endpoints that I am polling via a node.js / coffeescript script: an /addresses endpoint that returns a list of home addresses in a given city and a /homevalue endpoint that returns the value of a home at a given address.

I am polling each endpoint in series for a given city, let's say Buffalo. For auditing purposes, I am saving the content of each in local directories, at .../addresses/addresses.txt and .../homeValues/homeValues.txt . The script runs though all of the homes in a city, then saves these to the addresses directory, then polls the /homevalue endpoint and saves the results in a text file in the homeValues directory.

I then do some transformative work to convert both addresses and home values into a canonicalized format, saving each of these into a separate directory, .../canonicalAddresses and .../canonicalHomeValues . I then merge the canonical addresses and home values into a text file at .../unifiedAddresses/unifiedAddresses.txt

I cannot save these files as JSON, I have to save them in a text fileas a series of json objects, one per line. I am also doing this synchronously rather than async because I want to maintain an audit trail.

The canonicalized address file is a series of lines like:

{id: 12345, address: {...}}
{id: XYZAB, address: {...}}

The home values list is historical by year and is a series of lines like:

[{id: 12345, homevalue: {year: 1990,...}, {id: 12345, homevalue: {year: 1991,...}}...]
[{id: XYZAB, homevalue: {year: 1990,...}, {id: 12346, homevalue: {year: 1991,...}}...]

This is my greatly simplified pseudocode for that merge, which requires that I read both .../addresses/addresses.txt and .../homeValues/homeValues.txt from disk:

canonicalizedHomeValuesFile = "..."
canonicalizedAddressesFile = "..."
unifiedAddressFile = "..."

getHomeValue = (addressID) ->
   fs.readFileSync(canonicalizedHomeValuesFile).toString().split('\n').forEach((homevalue)=>
       << return the canonicalized home value if homevalue.ID is addressID >>
   )

fs.readFileSync(canonicalizedAddressFile).toString().split('\n').forEach((address)=>   
   address.value = getHomeValue(address.ID)
   fs.appendFileSync(unifiedAddressFile, JSON.stringify(address) + "\n")
) 

This approach works fine for small numbers of houses but is insanely slow to unify large numbers of addresses. For about 2000 houses, this approach takes upwords of 4 minutes per house.

It seems to me the real bottleneck is the getHomeValue() function. What is a more efficient way to approach that lookup?

Answer 1

If the data is large enough it might be worth preloading the objects and then matching them using a binary search. It looks like you are loading the file from disk each time you get the home value. It also looks like you are writing to disk with each iteration in readFileSync. If that is the case you might consider writing the file after all transactions are complete. I would minimize drive access as much as possible by batching the load and save.