[英]Synchronous Multiple Requests in foor loop with Node js
我有點用Java腳本開始,我需要幫助來弄清楚如何在遍歷for循環時使此代碼同步。 基本上,我正在做的是在for循環內發出多個POST請求,然后使用X-Ray庫即時刪除數據,最后將結果保存到Mongo數據庫中。 輸出正常,但以無序方式出現,突然掛起,我必須使用ctrl + C強制關閉。 這是我的功能:
function getdata() {
const startYear = 1996;
const currentYear = 1998; // new Date().getFullYear()
for (let i = startYear; i <= currentYear; i++) {
for (let j = 1; j <= 12; j++) {
if (i === startYear) {
j = 12;
}
// Form to be sent
const form = {
year: `${i}`,
month: `${j}`,
day: '01',
};
const formData = querystring.stringify(form);
const contentLength = formData.length;
// Make HTTP Request
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded',
},
uri: 'https://www.ipma.pt/pt/geofisica/sismologia/',
body: formData,
method: 'POST',
}, (err, res, html) => {
if (!err && res.statusCode === 200) {
// Scrapping data with X-Ray
x(html, '#divID0 > table > tr', {
date: '.block90w',
lat: 'td:nth-child(2)',
lon: 'td:nth-child(3)',
prof: 'td:nth-child(4)',
mag: 'td:nth-child(5)',
local: 'td:nth-child(6)',
degree: 'td:nth-child(7)',
})((error, obj) => {
const result = {
date: obj.date,
lat: obj.lat.replace(',', '.'),
lon: obj.lon.replace(',', '.'),
prof: obj.prof == '-' ? null : obj.prof.replace(',', '.'),
mag: obj.mag.replace(',', '.'),
local: obj.local,
degree: obj.degree,
};
// console.log(result);
upsertEarthquake(result); // save to DB
});
}
});
}
}
}
我想我必須使用promise或callbacks,但是我不明白該怎么做,我已經嘗試使用異步等待但沒有成功。 如果需要任何其他信息,請告訴我,謝謝。
您正在循環內調用請求。
異步函數是在主線程邏輯結束后獲取結果(又稱為回調函數,接收響應)的函數。
這樣,如果我們有這個:
for (var i = 0; i < 12; i++) {
request({
data: i
}, function(error, data) {
// This is the request result, inside a callback function
});
}
邏輯將是在調用回調之前對12個request
進行處理,因此將在所有主循環運行之后對回調進行堆棧和調用。
無需輸入所有ES6生成器的東西(因為我認為這會使它變得更復雜,並且還可以從低層次學習發生了什么對您來說更好),您要做的就是調用request
,等待他的回調要調用的函數並調用下一個request
。 怎么做? 有很多方法,但是我通常會這樣:
var i= 0;
function callNext() {
if (i>= 12) {
requestEnded();
} else {
request({
data: i++ // Increment the counter as we are not inside a for loop that increments it
}, function(error, data) {
// Do something with the data, and also check if an error was received and act accordingly, which is very much possible when talking about internet requests
console.log(error, data);
// Call the next request inside the callback, so we are sure that the next request is ran just after this request has ended
callNext();
})
}
}
callNext();
requestEnded() {
console.log("Yay");
}
在這里,您會看到邏輯。 您有一個名為callNext
的函數,如果不需要更多呼叫,它將進行下一個呼叫或呼叫requestEnded
。
當request
被稱為內callNext
,它會等待要接收的回調(這將是異步的,在將來某個時候),將處理接收到的數據,然后回調里面你叫他再打電話callNext
。
無需循環,您可以使用開始和結束年份創建一個數組,然后將其映射到您的請求配置,然后將其結果映射到x射線返回的值(x射線返回一個像這樣的promise,因此不需要回調)。 然后使用返回諾言的函數將刮擦的結果放入mongodb中。
如果拒絕,則創建一個Fail
類型的對象並使用該對象進行解析。
使用Promise.all並行啟動所有請求,x射線和mongo,但使用節流限制活動請求的數量。
這是代碼中的樣子:
//you can get library containing throttle here:
// https://github.com/amsterdamharu/lib/blob/master/src/index.js
const lib = require('lib');
const Fail = function(details){this.details=details;};
const isFail = o=>(o&&o.constructor)===Fail;
const max10 = lib.throttle(10);
const range = lib.range;
const createYearMonth = (startYear,endYear)=>
range(startYear,endYear)
.reduce(
(acc,year)=>
acc.concat(
range(1,12).map(month=>({year,month}))
)
,[]
);
const toRequestConfigs = yearMonths =>
yearMonths.map(
yearMonth=>{
const formData = querystring.stringify(yearMonth);
return {
headers: {
'Content-Length': formData.length,
'Content-Type': 'application/x-www-form-urlencoded',
},
uri: 'https://www.ipma.pt/pt/geofisica/sismologia/',
body: formData,
method: 'POST',
};
}
);
const scrape = html =>
x(
html,
'#divID0 > table > tr',
{
date: '.block90w',
lat: 'td:nth-child(2)',
lon: 'td:nth-child(3)',
prof: 'td:nth-child(4)',
mag: 'td:nth-child(5)',
local: 'td:nth-child(6)',
degree: 'td:nth-child(7)'
}
);
const requestAsPromise = config =>
new Promise(
(resolve,reject)=>
request(
config,
(err,res,html)=>
(!err && res.statusCode === 200)
//x-ray returns a promise:
// https://github.com/matthewmueller/x-ray#xraythencb
? resolve(html)
: reject(err)
)
);
const someMongoStuff = scrapeResult =>
//do mongo stuff and return promise
scrapeResult;
const getData = (startYear,endYear) =>
Promise.all(
toRequestConfigs(
createYearMonth(startYear,endYear)
)
.map(
config=>
//maximum 10 active requests
max10(requestAsPromise)(config)
.then(scrape)
.then(someMongoStuff)
.catch(//if something goes wrong create a Fail type object
err => new Fail([err,config.body])
)
)
)
//how to use:
getData(1980,1982)
.then(//will always resolve unless toRequestConfigs or createYearMonth throws
result=>{
//items that were successfull
const successes = result.filter(item=>!isFail(item));
//items that failed
const failed = result.filter(isFail);
}
)
抓取經常發生的是,目標站點不允許您在y期間發出超過x個請求,並開始將IP列入黑名單,如果超過該請求,則拒絕提供服務。
假設您希望每5秒限制10個請求,然后可以將上述代碼更改為:
const max10 = lib.throttlePeriod(10,5000);
其余代碼相同
你有sync for...loop
,里面有async methods
問題。
解決此問題的一種干凈方法是
ES2017
async/await
語法
假設您要在upsertEarthquake(result)
之后停止每次迭代, upsertEarthquake(result)
更改代碼。
function async getdata() {
const startYear = 1996;
const currentYear = 1998; // new Date().getFullYear()
for (let i = startYear; i <= currentYear; i++) {
for (let j = 1; j <= 12; j++) {
if (i === startYear)
j = 12;
// Form to be sent
const form = {
year: `${i}`,
month: `${j}`,
day: '01',
};
const formData = querystring.stringify(form);
const contentLength = formData.length;
//Make HTTP Request
await new Promise((next, reject)=> {
request({
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/x-www-form-urlencoded',
},
uri: 'https://www.ipma.pt/pt/geofisica/sismologia/',
body: formData,
method: 'POST',
}, (err, res, html) => {
if (err || res.statusCode !== 200)
return next() //If there is an error jump to the next
//Scrapping data with X-Ray
x(html, '#divID0 > table > tr', {
date: '.block90w',
lat: 'td:nth-child(2)',
lon: 'td:nth-child(3)',
prof: 'td:nth-child(4)',
mag: 'td:nth-child(5)',
local: 'td:nth-child(6)',
degree: 'td:nth-child(7)',
})((error, obj) => {
const result = {
date: obj.date,
lat: obj.lat.replace(',', '.'),
lon: obj.lon.replace(',', '.'),
prof: obj.prof == '-' ? null : obj.prof.replace(',', '.'),
mag: obj.mag.replace(',', '.'),
local: obj.local,
degree: obj.degree,
}
//console.log(result);
upsertEarthquake(result); // save to DB
next() //This makes jump to the next for... iteration
})
})
}
}
}
}
我假設upsertEarthquake
是一個異步函數,或者是即upsertEarthquake
即忘的類型。
如果有錯誤,可以使用next()
,但是如果要中斷循環,請使用reject()
if (err || res.statusCode !== 200)
return reject(err)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.