[英]MySQL Selecting million records to generate urls
我目前從不同的表格中獲得200萬條記錄,以生成一個用於創建站點地圖的網址。 該腳本占用了太多資源,並占用了100%的服務器性能
詢問
SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1
//我將其存儲在$ url_list數組中,然后創建站點地圖..但是這需要時間和大量資源
// i嘗試使用LIMIT 0,50000
批量獲取數據,但獲取分頁的最大行需要時間。 代碼也不適合我必須運行兩個具有大數據的查詢
$url_list = array();
$maxrow = SELECT COUNT(*) AS max from (
SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1) as tmp
$limit = 50,000;
$bybatch = ceil($maxrow/$limit);
$start = 0;
for($i = 0;$i < $bybatch; $i++){
// run query and store to $result
(SELECT CONCAT("/url/profile/id/",u.id,"/",nickname) as url FROM users AS u
UNION ALL
Select CONCAT("url/city/", c.id, "/paramId/",p.id,"/",Replace(p.title, " ", "+"),"/",r.region_Name,"/",c.city_Name) AS url
From city c
Join region r On r.id = c.id_region
Join country country On country.id = c.id_country
cross join param p
Where country.used = 1
And p.active = 1 LIMIT $start,$limit);
$start += $limit;
//push to $url_list
$url_list = array_push($result);
}
//完成后,我將使用它來創建站點地圖
$linkCount = 1;
$fileNomb = 1;
$i = 0;
foreach ($url_list as $ul) {
$i += 1;
if ($linkCount == 1) {
$doc = new DOMDocument('1.0', 'utf-8');
$doc->formatOutput = true;
$root = $doc->createElementNS('http://www.sitemaps.org/schemas/sitemap/0.9', 'urlset');
$doc->appendChild($root);
}
$url= $doc->createElement("url");
$loc= $doc->createElement("loc", $ul['url']);
$url->appendChild($loc);
$priority= $doc->createElement("priority",1);
$url->appendChild($priority);
$root->appendChild($url);
$linkCount += 1;
if ($linkCount == 49999) {
$f = fopen($this->siteMapMulti . $fileNomb .'.xml', "w");
fwrite($f,$doc->saveXML());
fclose($f);
$linkCount = 1;
$fileNomb += 1;
}
}
還有更好的方法嗎? 還是要加快性能?
添加
為什么這比sql查詢要快,但卻消耗了100%的服務器資源和性能
$this->db->query('SELECT c.id, c.city_name, r.region_name, cr.country_name FROM city AS c, region AS r, country AS cr WHERE r.id = c.id_region AND cr.id = c.id_country AND cr.id IN (SELECT id FROM country WHERE use = 1)');
$arrayCity = $this->db->recordsArray(MYSQL_ASSOC);
$this->db->query('SELECT id, title FROM param WHERE active = 1');
$arrayParam = $this->db->recordsArray(MYSQL_ASSOC);
foreach ($arrayCity as $city) {
foreach ($arrayParam as $param) {
$paramTitle = str_replace(' ', '+', $param['title']);
$url = 'url/city/'. $city['id'] .'/paramId/'. $param['id'] .'/'. $paramTitle .'/'. $city['region_name'] .'/'. $city['city_name'];
$this->addChild($url);
}
}
我建議您不要使用UNION
而只發出兩個分開的查詢。 它將加速查詢本身。 另外,正如您上面提到的,最好分批獲取數據。
最后,不要收集內存中的所有數據。 立即將其寫入循環中的文件。
只是在開始時打開文件,在循環中寫入每個URL條目,然后在結束時關閉文件。
—打開文件進行寫入
—計數查詢用戶表
—用LIMIT
循環進行幾次選擇(如您已經做的那樣)
—就在這里循環, while ($row = mysql_fetch_array())
將每一行寫入文件
然后再針對另一張表重復這種算法。 實現將數據寫入文件的功能將很有用,因此您可以調用該功能並遵守DRY原理。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.