[英]MYSQL code to create transitive closure table fails
我正在使用當前的SNOMED數據和示例,並且想創建一個傳遞閉包表,但是mysql5.6默認服務器設置中的某些操作失敗。
對於那些不知道的人,SNOMED是醫學數據庫。 有210萬個關系和446697個概念。 該查詢在第二部分停滯了-因此我猜想它的內存不足。 但是我需要調整哪些設置? join_buffer_size?
這是代碼:
DELIMITER ;;
CREATE DEFINER=`snomed`@`localhost` PROCEDURE `createTc`()
BEGIN
drop table if exists tc;
CREATE TABLE tc (
source BIGINT UNSIGNED NOT NULL ,
dest BIGINT UNSIGNED NOT NULL
) ENGINE = InnoDB CHARSET=utf8;
insert into tc (source, dest)
select distinct rel.sourceid, rel.destinationid
from rf2_ss_relationships rel
inner join rf2_ss_concepts con
on rel.sourceid = con.id and con.active = 1
where rel.typeid = 116680003 # IS A relationship
and rel.active = 1;
REPEAT
insert into tc (source, dest)
select distinct b.source, a.dest
from tc a
join tc b on a.source = b.dest
left join tc c on c.source = b.source and c.dest = a.dest
where c.source is null;
set @x = row_count();
select concat('Inserted ', @x);
UNTIL @x = 0 END REPEAT;
create index idx_tc_source on tc (source);
create index idx_tc_dest on tc (dest);
END;;
DELIMITER ;
CREATE TABLE `rf2_ss_relationships` (
`id` bigint(20) unsigned NOT NULL,
`effectiveTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`active` tinyint(4) DEFAULT '1',
`moduleId` bigint(20) unsigned NOT NULL,
`sourceId` bigint(20) unsigned NOT NULL,
`destinationId` bigint(20) unsigned NOT NULL,
`relationshipGroup` bigint(20) unsigned NOT NULL,
`typeId` bigint(20) unsigned NOT NULL,
`characteristicTypeId` bigint(20) unsigned NOT NULL,
`modifierId` bigint(20) unsigned NOT NULL,
PRIMARY KEY (`id`,`effectiveTime`),
KEY `moduleId_idx` (`moduleId`),
KEY `sourceId_idx` (`sourceId`),
KEY `destinationId_idx` (`destinationId`),
KEY `relationshipGroup_idx` (`relationshipGroup`),
KEY `typeId_idx` (`typeId`),
KEY `characteristicTypeId_idx` (`characteristicTypeId`),
KEY `modifierId_idx` (`modifierId`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `rf2_ss_concepts` (
`id` bigint(20) unsigned NOT NULL,
`effectiveTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`active` tinyint(4) DEFAULT NULL,
`moduleId` bigint(20) unsigned NOT NULL,
`definitionStatusId` bigint(20) unsigned NOT NULL,
PRIMARY KEY (`id`,`effectiveTime`),
KEY `moduleId_idx` (`moduleId`),
KEY `definitionStatusId_idx` (`definitionStatusId`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
我不知道這是否是最好的答案,但是它確實可以工作...我更改了create table語法以在創建時添加索引-而不是在完成之后。 我將innodb_buffer_pool_size = 8G的mysqld設置更改為
CREATE TABLE tc (
source BIGINT UNSIGNED NOT NULL ,
dest BIGINT UNSIGNED NOT NULL,
KEY source_idx (source),
KEy dest_idx (dest)
) ENGINE = InnoDB CHARSET=utf8;
甚至在帶SSD的i7 mac上,執行速度仍然不快,但確實有效,並且傳遞關閉表為5180059行....
mysql> call createTc;
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 654161 |
+-------------------------+
1 row in set (1 min 55.13 sec)
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 1752024 |
+-------------------------+
1 row in set (3 min 5.60 sec)
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 2063816 |
+-------------------------+
1 row in set (10 min 42.07 sec)
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 275904 |
+-------------------------+
1 row in set (28 min 5.49 sec)
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 280 |
+-------------------------+
1 row in set (46 min 29.78 sec)
+-------------------------+
| concat('Inserted ', @x) |
+-------------------------+
| Inserted 0 |
+-------------------------+
1 row in set (1 hour 5 min 20.05 sec)
Query OK, 0 rows affected (1 hour 5 min 20.05 sec)
我使用這種遞歸方法。 在閱讀關系時,我已經將所有直接后代添加到概念對象的列表中(我使用了休眠),因此它們可用。
然后,通過瀏覽概念列表來啟動此遞歸函數。 參見示例。 它遍歷所有直接父母的清單中的每個概念:
for (Sct2Concept c : concepts.values()) {
for(Sct2Relationship parentRelation : c.getChildOfRelationships()){
addParentToList(concepts, sct2TransitiveClosureList, parentRelation, c);
}
}
如您所見,TransitiveClosure內存存儲區是一個Set,以便在智能且非常成熟的Java庫內部代碼上檢查唯一值。
private void addParentToList(Map<String, Sct2Concept> concepts, Set<Sct2TransitiveClosure> sct2TransitiveClosureList, Sct2Relationship parentRelation, Sct2Concept c){
if(!parentRelation.isActive())
return;
Sct2TransitiveClosure tc = new Sct2TransitiveClosure(parentRelation.getDestinationSct2Concept().getId(), c.getId());
if(sct2TransitiveClosureList.add(tc)){
Sct2Concept s = concepts.get(Long.toString(tc.getParentId()));
for(Sct2Relationship newParentRelation : s.getChildOfRelationships()){
addParentToList(concepts, sct2TransitiveClosureList, newParentRelation, c);
}
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.