[英]Importing a large xml file to Neo4j with Py2neo
I have a problem in importing a very big XML file with 36196662 lines. 我在导入一个包含36196662行的非常大的XML文件时遇到问题。 I am trying to create a Neo4j Graph Database of this XML file with Py2neo my xml file look like that:
我正在尝试用Py2neo我的xml文件创建这个XML文件的Neo4j图形数据库 ,如下所示:
http://imgur.com/pLylHeG http://imgur.com/pLylHeG
and My python code to import the xml data into Neo4j is like that: 和我的python代码将xml数据导入Neo4j是这样的:
from xml.dom import minidom
from py2neo import Graph, Node, Relationship, authenticate
from py2neo.packages.httpstream import http
http.socket_timeout = 9999
import codecs
authenticate("localhost:7474", "neo4j", "******")
graph = Graph("http://localhost:7474/db/data/")
xml_file = codecs.open("User_profilesL2T1.xml","r", encoding="latin-1")
xml_doc = minidom.parseString (codecs.encode (xml_file.read(), "utf-8"))
#xml_doc = minidom.parse(xml_file)
persons = xml_doc.getElementsByTagName('user')
label1 = "USER"
# Adding Nodes
for person in persons:
if person.getElementsByTagName("id")[0].firstChild:
Id_User=person.getElementsByTagName("id")[0].firstChild.data
else:
Name="NO ID"
print ("******************************USER***************************************")
print(Id_User)
print ("*************************")
if person.getElementsByTagName("name")[0].firstChild:
Name=person.getElementsByTagName("name")[0].firstChild.data
else:
Name="NO NAME"
# print("Name :",Name)
print ("*************************")
if person.getElementsByTagName("screen_name")[0].firstChild:
Screen_name=person.getElementsByTagName("screen_name")[0].firstChild.data
else:
Screen_name="NO SCREEN_NAME"
# print("Screen Name :",Screen_name)
print ("*************************")
if person.getElementsByTagName("location")[0].firstChild:
Location=person.getElementsByTagName("location")[0].firstChild.data
else:
Location="NO Location"
# print("Location :",Location)
print ("*************************")
if person.getElementsByTagName("description")[0].firstChild:
Description=person.getElementsByTagName("description")[0].firstChild.data
else:
Description="NO description"
# print("Description :",Description)
print ("*************************")
if person.getElementsByTagName("profile_image_url")[0].firstChild:
Profile_image_url=person.getElementsByTagName("profile_image_url")[0].firstChild.data
else:
Profile_image_url="NO profile_image_url"
# print("Profile_image_url :",Profile_image_url)
print ("*************************")
if person.getElementsByTagName("friends_count")[0].firstChild:
Friends_count=person.getElementsByTagName("friends_count")[0].firstChild.data
else:
Friends_count="NO friends_count"
# print("Friends_count :",Friends_count)
print ("*************************")
if person.getElementsByTagName("url")[0].firstChild:
URL=person.getElementsByTagName("url")[0].firstChild.data
else:
URL="NO URL"
# print("URL :",URL)
node1 = Node(label1,ID_USER=Id_User,NAME=Name,SCREEN_NAME=Screen_name,LOCATION=Location,DESCRIPTION=Description,Profile_Image_Url=Profile_image_url,Friends_Count=Friends_count,URL=URL)
graph.merge(node1)
My problem is when i run the code, it's take a long time to import this file almost a week to do that, so if can anyone help me to import data more faster than that i will be very grateful. 我的问题是当我运行代码时,需要很长时间才能导入这个文件差不多一个星期才能做到这一点,所以如果有人能帮助我更快地导入数据,那将非常感激。
NB: My laptop configuration is: 4Gb RAM, 500Gb Hard Disc, i5 注意:我的笔记本配置是:4Gb RAM,500Gb硬盘,i5
I think you should use a streaming parser, otherwise it might be even on the python side that you overflow on memory. 我认为你应该使用一个流解析器,否则它甚至可能在python一侧溢出内存。
Also I recommend doing transactions in Neo4j with batches of 10k to 100k updates per transaction. 另外,我建议在Neo4j中进行交易,每次交易批量为10k到100k。
Don't store "NO xxxx"
fields, just leave them off it is just a waste of space and effort. 不要存储
"NO xxxx"
字段,只要将它们关闭它只是浪费空间和精力。
I don't know how merge(node) works. 我不知道merge(node)是如何工作的。 I recommend creating a unique constraint on :User(userId) and using a cypher query like this:
我建议在:User(userId)上创建一个唯一约束,并使用如下的密码查询:
UNWIND {data} as row
MERGE (u:User {userId: row.userId}) ON CREATE SET u += {row}
where {data}
parameter is a list (eg 10k entries) of dictionaries with the properties. 其中
{data}
参数是具有属性的字典列表(例如10k条目)。
If you are importing data into a new database you may want to try the import-tool: https://neo4j.com/docs/operations-manual/current/#import-tool 如果要将数据导入新数据库,可能需要尝试导入工具: https : //neo4j.com/docs/operations-manual/current/#import-tool
In that case you should parse your XML file as you already do but instead of using py2neo to insert data into Neo4j, just write a CSV file and then call the import-tool afterwards. 在这种情况下,您应该像现在一样解析XML文件,而不是使用py2neo将数据插入Neo4j,只需编写一个CSV文件,然后再调用导入工具。
See below a possible way to do it: 请参阅下面的可能方法:
import csv
from xml.dom import minidom
def getAttribute(node,attribute,default=None):
attr = node.getElementsByTagName(attribute)[0]
return attr.firstChild.data if attr.firstChild else default
xml_doc = minidom.parse(open("users.xml"))
persons = xml_doc.getElementsByTagName('user')
users = []
attrs = ['name','screen_name','location','description','profile_image_url','friends_count','url']
mapping = {'user_id': 'user_id:ID(User)',
'name': 'name:string',
'screen_name': 'screen_name:string',
'location': 'location:string',
'description': 'description:string',
'profile_image_url': 'profile_image_url:string',
'friends_count': 'friends_count:int',
'url': 'url:string'}
with open('users.csv','w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=mapping.values())
writer.writeheader()
for person in persons:
user = {mapping[attr]: getAttribute(person, attr) for attr in attrs}
user[mapping['user_id']] = getAttribute(person, 'id')
writer.writerow(user)
Once you have converted the xml to a csv file, run the import-tool: 将xml转换为csv文件后,运行import-tool:
$ neo4j-import --into neo4j-community-3.0.3/data/databases/users.db --nodes:User users.csv
I guess you will also want to create relationships between nodes (?). 我想你也想在节点之间建立关系(?)。 You should read the import-tool docs and call the import-tool with csv files for both nodes and relationships
您应该阅读导入工具文档,并使用csv文件为两个节点和关系调用import-tool
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.