使用Python将多个CSV文件导入MySQL

Question

I am trying to use examples that are fun/interesting to me in order to learn MySQL and Python. I am new to both, as I am a finance guy by trade. I have learned quite a bit along the way, but I am at a point where I am not sure where to go.

The data files I am looking to import are contained in a zip file, hosted: here

I extracted the directory, are there are well over 1000 files. In the code below, I am simply trying to reference one of the files individually to parse it into MySQL.

I am fully aware that there has to be an easier way to go about this, but I am learning, and I am sure my code reflects it (you can see that I typed in some basic prints to see that my code was correct).

Any help you can provide will be GREATLY appreciated. Essentially, when it comes to Python, I am looking as it as a hobby and a way to get at the large amount of data on the web that I want to play with. I pasted my code below so you can see where I am coming from.

• Brock

PS Sorry about the code below, can't figure out a better way to post it.

The code below is my create table script

> DROP TABLE IF EXISTS `nba`.`event`;
CREATE TABLE  `nba`.`event` (
  `a1` varchar(45) DEFAULT NULL,
  `a2` varchar(45) DEFAULT NULL,
  `a3` varchar(45) DEFAULT NULL,
  `a4` varchar(45) DEFAULT NULL,
  `a5` varchar(45) DEFAULT NULL,
  `h1` varchar(45) DEFAULT NULL,
  `h2` varchar(45) DEFAULT NULL,
  `h3` varchar(45) DEFAULT NULL,
  `h4` varchar(45) DEFAULT NULL,
  `h5` varchar(45) DEFAULT NULL,
  `period` int(11) DEFAULT NULL,
  `time` time DEFAULT NULL,
  `team` varchar(3) DEFAULT NULL,
  `etype` varchar(15) DEFAULT NULL,
  `assist` varchar(45) DEFAULT NULL,
  `away` varchar(45) DEFAULT NULL,
  `block` varchar(45) DEFAULT NULL,
  `entered` varchar(45) DEFAULT NULL,
  `home` varchar(45) DEFAULT NULL,
  `left` varchar(45) DEFAULT NULL,
  `num` int(11) DEFAULT NULL,
  `opponent` varchar(45) DEFAULT NULL,
  `outof` varchar(45) DEFAULT NULL,
  `player` varchar(45) DEFAULT NULL,
  `points` int(11) DEFAULT NULL,
  `possession` varchar(45) DEFAULT NULL,
  `reason` varchar(50) DEFAULT NULL,
  `result` varchar(10) DEFAULT NULL,
  `steal` varchar(45) DEFAULT NULL,
  `type` varchar(30) DEFAULT NULL,
  `x` varchar(2) DEFAULT NULL,
  `y` varchar(2) DEFAULT NULL,
  `gameid` varchar(15) NOT NULL,
  `seqnum` int(11) NOT NULL AUTO_INCREMENT,
  `updated` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`seqnum`,`gameid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Datafiles of play by play from basketballgeek.com';

Here is my python script. I put the table above in a schema called NBA. I set my table up so that, I believe, a sequence number is created for every record entered into the db. I pass it the gameid as a string, and also assume that the timestamp will be created for every insert statement. I know there is an error, but I can't figure out what it is.

 sql = """LOAD DATA INFILE '%s' INTO TABLE event FIELDS TERMINATED BY ',' LINES TERMINATED BY '\\\\n';""" % path print sql try: cursor.execute(sql) db.commit() except: print "ERROR" db.rollback() db.close() 

Answer 1

Why not use MySQLs own CSV import functionality?

http://dev.mysql.com/doc/refman/5.0/en/load-data.html

LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n';

Answer 2

You'll want to output the actual error, like this:

try:
    cursor.execute(sql)
    db.commit()
except StandardError, e:
    print e
    db.rollback()

For example, when I execute the above, I get following output

(29, "File '/opt/mysql/data/51/test/data.csv' not found (Errcode: 2)")

Note that you want to specify the LOCAL keyword for LOAD DATA. Without it, it will assume that the file is locate on the machine running your MySQL server, which might not be the same as the one running your script?

Answer 3

I guess you have an error similar to what I had.

try

LOAD DATA  LOCAL INFILE 'data.txt' INTO TABLE tbl_name
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n';

instead of

LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n';

This syntax is to make sure the data is being read relative to the client and not the server.