January 18, 2013

How to import a large csv file into MySQL

I've been fooling around with big data which in this case is defined as an 188 mb large csv-file. You can't read the file each time you are using it in you favorite programming language since it will take too long time to load it, so what you can do is to insert the data from the file into an MySQL database. You can't import it into MySQL like you would have done with a smaller file since your computer will probably run out of memory, but what you can do is to write a smaller SQL-code and the file will be imported in a matter of  seconds. It took me some time to Google the answer, and hopefully it will be faster to find this post, anyway, here's the magic code:

LOAD DATA LOCAL INFILE 'c:/Projekt/Big Data/events.csv' INTO TABLE database.table FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES;

IGNORE 1 LINES - will skip the first line in the csv file which is the description.

1 comment: