将PostgreSQL本地文件分块读取到R中并导出到.csv

Question

I have a local file in PostgreSQL format that I would like to read into R into chunks and export it as .csv .

I know this might be a simple question but I'm not at all familiar with PostgreSQL or SQL . I've tried different things using R libraries like RPostgreSQL , RSQLite and sqldf but I couldn't get my head around this.

Answer 1

If your final goal is to create a csv file, you can do it directly using PostgreSQL.

You can run something similar to this: COPY my_table TO 'C:\\my_table.csv' DELIMITER ',' CSV HEADER;

Sorry if I misunderstood your requirement.

Answer 2

The requirement is to programmatically create a very large .csv file from scratch and populate it from data in a database? I would use this approach.

Step 1 - isolate the database data into a single table with an auto incrementing primary key field. Whether you always use the same table or create and drop one each time depends on the possibility of concurrent use of the program.

Step 2 - create the .csv file with your programming code. It can either be empty, or have column headers, depending on whether or not you need column headers.

Step 3 - get the minimum and maximum primary key values from your table.

Step 4 - set up a loop in your programming code using the values from Step 3. Inside the loop:

• query the table to get x rows
• append those rows to your file
• increment the variables that control your loop

Step 5 - Do whatever you have to do with the file. Don't try to read it with your programming code.