Large Table Export to Flat File - Multiple Files

Question

Please note: I can only use solutions that involve SSIS and/or Stored Procedures due to limits placed on our team and environment. As much as I understand and agree there are better solutions, we are limited to only things that involve SSIS and commands that can be executed inside a stored procedure.

We are having an issue, and I am not exactly sure how to solve it and hoping others can give me suggestions, pointers, etc. We have a process that runs and exports data to a file. The file is 16GB in size (please, I understand this isn't the best solution, but again, due to our environment, we can only create a file, place it on a server and have other teams pick it up, as much as we would like to have direct connections, ETL tools, etc, which we are planning on implementing in the next budget year, but unfortunately that doesn't help the immediate issue at hand). The downside is the team that ingests this file, can only ingest 5GB files. As such, we need to break out our current process into multiple files.

Our current process is simply an SSIS package that "SELECTS * FROM TABLE" and then writes the results to a file. I am having a terrible time figuring out how to break this process into multiple steps so we can create multiple files. For instance "SELECT TOP 1000000 FROM TABLE", then output that to a file and grab the next 1,000,000 records, and output those to a file, until all records in the table have been exported. Also, we would need to update the filename with each iteration: file_1, file_2, file_3, etc.

Any help and/or pointers to research would be greatly appreciate.

Answer 1

Please try the following conceptual example.

SQL

-- DDL and data population, start
DECLARE @tbl TABLE (
   ID INT PRIMARY KEY
   , [Description] VARCHAR(100) NOT NULL
   );

INSERT INTO @tbl
VALUES (1, 'One')
   , (2, 'Two')
   , (3, 'Three')
   , (4, 'Four')
   , (5, 'Five')
   , (6, 'Six')
   , (7, 'Seven')
   , (8, 'Eight')
   , (9, 'Nine')
   , (10, 'Ten')
   , (11, 'Elleven')
   , (12, 'Twelve')
   , (13, 'Thirteen');
-- DDL and data population, end

DECLARE @BatchNo INT = 1
   , @NumberOfBatchesTotal DECIMAL = 3
   , @RowsPerBatch INT
   , @RowTotal INT;

-- To calculate # of rows per each Batch
SET @RowTotal = (SELECT COUNT(*) FROM @tbl);
SET @RowsPerBatch = CEILING(@RowTotal/@NumberOfBatchesTotal);

WHILE @BatchNo <= @NumberOfBatchesTotal
BEGIN
   -- call bcp.exe here passing the SELECT statement below, or package it as a SP with parameters
   SELECT * FROM @tbl
   ORDER BY ID
      OFFSET (@BatchNo - 1) * @RowsPerBatch ROWS
      FETCH NEXT @RowsPerBatch ROWS ONLY;

   SET @BatchNo += 1;
END;