SQL Server - 将大量记录导出到多个 csv 文件中

Question

I'm running a query which gives me more than 8 millions of records. I've exported the result into a .csv file but the file is way too big to be processed (8GB...).

What I'm trying to do is to split the result csv into multiple files but with one condition :

There is a column name "Locator" which represents an ID. I've managed to group my records in order to have that kind of result :

Locator | Name | LastName
___________________________ 
ABCDEFH | Foo  | Oof 
ABCDEFH | Foo2 | Oof2 
ABCDEFH | Foo3 | Oof3 
TUVWXYZ | Mark | Mark
TUVWXYZ | Mark2| Mark2
...     | ...  | ...

So what I want to do is basically to split the records without splitting the groupings... Is is possible to do that?

EDIT : Here's the query with the NTILE :

with locator as 
(
    select distinct
    pnrlctrnum,
    NTILE(8) OVER(ORDER BY pnrlctrnum) as Tile_Num

from ttddocseg, ttddoc, ttdhdr
where ttddocseg.tdtrxnum = ttddoc.tdtrxnum and ttdhdr.tdtrxnum = ttddoc.tdtrxnum
)
select ttdhdr.pnrlctrnum, ttddoc.*, ttddocseg.*
from ttddocseg, ttddoc, ttdhdr
inner join locator on locator.pnrlctrnum = ttdhdr.pnrlctrnum
where Tile_num = 7 and ttddocseg.tdtrxnum = ttddoc.tdtrxnum and ttdhdr.tdtrxnum = ttddoc.tdtrxnum

Answer 1

You could use the ntile function to generate groups of groups. This would ensure the groups are kept together and you can then split your exports out into however many partitions you like.

Change thr ntile number to change the number of groupings you would like, you can then integrate this into your export process however you like.

with locator as 
(
    select ditinct
    Locator,
    NTILE(4) OVER(ORDER BY Locator) as Tile_Num
    from tbl 
)
select *
from tbl
inner join locator on locator.Locator = tbl.Locator
where Tile_num = 3

Answer 2

There is a mistake in this approach. Reproducing the same example on sample AdventureWorks DB in Person.Address table

with ps as 
(
SELECT distinct PostalCode, ntile(6) over (order by PostalCode) as part
  FROM [Person].[Address]
)

 select * from( select PostalCode, count(distinct part) as part_count from ps group by PostalCode ) as tmp where part_count>1

The result is:

PostalCode	part_count
3977	2
78400	2
92118	2
97301	2
GA10	2

There are some postal codes that belong in more than 1 group.This is because the separation is being executed in first place. The ntile tries to separate in groups of the same size without taking into account the values and by using distinct in select you end end up with distinct values in each group but some values may are repeated as we saw.

To avoid this recurrence you should get first the distinct postal codes, then split them in groups and in the end perform the join with the primary table.

 with ps as
 (
 select PostalCode, ntile(6) over (order by PostalCode) as part   
 from (select distinct PostalCode from Person.Address ) as tmp
 )
 
 select * from Person.Address a inner join ps on a.PostalCode= ps.PostalCode

This way you can keep groups of postal code in the same partition.As you can see there is no max value in a partition equal to a min value of the next partition.

 select ps.part, min(a.PostalCode)  as minimum, max(a.PostalCode) as maximum from Person.Address a inner join ps 
        on a.PostalCode= ps.PostalCode group by ps.part

part	minimum	maximum
1	01071	32960
2	33000	59100
3	59101	84070
4	84074	94066
5	94070	EM15
6	G1R	YO15