Print sql query results to file automatically

Question

Hello everyone I am currently trying to write the results of a query to a tab delimited csv file automatically on a schedule. Currently I am using powershell to do this

$results = Invoke-SQLCmd -ServerInstance $Server -Database $Database -query "select distinct * from WHIProducts"
$results | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-File "$inventorypath\inventory_$date\$filename" -fo -en ascii

The problem with this is that the results are so big I am getting a system.outofmemoryexception error. I have tried increasing the maxmemorypershell but I still get the same error. I need to this automatically so going into SSMS and doing it manually is not enough. Any ideas?

I am trying to print ~170k rows. There will be more eventually, probably up to about 300k. Here is the powershell error.

ConvertTo-csv : Exception of type 'System.OutOfMemoryException' was thrown. At C:\\Users\\pmaho\\Dropbox\\MASSFILEStest\\scripts\\daily_inventory‌​.ps1:59 char:12 + $results | ConvertTo-csv -NoTypeInformation -Delimiter "`t" | Out-Fil ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [ConvertTo-Csv], OutOfMemoryException + FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.Co‌​nvertToCsvCommand –

I am using SQL Server express edition

Answer 1

Try piping the output of Invoke-SqlCmd to the Export-CSV cmdlet. Here is the script.

Push-Location; Import-Module SQLPS -DisableNameChecking; Pop-Location

$SQLServer = "localhost\inst1"
$DBName = "ExportCSVTesting"
$ExportFile = "C:\Users\BIGRED-7\Documents\Git\csvfiles\addresses.csv"
$Counter = 0

while ( $true )
{
    # Remove the export file
    if (Test-Path -Path $ExportFile -PathType Leaf) {
        Remove-Item $ExportFile -Force
    }

    # Clear the buffer cache to make sure each test is done the same
    $ClearCacheSQL = "DBCC DROPCLEANBUFFERS"
    Invoke-Sqlcmd -ServerInstance $SQLServer -Query $ClearCacheSQL

    # Export the table through the pipeline and capture the run time. Only the export is included in the run time.
    $ExportSQL = "SELECT * FROM [addresses] ;"

    $sw = [Diagnostics.Stopwatch]::StartNew()
    Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $ExportSQL | Export-CSV -Path $ExportFile -NoTypeInformation
    $sw.Stop()
    $sw.Elapsed
    $Milliseconds = $sw.ElapsedMilliseconds

    # Get a row count for display
    $RowCountSQL = "SELECT COUNT(0) AS [Count] FROM [addresses] ;"
    $RowCount = Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $RowCountSQL
    $RowCount = $RowCount.Count

    $Counter++
    Write-Output ("Run $Counter of RowCount: $RowCount")

    # Log the run statistics
    $StatsSQL = "INSERT INTO [RunStats] (Counter,Milliseconds,Notes) VALUES ($RowCount,$Milliseconds,'Pipeline')"
    Invoke-Sqlcmd -ServerInstance $SQLServer -Database $DBName -Query $StatsSQL
}