从 SQL 服务器表中获取超过百万行

Question

In past I got System.OutOfMemoryException during loading more than a million rows into a DataTable from a SQL Server database. Is it true that instead of filling DataTable , it is much faster eg to load it somehow to CSV or any other format besides? Please of your advice, eventually with given propose / example.

Answer 1

Not knowing exactly how you might use the data, you could pull all the data using a SqlDataReader and than write that to a SQLite database file. This essentially gives you a local cached version of the million rows that you can query as if it were stored in the database, but doesn't take up memory since it's written to a local file (and provides in-memory cache if needed).

https://docs.microsoft.com/en-us/dotnet/standard/data/sqlite/?tabs=netcore-cli