SQL Server System.OutOfMemoryException + too many outcomes

Question

I'm a self-learning SQL programmer.

When I tried to join the tables (there are probably near to 100 mil outcome), a System.OutOfMemoryException is thrown. My main purpose is to generate the combination of values; the other table (which stored a set of other numbers) will take these values in every rows, compared with itself and devise the buy/sell strategy. This italic part is where the outofmemory error occurs. Thus you can imagine the dataset will be huge.

This is the code:

select * 
into #MyTempTable 
from
    (Select distinct Parm_H 
     from [dbo].[Parmset1]) A1
inner join 
    (Select distinct Parm_D 
     from [dbo].[Parmset1]) A2 on A1.Parm_H > A2.Parm_D
inner join
    (Select distinct Parm_A 
     from [dbo].[Parmset1]) A3 on A2.Parm_D > A3.Parm_A


select * 
from [dbo].[MatchRegression]
cross join #MyTempTable

My computer spec is as below

• i3-6100U
• Ram: 8 GB
• System type: 64 bit

My question is:

• While online resources said it is the RAM issue, does only add RAM without upgrading to I7 helps?

• Discussed with some experienced programmers, it is the code itself that matters as too many joins are involved- so changing RAM does not help. Any ideas that I can achieve my main objective without going through much joinings.

• Since I was using SQL Server, any other software would be specifically designed for handling huge set of data (Free of charge; more applicable use) and compatible to the current spec of my computer?

I'm willing to learn anyway.

Thanks, Vincent

Answer 1

The out of memory exception is generated by the 32 bit SSMS client that you are using.

select * from [dbo].[MatchRegression] cross join #MyTempTable

Obviously, you are not manually going to view the data and do anything with it, so my suggestion would be to insert this into another table and run queries on it , not trying to select all data to show in the datagrid or client application. SQL Server should easily be able to handle billions of rows of data without any problem.

Though, you might need to look at the edition comparison sheet to look at what edition of SQL Server would be best for your application depending on size of data, requirements etc