Extracting data from table into array, then reprinting onto new worksheet

Question

I have a very heavy Excel table (roughly 65,000 lines and 7 columns) that I want to extract, piece-by-piece, onto different worksheets. I've started the program and it seems to run fine, but it always stops the recopying at line 1771, despite me not seeing any glaring errors in the code. The recopying should go to at least line 3500. If I remove the If statement and consolidate the 2 For Loops into 1 For Loop, it still stops at line 1771. Are arrays limited in terms of storage?

Sub extract_collar()

Dim myArray() As Variant
Dim myTable As ListObject
Dim cell As Range
Dim x As Long


Application.ScreenUpdating = False

Set myTable = ActiveWorkbook.Sheets("FullCarriers").ListObjects("CarrierTable")

For i = 1 To 2

    TempArray = myTable.DataBodyRange.Columns(i)

    myArray = Application.Transpose(TempArray)

    For x = LBound(myArray) To UBound(myArray)
        If Mid(myArray(x), 13, 2) = "01" Then
            ActiveWorkbook.Sheets("Collar").Cells(x + 1, i) = myArray(x)
        End If

    Next x

Next i

For i = 3 To 7

    TempArray = myTable.DataBodyRange.Columns(i)

    myArray = Application.Transpose(TempArray)

    For x = LBound(myArray) To UBound(myArray)
        ActiveWorkbook.Sheets("Collar").Cells(x + 1, i) = myArray(x)
    Next x

Next i

Application.ScreenUpdating = True

End Sub

Answer 1

Numerous sources I have found have suggested that the size of arrays for VBA code depends upon the amount of memory in the machine.

It would be nice if there was an Application.UseMoreMemory() function that we could just call :-)

Alas, I know of none.

All the docs I've seen say that it's limited by memory, but it's not physical memory that's the issue, it's the virtual address space you have available to you.

You should keep in mind that, while the increase from 500 to 600 only looks like a moderate increase (though 20% is large enough on its own), because you're doing that in three dimensions, it works out to be close to double the storage requirements.

From memory, Excel 2007 used short integers (16 bits) for boolean type so, at a minimum, your 5003 array will take up about 250M (500x500x500x2).

Increasing all dimensions to 600 would give you 600x600x600x2, or about 432M.

All well within the 2G usable address space that you probably have in a 32-bit machine (I don't know that Excel 2007 had a 64-bit version), but these things are not small, and you have to share that address space with other things as well.

It'd be interesting to see at what point you started getting the errors.

As a first step, I'd be looking into the need for such a large array. It may be doable a different way, such as partitioning the array so that only part of it is in memory at any one time (sort of manual virtual memory).

That's unlikely to perform that well for truly random access but shouldn't be too bad for more sequential access and will at least get you going (a slow solution is preferable to a non-working one).

Another possibility is to abstract away the bit handling so that your booleans are actually stored as bits rather than words.

You would have to provide functions for getBool and setBool , using bitmask operators on an array of words and, again, the performance wouldn't be that crash-hot, but you would at least be able to then go up to the equivalent of:

' Using bits instead of words gives 16 times as much. '
Dim arr(8000, 8000, 8000) As Boolean

As always, it depends on what you need the array for, and its usage patterns.

I hope I was able to help you.

Answer 2

If I'm understanding correctly:

Sub extract_collar()

    Dim inArray As Variant, outArray As Variant
    Dim myTable As ListObject
    Dim cell As Range
    Dim x As Long, i As Long, c As Long, cols As Long

    Set myTable = ActiveWorkbook.Sheets("FullCarriers").ListObjects("CarrierTable")

    inArray = myTable.DataBodyRange.Value 'get all the data
    ReDim outArray(1 To UBound(inArray, 1), 1 To cols) 'size the output array
    cols = UBound(inArray, 2)             'how many columns

    x = 0
    'loop over the data and check if we need the row
    For i = 1 To UBound(inArray, 1)
        If Mid(inArray(i, 1), 13, 2) = "01" Or Mid(inArray(i, 2), 13, 2) = "01" Then           
            x = x + 1 'increment "row" in output array
            'copy the row to the "out" array
            For c = 1 To cols 
                outArray(x, c) = inArray(i, c)
            Next c
        End If
    Next

    'dump the results to a worksheet
    If x > 0 Then
        ActiveWorkbook.Sheets("Collar").Range("A2").Resize(x, cols).Value = outArray
    End If

End Sub