春季批处理在分区后挂起

Question

I am dealing with 2 million records and grid size of 20 for my spring batch application. For 100-200k records it works fine, but for 2 million it fails after partitioning.

Partition info:

{
    partition - 20 = {
        startKey = 1980865.0,
        endKey = 2069192.0,
        partitionSize = 72,
        partitionId = 20,
        partitionRecordCount = 88328
    },
    partition - 10 = {
        st
        artKey = 938305.0,
        endKey = 1042560.0,
        partitionSize = 72,
        partitionId = 10,
        partitionRecordCount = 104256
    },
    partition - 11 = {
        startKey = 1042561.0,
        endKey = 1146816.0,
        partitionSize = 72,
        partitionId = 11,
        partitionRecordCount = 104256
    },
    partition - 12 = {
        startKey = 1146817.0,
        endKey = 1251072.0,
        partitionSize = 72,
        partitionId = 12,
        partitionRecordCount = 104256
    },
    partition - 13 = {
        startKey = 1251073.0,
        endKey = 1355328.0,
        partitionSize = 72,
        partitionId = 13,
        partitionRecordCount = 104256
    },
    par
    tition - 14 = {
        startKey = 1355329.0,
        endKey = 1459584.0,
        partitionSize = 72,
        partitionId = 14,
        partitionRecordCount = 104256
    },
    partition - 15 = {
        startKey = 1459585.0,
        end
        Key = 1563840.0,
        partitionSize = 72,
        partitionId = 15,
        partitionRecordCount = 104256
    },
    partition - 16 = {
        startKey = 1563841.0,
        endKey = 1668096.0,
        partitionSize = 72,
        p
        artitionId = 16,
        partitionRecordCount = 104256
    },
    partition - 17 = {
        startKey = 1668097.0,
        endKey = 1772352.0,
        partitionSize = 72,
        partitionId = 17,
        partitionRecordCoun
        t = 104256
    },
    partition - 18 = {
        startKey = 1772353.0,
        endKey = 1876608.0,
        partitionSize = 72,
        partitionId = 18,
        partitionRecordCount = 104256
    },
    partition - 19 = {
        startKey =
            1876609.0,
        endKey = 1980864.0,
        partitionSize = 72,
        partitionId = 19,
        partitionRecordCount = 104256
    },
    partition - 1 = {
        startKey = 1.0,
        endKey = 104256.0,
        partitionSize
             = 72,
        partitionId = 1,
        partitionRecordCount = 104256
    },
    partition - 2 = {
        startKey = 104257.0,
        endKey = 208512.0,
        partitionSize = 72,
        partitionId = 2,
        partitionRecordCou
        nt = 104256
    },
    partition - 3 = {
        startKey = 208513.0,
        endKey = 312768.0,
        partitionSize = 72,
        partitionId = 3,
        partitionRecordCount = 104256
    },
    partition - 4 = {
        startKey = 3127
            69.0,
        endKey = 417024.0,
        partitionSize = 72,
        partitionId = 4,
        partitionRecordCount = 104256
    },
    partition - 5 = {
        startKey = 417025.0,
        endKey = 521280.0,
        partitionSize = 7
            2,
        partitionId = 5,
        partitionRecordCount = 104256
    },
    partition - 6 = {
        startKey = 521281.0,
        endKey = 625536.0,
        partitionSize = 72,
        partitionId = 6,
        partitionRecordCount
             = 104256
    },
    partition - 7 = {
        startKey = 625537.0,
        endKey = 729792.0,
        partitionSize = 72,
        partitionId = 7,
        partitionRecordCount = 104256
    },
    partition - 8 = {
        startKey = 729793
            .0,
        endKey = 834048.0,
        partitionSize = 72,
        partitionId = 8,
        partitionRecordCount = 104256
    },
    partition - 9 = {
        startKey = 834049.0,
        endKey = 938304.0,
        partitionSize = 72,
        partitionId = 9,
        partitionRecordCount = 104256
    }
}
Total no. of Partitions: 20

the last log it prints is the partition created and then control never comes into writer. For 200k records it was struggling, so I increased the connection pool size, which fixed the issue but it fails for 2 million rows.

There is no error, just it hangs there.

Answer 1

This could be related to db. Can you check if the application is able to access db with write credentials. I did notice this kind of issue where partitioning is done and the actual processing hung. After troubleshooting, realized it is not able to write to db. Once the db was fixed it ran fine.