关于CUDA中每个SM分配的寄存器数量

Question

First Question. The CUDA C Programming Guide is written like below.

The same on-chip memory is used for both L1 and shared memory: It can be configured as 48 KB of shared memory and 16 KB of L1 cache or as 16 KB of shared memory and 48 KB of L1 cache

But, device query shows "Total number of registers available per block: 32768". I use GTX580.(CC is 2.0) The guide says default cache size is 16KB, but 32768 means 32768*4(byte) = 131072 Bytes = 128 KBytes. Actually, I don't know which is correct.

Second Question. I set like below,

dim3    grid(32, 32);            //blocks in a grid
dim3    block(16, 16);           //threads in a block
kernel<<<grid,block>>>(...);

Then, the number of threads per a block is 256. => we need 256*N registers per a block. N means the number of registers per a thread needed. (256*N)*blocks is the number of registers per a SM.(not byte) So, if default size is 16KB and threads/SM is MAX(1536), then N can't over 2. Because of "Maximum number of threads per multiprocessor: 1536". 16KB/4Bytes = 4096 registers, 4096/1536 = 2.66666...

In case of larger caches 48KB, N can't over 8. 48KB/4Bytes = 12288 registers, 12288/1536 = 8

Is that true? Actually I'm so confused.

---

Actually, My almost full code is here. I think, the kernel is optimized when the block dimension is 16x16. But, in case of 8x8, faster than 16x16 or similar. I don't know the why.

the number of registers per a thread is 16, the shared memory is 80+16 bytes.

I had asked same question, but I couldn't get the exact solution.: The result of an experiment different from CUDA Occupancy Calculator

#define WIDTH 512
#define HEIGHT 512
#define TILE_WIDTH 8
#define TILE_HEIGHT 8
#define CHANNELS 3
#define DEVICENUM 1 
#define HEIGHTs HEIGHT/DEVICENUM

__global__ void PRINT_POLYGON( unsigned char *IMAGEin, int *MEMin, char a, char b, char c){
        int Col = blockIdx.y*blockDim.y+ threadIdx.y;           //Col is y coordinate
        int Row = blockIdx.x*blockDim.x+ threadIdx.x;           //Row is x coordinate
        int tid_in_block = threadIdx.x + threadIdx.y*blockDim.x;
        int bid_in_grid = blockIdx.x + blockIdx.y*gridDim.x;
        int threads_per_block = blockDim.x * blockDim.y;
        int tid_in_grid = tid_in_block + threads_per_block * bid_in_grid;

        float result_a, result_b;
        __shared__ int M[15];
        for(int k = 0; k < 5; k++){
                M[k] = MEMin[a*5+k];
                M[k+5] = MEMin[b*5+k];
                M[k+10] = MEMin[c*5+k];
        }

        int result_a_up = (M[11]-M[1])*(Row-M[0]) - (M[10]-M[0])*(Col-M[1]);
        int result_b_up = (M[6] -M[1])*(M[0]-Row) - (M[5] -M[0])*(M[1]-Col);

        int result_down = (M[11]-M[1])*(M[5]-M[0]) - (M[6]-M[1])*(M[10]-M[0]);

        result_a = (float)result_a_up / (float)result_down;
        result_b = (float)result_b_up / (float)result_down;

        if((0 <= result_a && result_a <=1) && ((0 <= result_b && result_b <= 1)) && ((0 <= (result_a+result_b) && (result_a+result_b) <= 1))){
                IMAGEin[tid_in_grid*CHANNELS] += M[2] + (M[7]-M[2])*result_a + (M[12]-M[2])*result_b;      //Red Channel
                IMAGEin[tid_in_grid*CHANNELS+1] += M[3] + (M[8]-M[3])*result_a + (M[13]-M[3])*result_b;    //Green Channel
                IMAGEin[tid_in_grid*CHANNELS+2] += M[4] + (M[9]-M[4])*result_a + (M[14]-M[4])*result_b;    //Blue Channel
        }
}

struct DataStruct {
    int                 deviceID;
    unsigned char       IMAGE_SEG[WIDTH*HEIGHTs*CHANNELS];
};

void* routine( void *pvoidData ) { 
        DataStruct  *data = (DataStruct*)pvoidData;
        unsigned char *dev_IMAGE;
        int *dev_MEM;
        unsigned char *IMAGE_SEG = data->IMAGE_SEG;

        HANDLE_ERROR(cudaSetDevice(5));

        //initialize array
        memset(IMAGE_SEG, 0, WIDTH*HEIGHTs*CHANNELS);
        cudaDeviceSetCacheConfig(cudaFuncCachePreferL1);
        printf("Device %d Starting..\n", data->deviceID);

        //Evaluate Time
        cudaEvent_t start, stop;
        cudaEventCreate( &start );
        cudaEventCreate( &stop );

        cudaEventRecord(start, 0); 

        HANDLE_ERROR( cudaMalloc( (void **)&dev_MEM, sizeof(int)*35) );
        HANDLE_ERROR( cudaMalloc( (void **)&dev_IMAGE, sizeof(unsigned char)*WIDTH*HEIGHTs*CHANNELS) );

        cudaMemcpy(dev_MEM, MEM, sizeof(int)*35, cudaMemcpyHostToDevice);
        cudaMemset(dev_IMAGE, 0, sizeof(unsigned char)*WIDTH*HEIGHTs*CHANNELS);

        dim3    grid(WIDTH/TILE_WIDTH, HEIGHTs/TILE_HEIGHT);            //blocks in a grid
        dim3    block(TILE_WIDTH, TILE_HEIGHT);                         //threads in a block

        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 0, 1, 2);
        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 0, 2, 3);
        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 0, 3, 4);
        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 0, 4, 5);
        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 3, 2, 4);
        PRINT_POLYGON<<<grid,block>>>( dev_IMAGE, dev_MEM, 2, 6, 4);

        HANDLE_ERROR( cudaMemcpy( IMAGE_SEG, dev_IMAGE, sizeof(unsigned char)*WIDTH*HEIGHTs*CHANNELS, cudaMemcpyDeviceToHost ) );
        HANDLE_ERROR( cudaFree( dev_MEM ) );
        HANDLE_ERROR( cudaFree( dev_IMAGE ) );

        cudaEventRecord(stop, 0); 
        cudaEventSynchronize(stop);

        cudaEventElapsedTime( &elapsed_time_ms[data->deviceID], start, stop );
        cudaEventDestroy(start);
        cudaEventDestroy(stop);


        elapsed_time_ms[DEVICENUM] += elapsed_time_ms[data->deviceID];
        printf("Device %d Complete!\n", data->deviceID);

        return 0;
}

Answer 1

The blockDim 8x8 is faster than 16x16 due to the increase in address divergence in your memory access when you increase the block size.

Metrics collected on GTX480 with 15 SMs.

metric                         8x8         16x16
duration                        161µs       114µs
issued_ipc                     1.24        1.31
executed_ipc                    .88         .59
serialization                 54.61%      28.74%

The number of instruction replays clues us in that we likely have bad memory access patterns.

achieved occupancy            88.32%      30.76%
0 warp schedulers issues       8.81%       7.98%
1 warp schedulers issues       2.36%      29.54%
2 warp schedulers issues      88.83%      52.44%

16x16 appears to keep the warp scheduler busy. However, it is keeping the schedulers busy re-issuing instructions.

l1 global load trans          524,407     332,007
l1 global store trans         401,224     209,139
l1 global load trans/request    3.56        2.25
l1 global store trans/request  16.33        8.51

The first priority is to reduce transactions per request. The Nsight VSE source view can display memory statistics per instruction. The primary issue in your kernel is the interleaved U8 load and stores for IMAGEin[] += value. At 16x16 this is resulting in 16.3 transactions per request but only 8.3 for 8x8 configuration.

Changing IMAGEin[(i*HEIGHTs+j)*CHANNELS] += ...

to be consecutive increases performance of 16x16 by 3x. I imagine increasing channels to 4 and handling packing in the kernel will improve cache performance and memory throughput.

If you fix the number of memory transactions per request you will then likely have to look at execution dependencies and try to increase your ILP.

Answer 2

It is faster with block size of 8x8 because it is a lesser multiple of 32, as it is visible in the picture below, there are 32 CUDA cores bound together, with two different warp schedulers that actually schedule the same thing. So the same instruction is executed on these 32 cores in each execution cycle.

To better clarify this, in the first case (8x8) each block is made of two warps (64 threads) so it is finished within only two execution cycles, however, when you are using (16x16) as your block size, each takes 8 warps (256 threads), therefore taking 4 times more execution cycles resulting in a slower compound.

However, filling an SM with more warps is better in some cases, when memory access is high and each warp is likely to go into a memory stall (ie getting its operands from memory), then it will be replaced with another warp until the memory operation gets completed. Therefore resulting in more occupancy of the SM.

You should of course throw in the number of blocks per SM and number of SMs total in your calculations, for example, assigning more than 8 blocks to a single SM might reduce its occupancy, but probably in your case, you are not facing these issues, because 256 is generally a better number than 64, since it will balance your blocks among SMs whereas using 64 threads will result in more blocks getting executed in the same SM.

EDIT: This answer is based on my speculations, for a more scientific approach, see Greg Smiths answer.

Register pool is different from shared memory/cache, to the very bottom of their architecture!

Registers are made of Flip-flops and L1 cache are probably SRAM .

Just to get an idea, look at the picture below which represents FERMI architecture, then update your question to further specify the problem you are facing.

As a note, you can see how many registers and shared memory (smem) are taken by your functions by passing the option --ptxas-options = -v to nvcc.