在cuda中，加载到共享内存比加载到寄存器要慢

Question

I'm not an experienced CUDA programmer. I got a problem like this. I'm trying to load a tile (32x32) of a large matrix (10K*10K) from global memory into shared memory and I'm timing it while it happens. I realized that If I load it to private memory(registers), it loads 4-5 times faster than shared memory loading.

__global__ void speedtest( float *vel,int nx) {

int globalx = blockDim.x * blockIdx.x + threadIdx.x+pad;
int globalz = blockDim.y * blockIdx.y + threadIdx.y+pad;
int localx=threadIdx.x;
int localz=threadIdx.y;

float ptest;
__shared__ float stest[tile][tile];

//stest[localz][localx]=vel[globalz*nx+globalx]; //load to shared memory
ptest=vel[globalz*nx+globalx];  //load to private memory
__syncthreads();
}

I comment out stest and ptest one by one and calculate elapsed time with cudaeventrecord. stest took 3.2 ms and ptest took 0.75ms to load. What am I doing wrong? Timings should be very similar right? What am I missing?

Configuration: Cuda 7.5, gtx 980, only 32bit variables and calculations, no specific purpose is intended, I'm just playing with it.

I'm posting sample code as requested

#include<stdio.h>
#include <math.h>
#define tile 32
#include <helper_cuda.h>
void makeittwo(float *array,int nz,int nx)
{
//this just assigns a number into the vector
int n2;
n2=nx*nz;
for (int i=0;i<n2;i++)
array[i]=2000;
}
__global__ void speedtest( float *vel,int nx,int nz) {

int globalx = blockDim.x * blockIdx.x + threadIdx.x;
int globalz = blockDim.y * blockIdx.y + threadIdx.y;
int localx=threadIdx.x;
int localz=threadIdx.y;

float ptest; //declarations
__shared__ float stest[tile][tile];

if (globalx<nx && globalz<nz){
stest[localz][localx]=vel[globalz*nx+globalx]; //shared variable
//ptest=vel[globalz*nx+globalx];                        //private variable

//comment out ptest and stest one by one to test them  
}
__syncthreads();

}       

int main(int argc,char *argv)
{
int nx,nz,N;
float *vel;

nz=10000;nx=10000; //matrix dimensions
N=nz*nx; //convert matrix into vector

checkCudaErrors(cudaMallocHost(&vel,sizeof(float)*N)); //using pinned memory
makeittwo(vel,nz,nx);

dim3 dimBlock(tile,tile);
dim3 dimGrid;

int blockx=dimBlock.x;
int blockz=dimBlock.y;

dimGrid.x = (nx + blockx - 1) / (blockx);
dimGrid.y = (nz + blockz - 1) / (blockz);

float *d_vel;
checkCudaErrors(cudaMalloc(&d_vel,sizeof(float)*(N))); //copying to device
checkCudaErrors(cudaMemcpy(d_vel, vel, sizeof(float)*(N), cudaMemcpyHostToDevice));

cudaEvent_t start,stop;
float elapsedTime;

cudaEventCreate(&start);
cudaEventCreate(&stop);
cudaEventRecord(start,0);
speedtest<<<dimGrid,dimBlock>>>(d_vel,nx,nz); //calling the function
cudaEventRecord(stop,0);
cudaEventSynchronize(stop);
cudaEventElapsedTime(&elapsedTime,start,stop);

printf("time=%3.3f ms\n",elapsedTime);
checkCudaErrors(cudaMemcpy(vel, d_vel, sizeof(float)*N, cudaMemcpyDeviceToHost)); 
//calling the matrix back  to check if all went well (this fails if out of bound calls are made)

cudaDeviceReset();

}

Answer 1

The example code actually does not measure what the OP expects to measure, because some instructions are optimized away by the compiler.

In the local variable example ( ptest ) the load does not affect the state outside of the kernel. In this case the compiler is free to remove the instruction completely. This can be seen in the SASS code. The SASS code is same when ptest=vel[globalz*nx+globalx]; is active or both statements (ptest and stest) are removed. To inspect the SASS code you can run cuobjdump --dump-sass on the object file.

Apparently, the instructions are not optimized away in the shared memory example as can be checked in SASS code. (Actually, I would have expected the instructions are removed as well. Are there side-effects that miss?)

As already discussed in the comments, with a simple calculation ( ptest*=ptest ) and a write to global memory the compiler cannot remove the instruction because it changes the global state.

From the comments of the OP I assume that there is a misunderstanding in how the load operation to shared memory works. Actually the data is loaded from global memory to registers and then stored in shared memory . The (relevant) SASS instructions (for sm_30) that are generated look like this

LD.E R2, [R6]; // load to register R2
STS [R0], R2; // store from register R2 to shared memory

The following multiply and store to global memory example demonstrates another case where the compiler does not produce code that one may naively expect:

stest[localz][localx]=vel[globalz*nx+globalx]; // load to shared memory
stest[localz][localx]*=stest[localz][localx]; // multiply
vel[globalz*nx+globalx]=stest[localz][localx]; // save to global memory

The SASS code shows that the variable is only stored in shared memory after the computation (and never read form shared memory).

LD.E R2, [R6]; // load to register
FMUL R0, R2, R2; // multiply
STS [R3], R0; // store the result in shared memory
ST.E [R6], R0; // store the result in global memory

I am not really an expert in SASS code, please correct me if I am wrong or left out anything important.