CUDA中更快的矩阵乘法

Question

Currently, I made a neural networks program in the cuda c. Because I needed to manipulate the matrix multiplication, I did not use CUBLAS for MM. I use the following code for MM. I was wondering if any one has some advice to make it faster which can be very helpful since I need to use MM millions of times during learning. Thanks. This is the MakeFile:

# cuda root
_CUDA_ROOT_ = /usr/local/cuda

NVCC = nvcc
# include and lib paths
INCLUDES=-I${_CUDA_ROOT_}/include
LIB_PATH=-L${_CUDA_ROOT_}/lib64

# libraries to link against
LIB= -lcudart -lcublas
CU_SRC= main.cu
EXE=$(CU_SRC:.cu=)
#------------------------------
# Choose your gpu arch
SM = sm_35
all: $(EXE)
$(EXE): $(CU_SRC)
        $(NVCC) -arch $(SM) $(CU_SRC) -o $(EXE) $(LIB_PATH) $(LIB)

clean:
        rm -f *.o *.cu_o $(EXE)

This is the MM code:

__global__
void matrixMulti(float* A_d, float* B_d, float* C_d, int m, int k, int n)
{
    __shared__ float ds_A[TILE_WIDTH][TILE_WIDTH];
    __shared__ float ds_B[TILE_WIDTH][TILE_WIDTH];
    int col = blockIdx.x*blockDim.x + threadIdx.x;
    int row = blockIdx.y*blockDim.y + threadIdx.y;
    int tx = threadIdx.x;
    int ty = threadIdx.y;
    float sum = 0;

    for(int t=0; t<(n-1)/TILE_WIDTH+1; t++)
    {
        if(row<m && t*TILE_WIDTH+tx<n)
            ds_A[ty][tx] = A_d[row*n + t*TILE_WIDTH+tx];
        else
            ds_A[ty][tx] = 0.0;
        if(t*TILE_WIDTH+ty<n && col<k)
            ds_B[ty][tx] = B_d[(t*TILE_WIDTH+ty)*k + col];
        else
            ds_B[ty][tx] = 0.0;
        __syncthreads();
        for(int i=0; i<TILE_WIDTH; i++)
            sum += ds_A[ty][i] * ds_B[i][tx];
        __syncthreads();
    }
    if(row<m && col<k)
        C_d[col+row*k] = sum;
}

This is the example of main part of the code:

const int TILE_WIDTH = 32;

int main()
{
    int m, k, n;
    m = 10000, k = 10000, n = 10000;
    float *A, *B, *C;
    A = new float[m*n];
    B = new float[n*k];
    C = new float[m*k];
    float *A_d, *B_d, *C_d;
    for (int i=0; i<m*n; i++)
    {
        A[i] = 2;
    }
    for (int i=0; i<n*k; i++)
    {
        B[i] = 3;
    }
    cudaMalloc(&A_d, sizeof(float)*m*n);
    cudaMalloc(&B_d, sizeof(float)*n*k);
    cudaMalloc(&C_d, sizeof(float)*m*k);
    cudaMemcpy(A_d, A, sizeof(float)*m*n, cudaMemcpyHostToDevice);
    cudaMemcpy(B_d, B, sizeof(float)*k*n, cudaMemcpyHostToDevice);
    dim3 dimGrid((k-1)/TILE_WIDTH+1, (m-1)/TILE_WIDTH+1, 1);
    dim3 dimBlock(TILE_WIDTH, TILE_WIDTH, 1);
    matrixMulti<<<dimGrid,dimBlock>>>(A_d, B_d, C_d, m, k, n);
    cudaMemcpy(C, C_d, sizeof(float)*m*k, cudaMemcpyDeviceToHost);
    return 0;
}

Answer 1

Firstly, be really sure this is what you want to do. Without describing the manipulations you want to do, it's hard to comment on this, but be aware that matrix multiplication is an n-cubed operation. If your manipulations are not the same complexity, chances are you'll do better simply using cuBLAS.

Why is this? cuBLAS will probably be faster than anything you'll write, and will be much more maintainable as it will follow new GPU architectures. The best implementation of something like GEMM will vary based on architecture, so any code you're writing now for your hardware will have to be re-optimised for new hardware.

Now, to the question. There's a number of techniques you should consider to optimise this code:

1. Compute multiple output values per thread . This reduces the pressure on your shared memory as tile data can be used in multiple calculations.
2. Fix the bank conflicts in shared memory . This should be covered well by the documentation.
3. Vectorise shared memory loads and stores . I notice you're compiling for sm_35. This architecture's shared memory banks each have a bandwidth of 64 bits/clock. Loading a single float is only 32 bits, so you won't get full bandwidth on floats without vectorization. You should look at float2/float4 types.
4. Consider double buffering . Load data into one shared memory tile while operating on another. This allows the high latency of global memory operations to be hidden much more effectively, reduces the synchronisation overhead, and often tends to perform better. It uses twice as much shared memory though, as you need two tiles at once.

There are a number of papers on the implementation of matrix multiplication on GPUs, I suggest you check them out. You'll get a lot more detail from these papers than you will asking broad questions on SO.

Finally... are you sure you don't want to use cuBLAS? I wouldn't count on getting 75% of cuBLAS performance, and even that will be a challenge.