CMake/CUDA 共享库的可重定位代码

Question

I've stumbled upon a bit of a doozy compile error I've never encountered before. The exact code cannot be shared, so I will present an analogous situation. I have a shared library I'm developing, which compiles __device__ tagged device code. These devices functions must be able to be used by __global__ functions written by the user. Here is a boiled down set of code which recreates the raised error:

The source code for the shared library: device_function.cu

__device__ int deviceFunction()
{
    return 1;
}

The source code for the executable meant to call the device function: soure.cu

#include <stdio.h>

__device__ int deviceFunction();

__global__ void globalFunction()
{
    printf("%i", deviceFunction());
}

int main()
{
    globalFunction<<<1,1>>>();
    cudaDeviceSynchronize();

    return 0;
}

The CMakeLists.txt file I've attempted to compile everything with:

cmake_minimum_required(VERSION 3.21)

set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR})
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR})

project(test)

find_package(CUDA REQUIRED)
enable_language(CUDA)

add_library(device_function SHARED device_function.cu)

add_executable(cu_test source.cu)
target_link_libraries(cu_test device_function)

Upon (attempted) compilation, I'm greeted with this:

[main] Building folder: relocatable-code 
[build] Starting build
[proc] Executing command: /snap/cmake/current/bin/cmake --build /home/legayone/Documents/research-pathfinding-projects/cuda-programming/relocatable-code/build --config Debug --target all -j 18 --
[build] Consolidate compiler generated dependencies of target device_function
[build] [ 50%] Built target device_function
[build] Consolidate compiler generated dependencies of target cu_test
[build] [ 75%] Building CUDA object CMakeFiles/cu_test.dir/source.cu.o
[build] ptxas fatal   : Unresolved extern function '_Z14deviceFunctionv'
[build] make[2]: *** [CMakeFiles/cu_test.dir/build.make:76: CMakeFiles/cu_test.dir/source.cu.o] Error 255
[build] make[1]: *** [CMakeFiles/Makefile2:111: CMakeFiles/cu_test.dir/all] Error 2
[build] make: *** [Makefile:91: all] Error 2
[build] Build finished with exit code 2

What I've Tried I have scoured the web and arrived at solutions which all ultimately boil down to CUDA_SEPARABLE_COMPILATION ON or some form of -rdc=true or -dc . I have attempted adding separable compilation in the 3 possible combinations for device_function and cu_test , and I've done the same for -rdc=true and -dc , where I try it on one, then the other, then both. This is the format which I add -rdc=true and -dc to things in:

target_compile_options(cu_test PRIVATE $<$<COMPILE_LANGUAGE:CUDA>:-c "-lcudart -lcudadevrt -lcuda -rdc=true">)

~or~

target_compile_options(device_function PRIVATE $<$<COMPILE_LANGUAGE:CUDA>:-c "-lcudart -lcudadevrt -lcuda -rdc=true">)

My question What am I missing or what am I doing wrong? I'd really like any executable to just be able to have the shared library device_function linked to it, and for it to be able to call the function inside that shared library. In the actual library this applies to, there are headers of course, but I have the includes sorted out:) its just linking.

Where I suspect the issue is I suspect the issue is code relocatability. I'm aware special things have to be done to permit device functions from a different compilation unit to be used by an executable (or another library,)? but what are those things and how do I do it in CMake?

Answer 1

A Kind Of Solution

So it would seem that shared library based __device__ functions being called by __global__ functions from a separate compilation unit is not possible . I should note in this that there is a tremendous amount of conflicting information, specifically from this article: https://developer.nvidia.com/blog/building-cuda-applications-cmake/ , which seems to suggest it is possible, however its presented solutions do not work. Here is what works for me:

CMakeLists.txt

cmake_minimum_required(VERSION 3.21)

set(CMAKE_CUDA_SEPARABLE_COMPILATION ON)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)

project(test LANGUAGES CXX CUDA)

include(CTest)

add_library(device_function STATIC device_function.cu)
add_library(shared_function SHARED shared_device_function.cu)
target_link_libraries(shared_function PUBLIC device_function)

add_executable(cu_test source.cu)
target_link_libraries(cu_test shared_function)

What Works

• linking works, as does compilation
• the device function in device_function.cu is able to be called from the global function in the executable source source.cu

What Does Not Work

• The actual file containing the device function must be a static library, but it can be linked to a shared library, allowing the user to use the whole library by just a single link to a whole shared library
• This static library method is, apparently, slower than the shared library method