CUDA内核性能随时间的变化

Question

I've been benchmarking some CUDA programs (2D and 3D lattice boltzmann solvers) and have come across something unusual; I would expect some random variation in the performance of the solver over time, but over a variety of different problem sizes, block sizes, OS' and GPUs (not to mention that the 2D and 3D codes are completely separate and not different configurations of the same program) I can see a very clear sinusoidal fluctuation in kernel execution times. For the two GPUs I've tested on (K5000m and K20c) the variation seems to have a frequency in the 10-12Hz range.

Is there any known explanation for this? My go-to idea is thermal/power management but I've not been able to prove it. Has anyone else experienced this?

FURTHER INFO AND AN EXAMPLE

A MSVC2010 project for a small example code can be found at https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxtYXJram1hd3NvbnxneDplOWMwNWNhNDA4MmMwMjg The project requires CUDA 5.0 and a sm_30 device,although there is only one file so building the project manually would be trivial. The code is fairly self explanatory, 100 iterations of a simple kernel (kernels to read from several arrays and write to several arrays are performed by default) are timed and their results printed to a file. Performing a FFT of the execution times yields visible peak near 11Hz on a K5000m. I would post an image but I don't have the reputation.

Answer 1

Windows has behavioral effects on the detailed timing of GPU kernel execution, especially when running the GPU in WDDM mode. Please re-run your observation and FFT preferably in a linux environment where X is not also running on the GPU. This will give you the most consistent behavior. The CUDA driver in a WDDM setup is subject, to some degree, to the windows operating system.

I ran your code on an SM35 device, CentOS 5.5, CUDA 5.5, and got the following Times.dat output:

0.007648 0.0024 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.001888 0.00192 0.00192 0.001856 0.00192 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.001888 0.001888 0.001856 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.002016 0.001888 0.001888 0.00192 0.001952 0.001888 0.001888 0.001888 0.001888 0.00192 0.00192 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.001888 0.001856 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.003904 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.00192 0.001856 0.001888 0.001856 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001952 0.0 01888 0.001888 0.00192 0.00192 0.00192 0.001888 0.001888 0.001952 0.001888 0.00192 0.001888 0.001856 0.001888 0.00192 0.001888 0.001888 0.001888 0.00192 0.001856 0.001888 0.001888 0.001888 0.001888 0.00192 0.00192 0.001888 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001888 0.00192 0.001888 0.001888 0.00192 0.001888 0.00192 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.004448 0.001888 0.001952 0.001888 0.001888 0.001888 0.001888 0.001888 0.001856 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.00192 0.001856 0.001888 0.001888 0.001888 0.001888 0.001856 0.001888 0.001888 0.001856 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888 0.001888

There is occasional variation however don't forget your cudaTime variable is capturing a time in milliseconds . So the variation I see in the above data is mostly on the order of less than 1 microsecond variation, run to run.

Throwing out the first number, the largest variation I see is about 2-3 microseconds in a few cases. Given that the execution time measured is less than 2 microseconds typically, this is a large variation but it's still in the noise, and not anything like the 10's of microseconds you're reporting.

To my untrained eye I also don't see any sinusoidal pattern in the data, but if you tell me there's an 11Hz frequency in there (not even sure what that means, since these data points are not time-stamped that I can see) -- I'll take your word for it.