CUDA streams
On this page
A CUDA stream runs kernels and copies asynchronously: the calls that queue them return at once.
Sooner or later the program needs the result, and cudaStreamSynchronize blocks the calling
thread until the GPU has finished. Cuda::Await waits from a task instead: the
task is suspended until the stream has finished, and the scheduler thread runs other tasks
meanwhile.
import ThinkMeta.ConcurrentTasks.Cuda.Await;The library module needs no CUDA headers and no CUDA library. Your program includes
cuda_runtime.h, links cudart.lib and compiles its kernels with nvcc; the CUDA Toolkit brings
all three, and its Visual Studio integration builds .cu files as part of the project.
Waiting for a stream
Queue the work on a stream of the task's own, then pass the stream and cudaLaunchHostFunc to
Await:
cudaError_t GpuJob::Compute() noexcept
{
auto status = cudaMemcpyAsync(deviceValues_, hostValues_, NumBytes, cudaMemcpyHostToDevice, stream_);
if (status == cudaSuccess)
status = LaunchScramble(stream_, deviceValues_, NumValues, KernelRounds); // queues the kernel
if (status == cudaSuccess)
status = cudaMemcpyAsync(hostValues_, deviceValues_, NumBytes, cudaMemcpyDeviceToHost, stream_);
if (status != cudaSuccess)
return status;
// Suspends this task until the stream has finished.
return Cuda::Await(stream_, cudaLaunchHostFunc);
}Await queues a host function behind the work on the stream. CUDA calls it on one of its own
threads once everything before it has finished, and it resumes the task.
| Situation | What Await does |
|---|---|
| The host function was queued | Suspends the task until the stream reaches it, then returns success (0) |
| The host function could not be queued | Returns the error of the launch function at once; the task did not wait |
| A kernel or a copy before it failed | Returns success all the same: fetch errors of the work from CUDA after the wait |
Important
Await suspends the calling task, so it may only be called from a task or from an OS thread
after ConvertThreadToTask. Give every task that waits a stream of
its own: the wait ends when its stream has finished, whatever other streams still do.
Asynchronous copies need page-locked host memory (cudaMallocHost). With ordinary memory
cudaMemcpyAsync may copy before it returns, and the calling thread waits after all.
Kernels and modules
nvcc compiles .cu files, and it does not know C++ modules. Keep the kernel and a small
function that launches it in the .cu file, declare that function in a header, and call it from
the task's code:
// Scramble.cu – compiled by nvcc
__global__ void Scramble(std::uint32_t* values, std::uint32_t count, std::uint32_t rounds);
cudaError_t LaunchScramble(cudaStream_t stream, std::uint32_t* deviceValues, std::uint32_t count, std::uint32_t rounds) noexcept
{
constexpr std::uint32_t BlockSize = 256;
Scramble<<<(count + BlockSize - 1) / BlockSize, BlockSize, 0, stream>>>(deviceValues, count, rounds);
return cudaGetLastError();
}Runtime API and driver API
Await takes the function that queues a host function, so it works with both CUDA APIs:
| API | Stream type | Launch function |
|---|---|---|
Runtime API (cuda_runtime.h) |
cudaStream_t |
cudaLaunchHostFunc |
Driver API (cuda.h) |
CUstream |
cuLaunchHostFunc |
It returns the status type of that function, cudaError_t or CUresult.
Tasks change threads
A task may resume on another scheduler thread than the one it was suspended on, and CUDA keeps some state per thread:
| What you use | What to do |
|---|---|
| Runtime API, one GPU | Nothing: every thread uses the same context of the device |
| Runtime API, several GPUs | The current device is kept per thread: call cudaSetDevice again after a wait |
| Driver API | Make the context current on every scheduler thread (cuCtxSetCurrent) in a thread callback |
When it pays
Await frees the scheduler thread, not the GPU. It pays when tasks have CPU work to do while the
GPU works: preparing the next input, checking or encoding a result, serving other requests. Many
tasks can then have work queued on the GPU at the same time while a few threads run their CPU
steps. A program that only waits for the GPU gains nothing over a blocked thread.
No timeouts
There is no Await with a time limit. The host function refers to state on the waiting task's
stack, so the task cannot leave before the stream has reached it.
See also
- Cuda::Await, Synchronization::Await
- WinRT operations – the general form for every API that reports its completion through a callback
- Thread callbacks, When to use ConcurrentTasks