Preliminary. ThinkMeta ConcurrentTasks and this documentation are preliminary; everything is subject to change.

CUDA streams

On this page

A CUDA stream runs kernels and copies asynchronously: the calls that queue them return at once. Sooner or later the program needs the result, and cudaStreamSynchronize blocks the calling thread until the GPU has finished. Cuda::Await waits from a task instead: the task is suspended until the stream has finished, and the scheduler thread runs other tasks meanwhile.

import ThinkMeta.ConcurrentTasks.Cuda.Await;

The library module needs no CUDA headers and no CUDA library. Your program includes cuda_runtime.h, links cudart.lib and compiles its kernels with nvcc; the CUDA Toolkit brings all three, and its Visual Studio integration builds .cu files as part of the project.

Waiting for a stream

Queue the work on a stream of the task's own, then pass the stream and cudaLaunchHostFunc to Await:

cudaError_t GpuJob::Compute() noexcept
{
    auto status = cudaMemcpyAsync(deviceValues_, hostValues_, NumBytes, cudaMemcpyHostToDevice, stream_);
    if (status == cudaSuccess)
        status = LaunchScramble(stream_, deviceValues_, NumValues, KernelRounds); // queues the kernel
    if (status == cudaSuccess)
        status = cudaMemcpyAsync(hostValues_, deviceValues_, NumBytes, cudaMemcpyDeviceToHost, stream_);
    if (status != cudaSuccess)
        return status;

    // Suspends this task until the stream has finished.
    return Cuda::Await(stream_, cudaLaunchHostFunc);
}

Await queues a host function behind the work on the stream. CUDA calls it on one of its own threads once everything before it has finished, and it resumes the task.

Situation What Await does
The host function was queued Suspends the task until the stream reaches it, then returns success (0)
The host function could not be queued Returns the error of the launch function at once; the task did not wait
A kernel or a copy before it failed Returns success all the same: fetch errors of the work from CUDA after the wait

Important

Await suspends the calling task, so it may only be called from a task or from an OS thread after ConvertThreadToTask. Give every task that waits a stream of its own: the wait ends when its stream has finished, whatever other streams still do.

Asynchronous copies need page-locked host memory (cudaMallocHost). With ordinary memory cudaMemcpyAsync may copy before it returns, and the calling thread waits after all.

Kernels and modules

nvcc compiles .cu files, and it does not know C++ modules. Keep the kernel and a small function that launches it in the .cu file, declare that function in a header, and call it from the task's code:

// Scramble.cu – compiled by nvcc
__global__ void Scramble(std::uint32_t* values, std::uint32_t count, std::uint32_t rounds);

cudaError_t LaunchScramble(cudaStream_t stream, std::uint32_t* deviceValues, std::uint32_t count, std::uint32_t rounds) noexcept
{
    constexpr std::uint32_t BlockSize = 256;

    Scramble<<<(count + BlockSize - 1) / BlockSize, BlockSize, 0, stream>>>(deviceValues, count, rounds);
    return cudaGetLastError();
}

Runtime API and driver API

Await takes the function that queues a host function, so it works with both CUDA APIs:

API Stream type Launch function
Runtime API (cuda_runtime.h) cudaStream_t cudaLaunchHostFunc
Driver API (cuda.h) CUstream cuLaunchHostFunc

It returns the status type of that function, cudaError_t or CUresult.

Tasks change threads

A task may resume on another scheduler thread than the one it was suspended on, and CUDA keeps some state per thread:

What you use What to do
Runtime API, one GPU Nothing: every thread uses the same context of the device
Runtime API, several GPUs The current device is kept per thread: call cudaSetDevice again after a wait
Driver API Make the context current on every scheduler thread (cuCtxSetCurrent) in a thread callback

When it pays

Await frees the scheduler thread, not the GPU. It pays when tasks have CPU work to do while the GPU works: preparing the next input, checking or encoding a result, serving other requests. Many tasks can then have work queued on the GPU at the same time while a few threads run their CPU steps. A program that only waits for the GPU gains nothing over a blocked thread.

No timeouts

There is no Await with a time limit. The host function refers to state on the waiting task's stack, so the task cannot leave before the stream has reached it.

See also