Thread callbacks

A scheduler creates its threads itself. To configure them — give them names, place them on processors, raise their priority, initialize per-thread state — register a thread callback in SchedulerInfo. The scheduler calls it on each of its threads twice: when the thread starts and when it exits. The examples come from guide-thread-callbacks.cpp, a complete program that checks its results.

Registering the callback

The callback is a plain function of type ThreadCallbackProc. Whatever it needs from the application comes through one pointer, threadCallbackContext, which is passed to every call unchanged:

// What the callback needs from the application. Passed as threadCallbackContext; it
// must outlive the scheduler's threads.
struct ThreadSetup {
    const wchar_t*             name; // prefix of the thread names
    std::atomic<unsigned>      started{0};
    std::atomic<unsigned>      exited{0};
    std::atomic<unsigned long> itemsDone{
        0}; // collected from the threads on Exiting
};

ThreadSetup setup{.name = L"compute"};

TC::SchedulerInfo info{};
info.threadCallback        = SetUpThread;
info.threadCallbackContext = &setup;
TC::ITaskScheduler* scheduler
    = TC::CreateTaskScheduler(TC::SchedulerKind::Default, &info);

When it is called

ThreadCallbackReason	Called …
`Starting`	on the new scheduler thread, before it runs its first task
`Exiting`	on the same thread, after it has left the scheduling loop (after Stop), right before the thread ends

Each call gets the scheduler, the thread's 0-based index within its scheduler, the thread's handle and the context. When Stop returns, every thread has run its Exiting call.

Configuring a thread

A typical callback names the thread, chooses its ideal processor and sets up per-thread state on Starting, and collects that state on Exiting:

// Runs on each scheduler thread itself: once before its first task, once after
// Stop. It is plain OS-thread code - no waiting, no synchronization primitives, no
// task functions.
static void SetUpThread(TC::ITaskScheduler*, DWORD threadIndex, HANDLE thread,
                        void* context, TC::ThreadCallbackReason reason) noexcept
{
    auto& setup = *static_cast<ThreadSetup*>(context);

    if (reason == TC::ThreadCallbackReason::Starting) {
        // A name shows up in the debugger, in profilers and in crash dumps.
        wchar_t name[64];
        std::swprintf(name, 64, L"%ls #%lu", setup.name, threadIndex);
        SetThreadDescription(thread, name);

        // Spread the threads over the processors of the current group.
        PROCESSOR_NUMBER ideal{};
        ideal.Number = static_cast<BYTE>(threadIndex % GetActiveProcessorCount(0));
        SetThreadIdealProcessorEx(thread, &ideal, nullptr);

        t_itemsDone = 0;
        setup.started.fetch_add(1);
    } else { // ThreadCallbackReason::Exiting
        setup.itemsDone.fetch_add(t_itemsDone);
        setup.exited.fetch_add(1);
    }
}

Names (SetThreadDescription) appear in the debugger's Threads window, in profilers and in crash dumps. With several schedulers, a prefix per scheduler tells their threads apart.
Placement: an ideal processor is a hint the Windows scheduler follows when it can; an affinity mask (SetThreadAffinityMask, SetThreadGroupAffinity) is a hard limit. On machines with more than 64 logical processors, threads also have to be spread over processor groups.
Priority: SetThreadPriority on the thread handle, for example to keep the threads of an I/O scheduler ahead of the computation.
Other per-thread setup — CoInitializeEx/CoUninitialize, a per-thread allocator arena, floating-point modes — belongs here too, paired between Starting and Exiting.

The thread handle belongs to the scheduler and stays valid until the scheduler is destroyed; duplicate it with DuplicateHandle to keep it longer.

Per-thread state

thread_local variables set up in the callback belong to one scheduler thread:

// Per-thread state: valid on one thread only. A task may continue on another thread
// after it has waited, so code uses it only between two waits.
thread_local unsigned long t_itemsDone = 0;

Important

A task does not own a thread. After it has waited — on a primitive, for I/O, or by yielding — it may continue on another thread of its scheduler. Read and change per-thread state only between two waits, never across one. State that must follow the task belongs in the task object.

Real-time threads

A scheduler can be dedicated to real-time work. This one has a single thread that joins the Multimedia Class Scheduler Service (MMCSS) for as long as it exists:

// A thread for real-time work, for example live audio: MMCSS raises its priority
// while the thread exists. The handle from AvSetMmThreadCharacteristics must be
// reverted on the same thread, so it is kept in thread-local state between Starting
// and Exiting.
thread_local HANDLE t_mmcss = nullptr;

static void JoinMmcss(TC::ITaskScheduler*, DWORD, HANDLE thread, void*,
                      TC::ThreadCallbackReason reason) noexcept
{
    if (reason == TC::ThreadCallbackReason::Starting) {
        SetThreadDescription(thread, L"audio");
        DWORD taskIndex = 0;
        t_mmcss         = AvSetMmThreadCharacteristicsW(L"Pro Audio", &taskIndex);
    } else if (t_mmcss != nullptr) {
        AvRevertMmThreadCharacteristics(t_mmcss);
        t_mmcss = nullptr;
    }
}

Tasks added to this scheduler run with the raised priority; everything else runs on the other schedulers. The MMCSS functions need avrt.lib.

Rules

Important

The callback runs on the plain OS thread, not in a task. It must not wait, must not use the synchronization primitives and must not call the task functions (such as GetCurrentTask or YieldTask). Keep it short: a scheduler thread starts running tasks only after its Starting call has returned.

The context must outlive the scheduler's threads — at least until Stop has returned.
The callback cannot report errors to the scheduler. Record them in the context and check them after Start, or after Stop for the Exiting calls.
Only the Default scheduler calls it; the TaskSwap scheduler does not.

Output

threads: 16 started, 16 exited; items: 64000
audio thread in MMCSS: yes
OK

The number of threads is the number of logical processors of the machine.