Performance first

The core and the library are built for speed, at the price of not being watertight against misuse. They check almost nothing at run time: no reference counting, no growing containers, no validation of arguments on the hot paths. Correct use is your responsibility.

Important

Debug builds assert some of the rules below. Release builds check nothing — a violation is undefined behavior: a crash, a hang or a corrupted queue, possibly much later than the mistake.

Lifetimes

An object must outlive everyone who can still use it. In detail:

Task objects live from AddTask (or Schedule) until their OnTaskExited has been called. Only then may they be destroyed or added again.
Synchronization objects — events, semaphores, critical sections, task groups — live as long as any task can still wait on them or signal them. The return of a Wait alone does not mean that the signaling side has finished touching the object; each type's reference page says when it may be destroyed. A TaskGroup in particular must survive the exit notification of its last task, which may still be running when Wait returns.
Asynchronous I/O requests and their buffers live until the operation has completed. Close a file or socket only when no operation on it is pending.
The scheduler is stopped only after your tasks have finished; Stop does not run queued or suspended tasks to completion.

The simplest way to get this right is structural: own the primitives in an object that is destroyed after the scheduler has stopped, and join every task before that. More in Lifetimes and sizing.

Sizing

Nothing grows on demand.

The scheduler's task queues have a fixed capacity (taskQueueCapacity, default 4096 per queue). It must exceed the number of tasks that can be runnable at the same time. Suspended tasks do not count.
A BoundedPointerQueue holds at most the capacity given to Initialize. Enqueueing into a full queue is invalid use and leaves the queue unusable — pair it with a semaphore that counts free slots if producers can outrun consumers.
A SpscVirtualMemoryPipe reserves address space for the size announced up front; the producer must not write more.

Single-waiter variants

Every waitable primitive comes in two variants: the plain name allows any number of waiting tasks, the _s name (for example Semaphore_s) allows at most one waiting task at a time. The _s variants are cheaper and support timeouts. Using an _s variant with two waiters is invalid use. See Single-waiter variants.

Who may call what

Calls that wait (Wait, Enter, the waiting forms of file and socket operations) must come from a task, or from an OS thread converted with ConvertThreadToTask.
Calls that signal (Set, Release, TryWait) never wait and may come from any thread.
Code inside ISuspendCallback::OnSuspended, thread callbacks and I/O completion routines must not wait at all.
Run is noexcept: an exception that escapes a task ends the process.
After a wait, a task may run on a different thread of its scheduler. Nothing thread-bound — OS locks, thread_local state, the thread id — may be carried across a wait.

Checklist

[ ] Every task object outlives its exit notification.
[ ] Every primitive outlives all its waiters and signalers.
[ ] Queue capacities cover the worst case.
[ ] _s variants have at most one waiter at a time.
[ ] Nothing waits outside a task or converted thread.
[ ] No OS lock or thread-local state is held across a wait.
[ ] Tasks are joined before Stop; I/O is idle before DisableAsyncIo.