OrderedFileWriter class
An output file built front to back in memory, with the copying and writing done by other tasks.
- Use
import ThinkMeta.ConcurrentTasks.AsyncIo.OrderedFileWriter;- Namespace
ThinkMeta::ConcurrentTasks::AsyncIo
Syntax
class OrderedFileWriterRemarks
The file is built in one reserved address range. One producer (a task or thread) appends bytes with Write and promises copies with WriteDeferred; EndBatch hands the promised copies to the copy tasks (CopyTask, any number). The writer task (Writer) writes the front of the range that is complete, segment by segment, and decommits it. WriteAt patches earlier bytes: in memory while they are not on their way to the file yet, else after the last write. A source of WriteDeferred must stay valid until LowestPendingSource no longer covers it.
Ordering: the file receives exactly the bytes in append order, at offset = Position at the time of the append, whatever order the copy tasks finish in. The writer only writes the prefix in which every deferred copy has finished (in whole segments until Close, then the rest), segment by segment in rising offset order, with up to OrderedFileWriterSettings::writesInFlight writes overlapping. WriteAt patches land either in memory before the write or, if the bytes have already been issued, are applied after the last write by Close; either way the final file holds the patched bytes.
Usage: Open, schedule Writer() once and any number of CopyTask objects (none is fine: the producer then runs the batches itself when a ring is full and in Close); then append with Write/WriteDeferred/EndBatch from one producer; then Close; then StopCopyTasks with the number of copy tasks, and wait for all tasks, the writer task included, to end before destroying the object (Close does not wait for that, see Close). Needs EnableAsyncIo. Producer calls are not thread-safe among themselves (one producer, a task or a thread after ConvertThreadToTask), and they may suspend it: WriteDeferred, EndBatch and LowestPendingSource can wait for or run copy batches, Close waits for the writer task.
Errors: Open returns a Win32 error code; Write, WriteDeferred and WriteAt report nothing. A failed Open or a failed append (address range exhausted, commit or allocation failed, WriteAt beyond Position) sets Failed, after which appends are dropped. A failed or short write makes the writer issue no further writes; Close returns false for any of these and for a failed size/patch finalization. The file is not deleted on failure.
See also
OrderedFileWriterSettings, OrderedFileWriterStats
Constructors
OrderedFileWriter
~OrderedFileWriter
Methods
Writer
The writer task.
Syntax
Tasks::Task& Writer() noexcept;Return value
Type: Tasks::Task&
The writer task, owned by this writer; schedule it, do not destroy it.
Remarks
Schedule it once (before or after Open); it ends with Close. It must be scheduled: Close waits until it has completed its last write. It suspends while there is nothing to write and while it waits for its writes. It may still be finishing when Close returns: to know when it has ended (before destroying the writer), count it into a Synchronization::TaskGroup or give it a notify callback (Tasks::Task::SetNotifyCallback) before scheduling it.
See also
Open
Creates the file.
Syntax
Windows::ErrorCode Open(
const wchar_t* path,
std::uint64_t bound,
const OrderedFileWriterSettings& settings = {}
) noexcept;Parameters
pathconst wchar_t*- Path of the file; must not be
nullptr; copied, need not stay valid. boundstd::uint64_t- The largest size the file may reach (only address space is reserved).
settingsconst OrderedFileWriterSettings&- Tuning of the writer; out-of-range values are corrected (see
OrderedFileWriterSettings).Default:{}
Return value
Type: Windows::ErrorCode
ERROR_SUCCESS if the file is open; otherwise the Win32 error code (see remarks) and the writer is marked failed.
Remarks
Call once, before any append, from the producer. Creates or truncates path (CREATE_ALWAYS, GENERIC_WRITE, FILE_SHARE_READ) and needs EnableAsyncIo. Does not suspend. Returns a Win32 error code: ERROR_NOT_ENOUGH_MEMORY for the bookkeeping, the CreateFileW/association error, or the error of reserving bound bytes (rounded up to a segment, plus one) of address space. On any error the writer is marked failed (Failed is true): appends are dropped, Close returns false, and the destructor frees what was allocated.
See also
Position
Bytes appended so far: the file offset of the next append. Producer only.
Syntax
std::uint64_t Position() const noexcept;Return value
Type: std::uint64_t
The number of bytes appended so far.
Failed
Write
Appends bytes.
Syntax
void Write(const void* data, std::size_t size) noexcept;Parameters
dataconst void*- The bytes;
nullptr: zeros. sizestd::size_t- Number of bytes to append; must fit within the bound given to
Open.
Remarks
Copies data at once (data may be reused after the call); never suspends.
See also
WriteDeferred
Appends room for bytes that a copy task fills later.
Syntax
void WriteDeferred(const void* source, std::size_t size) noexcept;Parameters
sourceconst void*- The bytes to copy; borrowed, must stay valid and unchanged until the copy has run (see remarks).
sizestd::size_t- Number of bytes to append;
0does nothing; must fit within the bound given toOpen.
Remarks
A copy task copies them from source after the next EndBatch. source is borrowed: it must stay valid and unchanged until LowestPendingSource is above it (or nullptr), at the latest until Close returns. May suspend the producer when the job ring is full (it then runs or waits for the oldest batch); a batch is submitted early when it reaches half the job ring. The bytes are not written to the file before the copy has finished.
See also
EndBatch
Hands the copies since the last batch to the copy tasks.
Syntax
void EndBatch() noexcept;Remarks
Also publishes the finished prefix to the writer (woken only when a new whole segment is complete). May suspend the producer when the batch ring is full. Without EndBatch the writer sees no progress until Close.
See also
LowestPendingSource
Lowest source address the unfinished copies still read (nullptr: none).
Syntax
const std::byte* LowestPendingSource() noexcept;Return value
Type: const std::byte*
The lowest source address still in use, or nullptr if no deferred copy is unfinished.
Remarks
Producer only; never suspends. Tracked per batch (its lowest source), so it is conservative: memory below the result may be reused or freed, memory at or above it may still be read. Most useful when sources are appended in rising address order.
See also
WriteAt
Overwrites bytes appended earlier.
Syntax
void WriteAt(std::uint64_t pos, const void* data, std::size_t size) noexcept;Parameters
posstd::uint64_t- File offset in bytes of the first byte to overwrite.
dataconst void*- The new bytes; copied at once, may be reused after the call.
sizestd::size_t- Number of bytes to overwrite;
pos+sizemust not exceedPosition.
Remarks
The range must lie before Position (the position does not change) and must not overlap a deferred copy that may still be running. Producer only; never suspends (it briefly takes the lock the writer issues under). A range beyond Position, or a failed allocation for a patch, sets Failed. Bytes already issued to the file are kept as a patch and written by Close after the last write.
Close
Finishes the file.
Syntax
bool Close() noexcept;Return value
Type: bool
true if all went well.
Remarks
Finishes the copies and writes, sets the size and applies the patches. Producer only. Suspends the producer until all batches are done (running them itself if no copy task does) and the writer task has completed all its writes, so the writer task (Writer) must have been scheduled. Size and patches go through a second, buffered handle after the overlapped one is closed. After a failed Open it only lets the writer task end and returns false. A second call does nothing but return the result again. On return the file is complete and closed and Stats is final, but the writer task may still be on its way out (it signals the producer just before it returns). Usage rule: destroy the writer only after the writer task has ended as well (its exit notification, e.g. through a Synchronization::TaskGroup), and after the copy tasks have ended (StopCopyTasks).
See also
StopCopyTasks
Lets copy tasks end.
Syntax
void StopCopyTasks(std::uint32_t count) noexcept;Parameters
countstd::uint32_t- Number of copy tasks to release; at least the number scheduled.
Remarks
Call after Close, or when giving up. count must be at least the number of copy tasks scheduled, or the rest stay suspended. Batches still submitted are run before a copy task ends. Any thread or task; never suspends.
See also
Stats
Counters (after Close).
Syntax
OrderedFileWriterStats Stats() const noexcept;Return value
Type: OrderedFileWriterStats
A snapshot of the counters.
Remarks
The writer task's counters are only safe to read once Close has returned.
See also
Fields
MaxWritesInFlight
Upper bound of OrderedFileWriterSettings::writesInFlight.
Syntax
static constexpr std::uint32_t MaxWritesInFlight = 16;SectorAlign
Sector size the last unbuffered write is padded to.
Syntax
static constexpr std::uint64_t SectorAlign = 4096;CopyTask class
A copy task: runs batches until StopCopyTasks.
Syntax
class CopyTask final : public Tasks::TaskRemarks
Schedule any number of them after Open; each suspends on a semaphore while there is no batch. The writer must outlive them: they end only after StopCopyTasks released them.
See also
CopyTask: Methods
Run
Runs submitted batches.
Syntax
void Run() noexcept override;Remarks
Ends once OrderedFileWriter::StopCopyTasks has released it and no batch is left.
Generated from ThinkMeta.ConcurrentTasks.API @ 137ec1d (2026-10-17).