StreamedFileReader class
A file streamed into memory, addressed like a mapped file, with bounded memory use.
- Use
import ThinkMeta.ConcurrentTasks.AsyncIo.StreamedFileReader;- Namespace
ThinkMeta::ConcurrentTasks::AsyncIo- Inherits
Task
Syntax
class StreamedFileReader : public Tasks::TaskRemarks
The file is streamed into one reserved address range, so every byte is addressed by its file position: the reader task (Run) commits and reads segment after segment, at most StreamedFileReaderSettings::readAhead past the oldest byte still in use, or as far as a user waits for. Users report the bytes they are done with (Release); whole segments below the lowest such position of all users are decommitted. Memory stays bounded by the read-ahead, not by the file size, only while the users progress evenly: committed is everything from the lowest release of all users up to the read-ahead past it or the farthest Wait, whichever is higher. A fast user's Wait far ahead is served while a slow user holds the front, and every user slot starts at 0, so nothing is decommitted until each user has released. Usage rule: keep the users within the read-ahead of each other and let every slot opened release (~0 when a user is done), or memory grows up to the file size. Wait and Release may be called from tasks and threads; Release of one user slot from one caller at a time.
Usage: Open, then schedule the object itself (it is the reader task) once; consumers call Wait(end) before touching bytes below end and Release when done with them. Needs EnableAsyncIo. The file is read in rising offset order; loaded only grows. Wait suspends the calling task (an OS thread must have called ConvertThreadToTask). Errors: Open returns a Win32 error code; a read error is kept in ReadError and does not stop the stream - Wait still returns, the affected bytes read as zero, so consumers must check ReadError. Destroy the object only after the reader task has ended (Run has returned).
See also
Constructors
StreamedFileReader
Creates a reader with no file.
Syntax
StreamedFileReader() noexcept = default;Remarks
Call Open before scheduling it.
~StreamedFileReader
Releases the address range.
Syntax
~StreamedFileReader() noexcept;Remarks
The reader task must have ended (no read in flight).
Methods
Open
Opens the file and reserves its address range.
Syntax
Windows::ErrorCode Open(
const wchar_t* path,
const StreamedFileReaderSettings& settings = {},
std::uint32_t users = 1
) noexcept;Parameters
pathconst wchar_t*- Path of an existing file; must not be
nullptr; need not stay valid after the call. settingsconst StreamedFileReaderSettings&- Tuning of the reader; out-of-range values are corrected (see
StreamedFileReaderSettings).Default:{} usersstd::uint32_t- How many
Releaseslots are in use.Default:1
Return value
Type: Windows::ErrorCode
ERROR_SUCCESS if the file is open and its range reserved; otherwise the Win32 error code (see remarks).
Remarks
Call once, before the reader task is scheduled. Opens OPEN_EXISTING, GENERIC_READ, FILE_SHARE_READ; needs EnableAsyncIo. Does not suspend. users 0 counts as 1, more than MaxUsers as MaxUsers; slots in use start at 0, so nothing is decommitted until each of them has released. Returns the Win32 error of CreateFileW/association, GetFileSizeEx or the reservation of the file size (rounded up to a segment, plus one).
See also
Data
Size
The file size, fixed at Open.
Syntax
std::uint64_t Size() const noexcept;Return value
Type: std::uint64_t
The file size in bytes.
Wait
Waits until the bytes before end are loaded.
Syntax
std::uint64_t Wait(std::uint64_t end) noexcept;Parameters
endstd::uint64_t- File offset in bytes up to which the data is needed; clamped to
Size.
Return value
Type: std::uint64_t
How far the file is loaded: at least end, less only when the reader was stopped before (Stop).
Remarks
end is clamped to Size. Returns at once if the bytes are loaded; otherwise asks the reader to read up to end (even beyond the read-ahead) and suspends the calling task until they are. A Wait beyond the read-ahead commits memory past the bound, kept until all users have released it (see the class remarks). Any number of tasks may wait at once. Bytes are loaded also when their read failed (they are zero then): check ReadError. The reader task must be scheduled.
See also
Release
Reports that a user is done with the bytes before a position.
Syntax
void Release(std::uint32_t user, std::uint64_t below) noexcept;Parameters
userstd::uint32_t- The user slot.
belowstd::uint64_t- Bytes before it are no longer used;
~0: all of them.
Remarks
Never suspends. Whole segments below the minimum over all slots (and below what is loaded) are decommitted, which also lets the reader read further ahead. Bytes below below must not be touched afterwards by that user. below should not decrease for a slot; user >= MaxUsers is ignored. If another caller is decommitting at the same moment, this call leaves the work to a later Release. A slot that never releases keeps everything from its position on committed.
See also
Stop
Stops the reader.
Syntax
void Stop() noexcept;Remarks
The reader issues no more reads and ends once the reads in flight are done. Any task or thread; never suspends. Waiters are then released with what was loaded. The reader also ends on its own at the end of the file.
See also
ReadError
The first read error (Windows::Error::Success if none).
Syntax
Windows::ErrorCode ReadError() const noexcept;Return value
Type: Windows::ErrorCode
ERROR_SUCCESS if every read so far succeeded; otherwise the Win32 error code of the first failed read.
Remarks
After an error the pages read as zero.
Waited
Seconds all callers of Wait waited together.
Syntax
double Waited() const noexcept;Return value
Type: double
The summed waiting time in seconds (microsecond resolution).
PeakWindow
Largest committed range (bytes) while reading.
Syntax
std::uint64_t PeakWindow() const noexcept;Return value
Type: std::uint64_t
The largest committed range in bytes.
Run
The reader task.
Syntax
void Run() noexcept override;Remarks
Schedule it once after Open; it ends at the end of the file or after Stop. Suspends while it may not read further (read-ahead used up and nobody waiting) and while it waits for the oldest read. Reads complete in any order on the I/O thread but are accounted in issue order, so loaded grows contiguously. A read error (or a short read: ERROR_HANDLE_EOF) is recorded once (ReadError); later segments are committed but not read. Closes the file before it ends.
Fields
MaxReadsInFlight
Upper bound of StreamedFileReaderSettings::readsInFlight.
Syntax
static constexpr std::uint32_t MaxReadsInFlight = 16;MaxUsers
SectorAlign
Sector size the last unbuffered read is rounded up to.
Syntax
static constexpr std::uint64_t SectorAlign = 4096;Generated from ThinkMeta.ConcurrentTasks.API @ 137ec1d (2026-10-17).