Combine Streams in One Delphi TStream Object

Why Combine Multiple Streams?

When working with Delphi applications that process large volumes of data, it is common to encounter situations where a single logical data set is distributed across several smaller streams. Examples include reading from multiple files, handling separate parts of a database backup, or assembling multimedia content that is split into chapters or tracks. In such cases, developers often need a single stream interface that presents the data as one continuous sequence. The most straightforward approach is to copy each part into a TMemoryStream and then expose that memory stream to the rest of the code. This technique works fine for small data sets, but it brings two significant drawbacks that become apparent as the data size grows or as the number of streams increases.

The first drawback is memory consumption. A TMemoryStream allocates a contiguous block of virtual memory that holds the entire combined data. If you have several gigabytes of data spread across dozens of streams, you may end up allocating more memory than your application can handle comfortably. The operating system may swap portions of the memory to disk, causing a noticeable slowdown and, in extreme cases, running out of address space on 32‑bit systems. The second drawback is performance. Copying each stream into memory requires a full read and write operation for each piece, which introduces overhead and stalls the thread until all data is transferred. For applications that need to be responsive or that process streams on the fly, this approach can be a bottleneck.

Because of these issues, many developers look for a solution that treats the individual streams as if they were one logical stream, but without the need to consolidate the data physically. Delphi’s TStream abstract class makes it possible to write custom stream types. By deriving a new class from TStream, you can override the standard Read, Seek, and Write methods and define how the data is accessed. The key idea is to keep the original streams in memory and to manage a virtual position that maps to the appropriate underlying stream at any point in time.

This approach has several benefits. First, it eliminates the need for large contiguous memory allocations; each underlying stream remains in its original state. Second, read and seek operations can be performed directly on the relevant stream, which keeps the overhead minimal. Third, you avoid unnecessary copying of data, which is particularly valuable when working with streams that are already being read from disk or network sources. The resulting class is lightweight, flexible, and scales well with both the number of streams and their individual sizes.

Of course, implementing a custom stream type is not without its challenges. You must keep track of the cumulative sizes of all constituent streams, calculate offsets correctly, and handle the case where a read operation spans multiple streams. Care must be taken to ensure that the Seek method can translate a virtual position into the correct stream index and internal offset. Despite these intricacies, the end result is a robust, read‑only composite stream that can be used wherever a standard TStream is expected.

In what follows we’ll walk through the design of such a class, focusing on the core methods that make it functional. The class is named TMultiStream and inherits from TStream. It exposes a constructor that accepts a list of streams, a Read method that fetches data across boundaries, a Seek method that positions the virtual pointer, and a Write method that raises an exception to indicate that the stream is read‑only. After understanding the implementation, we’ll see a concise usage example that demonstrates how easy it is to plug this class into existing code.

When you have a clear picture of the trade‑offs and the intended usage scenario, the decision to use a composite stream instead of a memory buffer becomes straightforward. The next section describes how to implement the TMultiStream class in detail, including the critical logic that keeps the virtual position in sync with the underlying streams.

Building the TMultiStream Class

The foundation of a composite stream is a data structure that holds references to the constituent streams and keeps a running total of their sizes. In Delphi, this is typically a TList or a TArray that stores the streams in the order they should appear in the combined view. The constructor of TMultiStream accepts this list and calculates the total length by iterating through each stream and summing its Size property. This total length becomes the Length property of the composite stream.

Next, the Read method needs to handle the possibility that a single read request may span more than one underlying stream. The implementation follows these steps: first, verify that the virtual position is not beyond the total length; if it is, return zero to signal end of stream. Then, determine which stream contains the current position by scanning the cumulative sizes. Once the appropriate stream is found, calculate the offset within that stream by subtracting the start position of that stream from the virtual position. The method reads as much data as possible from the current stream into the caller’s buffer, up to the remaining request size or the end of the stream. If the request size is larger than the remaining data in the current stream, the method updates the buffer offset and continues reading from the next stream until the buffer is full or all streams have been exhausted. After the loop, the virtual position is incremented by the number of bytes actually read.

The FindStream helper routine performs a linear search through the stream list, accumulating sizes until it identifies the stream that contains the requested position. Although a binary search could be faster for very long lists, the typical use case involves a handful of streams, so the simple approach keeps the code straightforward.

The Seek method must translate a virtual position relative to the entire composite stream into a position within a specific underlying stream. This requires the same logic used in Read to find the correct stream and offset. Once located, the method simply sets the Position property of the target stream to the calculated offset. The method supports the standard TSeekOrigin values (so). If the requested position is beyond the total size, the method returns the size of the stream and places the virtual position at the end. Seek also updates the internal FPosition field, which is used by subsequent Read operations.

Because TMultiStream is intended as a read‑only view, the Write method is intentionally left unimplemented. The simplest approach is to raise an EInvalidOperation exception, signaling that attempts to modify the composite stream are unsupported. This keeps the contract clear and prevents accidental writes that would otherwise corrupt the underlying streams.

With these core methods in place, the class behaves like any other TStream. It reports its Length, supports Position queries, and can be passed to routines that expect a stream without any modification to those routines. The internal bookkeeping ensures that each read or seek operation correctly maps to the appropriate underlying stream, preserving the illusion that all streams are concatenated.

It is worth noting that the composite stream does not modify the underlying streams themselves. If a stream is closed or destroyed elsewhere in the application, TMultiStream will attempt to read from a freed object, which will result in an access violation. Therefore, it is important to keep the constituent streams alive for as long as the composite stream is in use, or to copy them into a persistent container before creating the TMultiStream.

When you have the implementation ready, the next step is to see how to incorporate TMultiStream into a real Delphi project. The following section demonstrates a concise example that reads from three distinct memory streams and processes the combined data as a single block.

Using TMultiStream in Practice

To illustrate how the composite stream integrates into everyday code, consider a scenario where you have three separate TMemoryStream objects containing pieces of a larger message. Perhaps each stream was filled by a different part of the application, or they were loaded from separate files that represent different sections of a document. Instead of copying the data into a new buffer, you can create a TMultiStream that presents these three streams as a single sequence.

Here is a minimal example that shows how to build the composite stream, read its contents into a string, and then display the result. The code uses standard Delphi units and can be compiled in any recent Delphi IDE.

Running the procedure prints:

Notice that the code does not copy any data between streams. The TMultiStream simply redirects the read call to the appropriate underlying stream, preserving memory and processing time. Because the composite stream is read‑only, the example does not attempt any write operations, and the Write method would raise an exception if called.

In a more complex application you might use TMultiStream to stream a large video file that has been split into multiple parts, to process a multi‑part backup, or to merge chunks of a network protocol that arrive in separate packets. The key advantage is that the rest of your code can treat the data as a contiguous TStream, keeping your business logic clean and focused on the data itself rather than the plumbing that stitches it together.

To keep the application robust, it is advisable to manage the lifetimes of the constituent streams carefully. If any of the streams are opened from a file, you should keep the file open for the duration of the composite stream, or you should copy the file content into a TMemoryStream before creating the composite. This prevents the composite from attempting to read from a closed file handle, which would result in a crash.

For those interested in seeing the full source code, the example program and the TMultiStream implementation are bundled in a downloadable archive. The zip file contains the Delphi unit that defines TMultiStream, a test application, and documentation that explains the class API in detail. The code is available for download at the following link: