Professional BizTalk® Server 2006

13.1. Processing Large Messages

In my experience, solutions that must process large messages from end to end tend to be fairly rare. Generally, large input files can be debatched by a receive pipeline into smaller discrete messages. This debatching removes the need to worry about a large message and enables processing to be parallelized.

However, development of a BizTalk solution that must deal with large messages through pipelines and orchestrations can often prove problematic.

The golden rule when dealing with large messages might seem obvious, but it is often broken: You must limit or (the ideal approach) remove any operations that require the entire message to be loaded into memory.

BizTalk itself operates a streaming approach whereby the message is read through the use of a stream, meaning that the entire message doesn't have to be present in memory at any point in time. (This is discussed in detail in Chapter 4.) This streaming approach is used by the built-in pipeline components such as the Flat File Disassembler, XML Disassembler, and in fact the whole BizTalk engine, meaning that BizTalk will do the right thing with regard to processing large messages, but you need to ensure that any custom pipeline components, orchestrations, or helper classes follow this approach.

An exception to this rule occurs when XML messages containing a CDATA element that holds a large payload (greater than 1MB) are received. CDATA elements cannot be streamed; they must be requested in their ...