Pipeline in BizTalk Server

Introduction

BizTalk is a message-based system receiving and sending data inside messages. Sometimes the incoming and outgoing messages must be processed to fit to external formats. Pipelines, attached to send ports and receive locations, are the components through which the messages pass; then the data format is recognized and can be validated or changed if necessary; as well as the metadata is extracted and added to the message context. In this article, I will cover the principles of pipeline architecture in BizTalk Server.

Pipeline Architecture

In BizTalk Server 2006, pipelines can be executed from messaging ports as well as from orchestrations. In order to develop a pipeline, we need to go to BizTalk project and select the Add New Item option and from the dialog box click on the Pipeline files node in the Categories tree and then from the Templates pane, choose the type of pipeline such as Receive Pipeline (see Figure 1) or Send Pipeline (see Figure 2).



Figure 1



Figure 2

After that the Pipeline Designer is displayed (see Figure 3 for the Receive Pipeline and Figure 4 for the Send Pipeline).



Figure 3



Figure 4

As you can a pipeline comprises of stages and these stages differ for both send and receive pipeline, but essentially they enable to logically organize the flow of execution of the logic within the pipeline and develop custom pipeline components using the APIs. Each stage has the ability to contain multiple pipeline components (up to 255, except the Assemble stage) designed to execute in a particular stage. Pipeline components inside one stage execute sequentially from the first to the last; thus the output of component is the input of the next one. The exception is components inside the Disassemble stage which execute following the first-match rule, that is, the first component, that is able to process the message, is chosen to execute and remaining components are ignored.

You can customize the pipelines; this includes the minimum and maximum number of components in a given stage, the execution mode of the stages, and the name of the stage. You can find the pipeline configuration files in the %PROGRAM_FILES%\Microsoft BizTalk Server 2006\Developer Tools\Pipeline Policy Files\ directory.

Receive pipelines have four stages:

1. Decode. It can be used to prepare the message for the Disassemble stage for example decrypting, uncompress and decoding a message.
   
2. Disassemble. It can be used to produce multiple messages based on the input message by de-batching the incoming message into smaller messages. It can also recognize the format of incoming messages and then process it as well as promoting properties into the message context. For example, incoming messages formatted as XML are processed by the XML Disassembler or incoming messages formatted as CSV flat file will be parsed and prepared as individual XML units.
    
3. Validate. It's mainly used to validate the disassembled messages. For example, verifying the XML Schema of the message to be of a certain type using the XML Validator component.
    
4. Party resolution. It's used to determine the party that BizTalk receive the message from. This can be achieved by mapping the sender's digital certificate or sender's security identifier.

Send pipelines have three stages:

1. Pre-assemble. It prepares the message for the outbound process.
   
2. Assemble. It's responsible for combining multiple messages into one large message with the format that will be sent over the wire (Aggregating multiples messages in a batch). In this step, we can change XML format of a message into a flat file, or possible adding envelopment to the XML message. In this stage, you can also demote the properties (those properties promoted in the Disassemble stage of the Receive pipeline and used for the routing of the message) from the context message.
    
3. Encode. It's responsible for writing the outgoing message in a fashion way in order to be understood by the target system. It involves encoding, compression, encrypting, and signing the message.

Default Pipelines

BizTalk ships with default pipelines. The first type of pipeline is pass-through pipelines. There is one of these for receiving (PassThruReceive) and one for sending (PassThruTransmit). The pass-through pipelines are an empty pipeline.

The two other default pipelines are XMLReceive and XMLTransmit and they're responsible for XML processing on receive and send sides, respectively. The XMLReceive pipeline has all the stages empty but the Disassemble stage with the XMLDisassembler component and Party Resolution stage with Party Resolution components. The XMLTransmit has also all the stages empty but the Assemble stage with the XMLAssembler component.

Conclusion

In this article, I covered the principles of pipeline architecture in BizTalk Server. Now you can customized your own pipeline to apply for your own business scenarios.