Post
Ask Question

WCF: Streaming Large Data Files

WCF can be used to transfer image files, .pdf files or other large documents.  The most common way to do this is to use streaming.
 
Normally WCF processes messages in buffered mode.  This means incoming messages are not processed until the entire message has been received and buffered in memory.  Likewise, outgoing messages are not sent until the entire message has been buffered in memory.
 
Buffered transfer mode is also blocking.  This means the client is blocked until the entire message has been received and processed and the client receives a response.
Normally buffered transfer is fine.  When the messages contain large files, however, buffering creates serious performance problems.  The solution is to stream large messages instead of buffering.  Streaming allows the message recipient (which could be client or the service) to start processing the message before the entire message has been received.
 
To use streaming transfer, you need to configure the binding:
  1. <system.serviceModel>  
  2.     <services>  
  3.         <service name="MyDocumentService">  
  4.             <endpoint   
  5.                  binding="basicHttpBinding"   
  6.                  bindingConfiguration="DocumentServiceBinding"   
  7.                 ...     
  8.             </endpoint>  
  9.         </service>  
  10.     </services>  
  11.   
  12.     <bindings>  
  13.         <basicHttpBinding>  
  14.              <binding   
  15.                 name="DocumentServiceBinding"   
  16.                 messageEncoding="Mtom"   
  17.                 transferMode="Streamed"   
  18.                 maxBufferSize="65536"   
  19.                 maxReceivedMessageSize="5242880"   
  20.                 ...    
  21.                 <binding/>  
  22.         </basicHttpBinding>  
  23.     </bindings>  
  24.   
  25. </system.serviceModel>  
By default transferMode is Buffered.  As shown above, transferMode is Streamed, which means all requests and responses are streamed.  Other options include StreamedRequest (= streamed requests with buffered responses) and StreamedResponse (= buffered requests and streamed responses).
 
Restrictions
 
There are a number of important restrictions when you use streaming.
 
The message body cannot have a digital signature.  A digital signature cannot be checked until the entire message has been received.  That would require buffered transfer.  Likewise, the message cannot be encrypted.  You can still use transport level security such as SSL, but the message itself cannot be encrypted.
 
WCF reliable messaging requires buffering in order to guarantee that all messages are processed in the correct order.
 
Therefore, if you are streaming, there can be no digital signature, no message encryption and no reliable messaging.  This also rules out many binding types.  The only bindings (in .Net 4.0) allowed are:
  • BasicHttpBinding
  • BasicHttpContextBinding
  • NetTcpBinding
  • NetNamedPipeBinding
NetTcpBinding and NetNamedPipeBinding include reliable delivery and session support, so the lack of reliable messaging is not as important.
 
A common technique is to put all the operations which require streaming into one service endpoint which uses streaming, and to put all the other operations in a different endpoint which does not use streaming.
 
Contracts
 
When you're streaming, the message body must contain the stream and nothing else.  Any other data has to go in the message header.  So if you want to use a web service to upload documents and you also want to provide information about the document such as the author, the document stream goes in the message body and all other information goes in the header.
 
It should look something like the following.  Notice the document name and author go in the message header, while the document Stream itself is in the message body.
  1. [MessageContract]  
  2. public class DocumentUpload  
  3. {  
  4.     [MessageHeader]  
  5.      public string author;  
  6.   
  7.     [MessageHeader]  
  8.      public string documentName;  
  9.   
  10.     [MessageBodyMember]  
  11.      public Stream data;  
  12. }  
  13.   
  14. [ServiceContract]  
  15. interface IMyDocumentService  
  16. {  
  17.      [OperationContract]  
  18.      SaveDocument(DocumentRequest upload);  
  19.   
  20.     [OperationContract]  
  21.      RetrieveDocument(DocumentRequest download);  
  22. }  
MTOM
 
In the sample binding above, message encoding is set to MTOM.  This is not the default message encoding.  
 
MTOM creates a complex message header, and there is a good bit of processing required to do so.  Once this is done, however, the data itself is transferred very efficiently.  So MTOM makes sense only when the documents being transferred are large.  On the MSDN website, MTOM is only recommended when messages include more than 1 KB of binary data.  See Large Data and Streaming.
 
Maximum Message Size and Maximum Buffer Size
 
It's important to set a maximum message size.  Without it, extremely large files can simply overwhelm the service.  The default maximum message size is 64 Kb.  In the sample binding above, maximum message size is set to 5242880 bytes (= 5 Mb).  The larger the maximum message size, the more vulnerable your service becomes to denial of service attacks or simple server overload.
 
It's also important to set a maximum buffer size.  This is the maximum amount of memory that the service is permitted to devote to a single call.  Even when streaming is enabled, the service still buffers the message header.  Without setting a maximum buffer size, your service may still be vulnerable to denial of service attacks which include extremely large message headers.


Similar Articles