Download attachments from SOAP web service- streaming of contents

SOAP Client for Downloading Attachments - Streaming

We have seen uploading of larger data files using SOAP with attachments in the previous post(Upload attachments using SOAP). We will see how to download/stream larger files from SOAP web service.


Java Mail API Problem
The SOAP web service client for downloading attachments we will be discussing uses the MimeMessage structure but we will avoid using the JAVA MAIL package as it is not efficient while downloading the contents. If you use MimeMessage java api to get the MimeBodyPart then you'll be in trouble as MimeMessage loads the entire content as byte array into memory for calculating the size and number of MimeBodyParts. So we will avoid using it for larger files if you have small JVM memory.

How to stream larger files
The memory intensive operation can be avoided using our own custom code to parse the response from the SOAP web service. 

The following example shows the SOAP message response with the attachments. This example is for Oracle UCM's GetFileByName operation. 

The SOAP response message package is constructed using the  Multipart/Related  media type. The structure of the response stream is as follows:

1. The primary SOAP message must be carried in the root body part of the Multipart/Related structure. Consequently the type parameter of the Multipart/Related media header will always equal the Content-Type header for the primary SOAP 1.1 message, i.e., text/xml.
2. Referenced MIME parts must contain either a Content-ID MIME header.

Content-Type: multipart/related; type="text/xml"; start="rootContent"; boundary="----=_Part_12_21424854.1424891476173"

SOAPAction: "http://www.stellent.com/CheckIn/"

Content-Length: 1818

------=_Part_12_21424854.1424891476173

Content-Type: text/xml; charset=UTF-8

Content-Transfer-Encoding: 8bit

Content-ID: rootContent

<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:get=\"http://www.stellent.com/GetFile/\">

   <soapenv:Header/>

   <soapenv:Body>

      <get:GetFileByName>

         <get:dDocName>Test.txt</get:dDocName>    <get:revisionSelectionMethod>LatestReleased</get:revisionSelectionMethod>

         <get:rendition>primary</get:rendition>

         <get:extraProps>

            <get:property>

               <get:name>soapResponseType</get:name>

               <get:value>Multipart/Related</get:value>

            </get:property>

         </get:extraProps>

      </get:GetFileByName>

   </soapenv:Body>

</soapenv:Envelope>

------=_Part_12_21424854.1424891476173

Content-Type: text/plain

Content-Transfer-Encoding: 7bit

Content-ID: test.log

Hello

------=_Part_12_21424854.1424891476173--

As you can see above the response stream starts with boundary (------=_Part_12_21424854.1424891476173). Each MimeBodyParts are separated by boundary. The content of each MimeBodyPart starts after blank line. The blank line is used to seperate the headers and content of each MimeBodyPart. Using apache LineInputStream we read each line until we encounter the blank line for the attachment content. Once the blank line preceeding the attachment body is read we will switch back to BufferInputStream for efficiency. Don't forget to ignore the boundary line which indicates the end of response and exclude it from writing into the file.

1  import java.io.BufferedInputStream;    2  import java.io.BufferedOutputStream;    3  import java.io.FileOutputStream;    4  import java.io.IOException;    5  import java.io.InputStream;    6  import java.util.Properties;    7       8  import javax.activation.DataHandler;    9  import javax.mail.Session;   10  import javax.mail.internet.MimeBodyPart;   11  import javax.mail.internet.MimeMessage;   12  import javax.mail.internet.MimeMultipart;   13  import javax.mail.internet.PreencodedMimeBodyPart;   14      15  import org.apache.axis.utils.StringUtils;   16  import org.apache.commons.httpclient.Header;   17  import org.apache.commons.httpclient.HeaderElement;   18  import org.apache.commons.httpclient.HttpClient;   19  import org.apache.commons.httpclient.HttpException;   20  import org.apache.commons.httpclient.HttpMethod;   21  import org.apache.commons.httpclient.NameValuePair;   22  import org.apache.commons.httpclient.methods.PostMethod;   23      24  import com.sun.mail.util.LineInputStream;   25      26  public class UCMDownloader {   27      28          public static void main(String[] args) {   29                  HttpClient client = new HttpClient();   30                  PostMethod method = new PostMethod(   31                                  "http://www.oracle.com:16200/_dav/cs/idcplg");   32                  long start = System.currentTimeMillis();   33                  String fileName = "Test.txt";   34      35                  try {   36      37                          method.addRequestHeader("SOAPAction",   38                                          "\"http://www.stellent.com/GetFile/\"");   39      40                          MimeMessage message = new MimeMessage(   41                                          Session.getDefaultInstance(new Properties()));   42                          MimeMultipart mp = new MimeMultipart();   43      44                          // Create the first body part   45                          StringBuffer requestXML = constructRequest(fileName);   46                          long requestSize = requestXML.length();   47                          MimeBodyPart rootPart = new PreencodedMimeBodyPart("8bit");   48                          rootPart.setContentID("rootContent");   49                          rootPart.setDataHandler(new DataHandler(new RequestXmlDataSource(   50                                          requestXML.toString())));   51      52                          mp.addBodyPart(rootPart, 0);   53      54                          message.setContent(mp);   55                          message.saveChanges();   56      57                          String cType = message.getHeader("Content-Type")[0];   58                          int idx = cType.indexOf("boundary");   59                          int boundryLength = 0;   60                          if (idx != -1) {   61                                  boundryLength = cType.substring(idx + 9).length();   62                          }   63      64                          long contentLength = (boundryLength * 2) + 2 + requestSize + 6;   65      66      67                          method.setDoAuthentication(true);   68                          method.setRequestBody(requestXML.toString());   69      70                          method.addRequestHeader("Content-Length",   71                                          String.valueOf(requestXML.length()));   72                          method.addRequestHeader("Content-Type", "text/xml;charset=UTF-8");   73                          method.getParams().setParameter("http.socket.timeout",   74                                          new Integer(0));   75      76                          // send the request   77                          int status = client.executeMethod(method);                  78                            79                          // process the response   80                          contentLength = getContentLength(method);   81                            82                          String boundary = getBoundary(method);   83                          if (boundary != null) {   84                                  // Boundary foe each Body Part   85                                  boundary = "--" + boundary;   86                          }   87      88                          BufferedInputStream bin = new BufferedInputStream(   89                                          method.getResponseBodyAsStream());   90                          LineInputStream lin = new LineInputStream(bin);   91      92                          boolean rootBoundaryEncountered = false;   93                          boolean attachmentBoundaryEncountered = false;   94      95                          String line = null;   96                          long readSoFar = 0;   97                          while ((line = lin.readLine()) != null) {   98                                  readSoFar += line.length() +2; // 2 for \r\n   99                                  if (line.equals(boundary)) {  100                                          if (!rootBoundaryEncountered) {  101                                                  // boundary for root body part  102                                                  rootBoundaryEncountered = true;  103                                                  continue;  104                                          } else { // root bodypart already read so we are ready to  105                                                                  // read next body part  106                                                  attachmentBoundaryEncountered = true;  107                                          }  108                                  } else if (line.length() == boundary.length() + 2  109                                                  && line.startsWith(boundary) && line.endsWith("--")) {  110                                                  // End of response  111                                          line = null;  112                                          break;  113                                  } else if (!StringUtils.isEmpty(line)) {  114                                          System.out.println(line);  115                                          // header part and SOAP XML goes here  116                                  } else { // the attachment content always follows the blank line  117                                          if (attachmentBoundaryEncountered) {  118                                                  // save the attachment content into file  119                                                  saveAttachment(bin, boundary, contentLength, readSoFar);  120                                                  break;  121                                          }  122                                  }  123                          }  124     125                  } catch (HttpException e) {  126                          e.printStackTrace();  127                  } catch (IOException e) {  128                          e.printStackTrace();  129                  } catch (Exception e) {  130                          e.printStackTrace();  131                  } finally {  132                          long end = System.currentTimeMillis();  133                          System.out.println("Time taken for [" + fileName + "]: "  134                                          + (end - start));  135                          // release any connection resources used by the method  136                          method.releaseConnection();  137                  }  138          }  139     140          public static StringBuffer constructRequest(String fileName) {  141                  StringBuffer request = new StringBuffer();  142                  request.append("<soapenv:Envelope  143                          xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"  144                          xmlns:get=\"http://www.stellent.com/GetFile/\">\r\n"  145                                  + "   <soapenv:Header/>\r\n"  146                                  + "   <soapenv:Body>\r\n"  147                                  + "      <get:GetFileByName>\r\n"  148                                  + "         <get:dDocName>"+ fileName +"</get:dDocName>\r\n"  149                                  + "         <get:revisionSelectionMethod>  150                                  LatestReleased</get:revisionSelectionMethod>\r\n"  151                                  + "         <get:rendition>primary</get:rendition>\r\n"  152                                  + "         <get:extraProps>\r\n"  153                                  + "            <get:property>\r\n"  154                                  + "               <get:name>soapResponseType</get:name>\r\n"  155                                  + "               <get:value>Multipart/Related</get:value>\r\n"  156                                  + "            </get:property>\r\n"  157                                  + "         </get:extraProps>\r\n"  158                                  + "      </get:GetFileByName>\r\n"  159                                  + "   </soapenv:Body>\r\n"  160                                  + "</soapenv:Envelope>");  161     162                  return request;  163          }  164     165          public static String getBoundary(HttpMethod httpMethod) {  166                  Header h = null;  167                  String boundary = null;  168     169                  h = httpMethod.getResponseHeader("Content-Type");  170     171                  if (h != null) {  172                          HeaderElement[] elements = h.getElements();  173     174                          for (HeaderElement element : elements) {  175                                  String name = element.getName().toUpperCase();  176                                  if (name.startsWith("MULTIPART/")) {  177                                          NameValuePair parameter = element  178                                                          .getParameterByName("boundary");  179                                          if (parameter != null) {  180                                                  boundary = parameter.getValue();  181                                          }  182                                  }  183                          }  184                  }  185     186                  return boundary;  187          }  188           189          public static long getContentLength(HttpMethod httpMethod) {  190                  Header h = null;  191                  String contentLen = "0";  192     193                  h = httpMethod.getResponseHeader("Content-Length");  194                  if (h != null) {  195                          contentLen = h.getValue();  196                  }  197                   198                  long contentLength = Long.valueOf(contentLen).longValue();  199                   200                  return contentLength;  201          }  202     203          public static void saveAttachment(BufferedInputStream responseStream,  204                  String boundary, long totalBytes, long totalBytesRead) {  205                  try {  206                          if (responseStream != null) {  207                                  BufferedOutputStream out = new BufferedOutputStream(  208                                                  new FileOutputStream(  209                                                                  "c:\\temp\\test.txt"));  210                                  byte data[] = new byte[4096];  211                                  int bytesread = -1;  212                                  boolean boundaryEndHit = false;  213                                  int boundaryEnd = boundary.length()+6;  214                                   215                                  System.out.println("Attachment Starts Here ========= ");  216                                   217                                  while ((bytesread = responseStream.read(data)) != -1) {  218                                          totalBytesRead += bytesread;  219                                           220                                          if (totalBytesRead <= (totalBytes -  221                                                  boundaryEnd)) {  222                                                  out.write(data, 0, bytesread);  223                                          } else if (totalBytesRead > (totalBytes -  224                                                  boundaryEnd)) {  225                                                  out.write(data, 0, bytesread -  226                                                          (int) (totalBytesRead -  227                                                                  (totalBytes - boundaryEnd)));  228                                          }  229                                  }  230                                   231                                  responseStream.close();  232                                  out.close();  233                          }  234                  } catch (Exception e) {  235                          e.printStackTrace();  236                  }  237          }  238  }

The example was tested with 10 concurrent thread each downloaded 160MB of file from web service and the JVM memory utilization was mere 20MB.