Tuesday, March 10, 2015

Download attachments from SOAP web service- streaming of contents

SOAP Client for Downloading Attachments - Streaming

We have seen uploading of larger data files using SOAP with attachments in the previous post(Upload attachments using SOAP). We will see how to download/stream larger files from SOAP web service.


Java Mail API Problem
The SOAP web service client for downloading attachments we will be discussing uses the MimeMessage structure but we will avoid using the JAVA MAIL package as it is not efficient while downloading the contents. If you use MimeMessage java api to get the MimeBodyPart then you'll be in trouble as MimeMessage loads the entire content as byte array into memory for calculating the size and number of MimeBodyParts. So we will avoid using it for larger files if you have small JVM memory.

How to stream larger files
The memory intensive operation can be avoided using our own custom code to parse the response from the SOAP web service. 

The following example shows the SOAP message response with the attachments. This example is for Oracle UCM's GetFileByName operation. 

The SOAP response message package is constructed using the  Multipart/Related  media type. The structure of the response stream is as follows:
  1. The primary SOAP message must be carried in the root body part of the Multipart/Related structure. Consequently the type parameter of the Multipart/Related media header will always equal the Content-Type header for the primary SOAP 1.1 message, i.e., text/xml.
  2. Referenced MIME parts must contain either a Content-ID MIME header.
Content-Type: multipart/related; type="text/xml"; start="rootContent"; boundary="----=_Part_12_21424854.1424891476173"
SOAPAction: "http://www.stellent.com/CheckIn/"
Content-Length: 1818


------=_Part_12_21424854.1424891476173
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-ID: rootContent

<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:get=\"http://www.stellent.com/GetFile/\">
   <soapenv:Header/>
   <soapenv:Body>
      <get:GetFileByName>
         <get:dDocName>Test.txt</get:dDocName>    <get:revisionSelectionMethod>LatestReleased</get:revisionSelectionMethod>
         <get:rendition>primary</get:rendition>
         <get:extraProps>
            <get:property>
               <get:name>soapResponseType</get:name>
               <get:value>Multipart/Related</get:value>
            </get:property>
         </get:extraProps>
      </get:GetFileByName>
   </soapenv:Body>
</soapenv:Envelope>
------=_Part_12_21424854.1424891476173
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Content-ID: test.log

Hello

------=_Part_12_21424854.1424891476173--

As you can see above the response stream starts with boundary (------=_Part_12_21424854.1424891476173). Each MimeBodyParts are separated by boundary. The content of each MimeBodyPart starts after blank line. The blank line is used to seperate the headers and content of each MimeBodyPart. Using apache LineInputStream we read each line until we encounter the blank line for the attachment content. Once the blank line preceeding the attachment body is read we will switch back to BufferInputStream for efficiency. Don't forget to ignore the boundary line which indicates the end of response and exclude it from writing into the file.


   1 
import java.io.BufferedInputStream;
   2 
import java.io.BufferedOutputStream;
   3 
import java.io.FileOutputStream;
   4 
import java.io.IOException;
   5 
import java.io.InputStream;
   6 
import java.util.Properties;
   7 
 
   8 
import javax.activation.DataHandler;
   9 
import javax.mail.Session;
  10 
import javax.mail.internet.MimeBodyPart;
  11 
import javax.mail.internet.MimeMessage;
  12 
import javax.mail.internet.MimeMultipart;
  13 
import javax.mail.internet.PreencodedMimeBodyPart;
  14 
 
  15 
import org.apache.axis.utils.StringUtils;
  16 
import org.apache.commons.httpclient.Header;
  17 
import org.apache.commons.httpclient.HeaderElement;
  18 
import org.apache.commons.httpclient.HttpClient;
  19 
import org.apache.commons.httpclient.HttpException;
  20 
import org.apache.commons.httpclient.HttpMethod;
  21 
import org.apache.commons.httpclient.NameValuePair;
  22 
import org.apache.commons.httpclient.methods.PostMethod;
  23 
 
  24 
import com.sun.mail.util.LineInputStream;
  25 
 
  26 
public class UCMDownloader {
  27 
 
  28 
        public static void main(String[] args) {
  29 
                HttpClient client = new HttpClient();
  30 
                PostMethod method = new PostMethod(
  31 
                                "http://www.oracle.com:16200/_dav/cs/idcplg");
  32 
                long start = System.currentTimeMillis();
  33 
                String fileName = "Test.txt";
  34 
 
  35 
                try {
  36 
 
  37 
                        method.addRequestHeader("SOAPAction",
  38 
                                        "\"http://www.stellent.com/GetFile/\"");
  39 
 
  40 
                        MimeMessage message = new MimeMessage(
  41 
                                        Session.getDefaultInstance(new Properties()));
  42 
                        MimeMultipart mp = new MimeMultipart();
  43 
 
  44 
                        // Create the first body part
  45 
                        StringBuffer requestXML = constructRequest(fileName);
  46 
                        long requestSize = requestXML.length();
  47 
                        MimeBodyPart rootPart = new PreencodedMimeBodyPart("8bit");
  48 
                        rootPart.setContentID("rootContent");
  49 
                        rootPart.setDataHandler(new DataHandler(new RequestXmlDataSource(
  50 
                                        requestXML.toString())));
  51 
 
  52 
                        mp.addBodyPart(rootPart, 0);
  53 
 
  54 
                        message.setContent(mp);
  55 
                        message.saveChanges();
  56 
 
  57 
                        String cType = message.getHeader("Content-Type")[0];
  58 
                        int idx = cType.indexOf("boundary");
  59 
                        int boundryLength = 0;
  60 
                        if (idx != -1) {
  61 
                                boundryLength = cType.substring(idx + 9).length();
  62 
                        }
  63 
 
  64 
                        long contentLength = (boundryLength * 2) + 2 + requestSize + 6;
  65 
 
  66 
 
  67 
                        method.setDoAuthentication(true);
  68 
                        method.setRequestBody(requestXML.toString());
  69 
 
  70 
                        method.addRequestHeader("Content-Length",
  71 
                                        String.valueOf(requestXML.length()));
  72 
                        method.addRequestHeader("Content-Type", "text/xml;charset=UTF-8");
  73 
                        method.getParams().setParameter("http.socket.timeout",
  74 
                                        new Integer(0));
  75 
 
  76 
                        // send the request
  77 
                        int status = client.executeMethod(method);               
  78 
                       
  79 
                        // process the response
  80 
                        contentLength = getContentLength(method);
  81 
                       
  82 
                        String boundary = getBoundary(method);
  83 
                        if (boundary != null) {
  84 
                                // Boundary foe each Body Part
  85 
                                boundary = "--" + boundary;
  86 
                        }
  87 
 
  88 
                        BufferedInputStream bin = new BufferedInputStream(
  89 
                                        method.getResponseBodyAsStream());
  90 
                        LineInputStream lin = new LineInputStream(bin);
  91 
 
  92 
                        boolean rootBoundaryEncountered = false;
  93 
                        boolean attachmentBoundaryEncountered = false;
  94 
 
  95 
                        String line = null;
  96 
                        long readSoFar = 0;
  97 
                        while ((line = lin.readLine()) != null) {
  98 
                                readSoFar += line.length() +2; // 2 for \r\n
  99 
                                if (line.equals(boundary)) {
 100 
                                        if (!rootBoundaryEncountered) {
 101 
                                                // boundary for root body part
 102 
                                                rootBoundaryEncountered = true;
 103 
                                                continue;
 104 
                                        } else { // root bodypart already read so we are ready to
 105 
                                                                // read next body part
 106 
                                                attachmentBoundaryEncountered = true;
 107 
                                        }
 108 
                                } else if (line.length() == boundary.length() + 2
 109 
                                                && line.startsWith(boundary) && line.endsWith("--")) {
 110 
                                                // End of response
 111 
                                        line = null;
 112 
                                        break;
 113 
                                } else if (!StringUtils.isEmpty(line)) {
 114 
                                        System.out.println(line);
 115 
                                        // header part and SOAP XML goes here
 116 
                                } else { // the attachment content always follows the blank line
 117 
                                        if (attachmentBoundaryEncountered) {
 118 
                                                // save the attachment content into file
 119 
                                                saveAttachment(bin, boundary, contentLength, readSoFar);
 120 
                                                break;
 121 
                                        }
 122 
                                }
 123 
                        }
 124 
 
 125 
                } catch (HttpException e) {
 126 
                        e.printStackTrace();
 127 
                } catch (IOException e) {
 128 
                        e.printStackTrace();
 129 
                } catch (Exception e) {
 130 
                        e.printStackTrace();
 131 
                } finally {
 132 
                        long end = System.currentTimeMillis();
 133 
                        System.out.println("Time taken for [" + fileName + "]: "
 134 
                                        + (end - start));
 135 
                        // release any connection resources used by the method
 136 
                        method.releaseConnection();
 137 
                }
 138 
        }
 139 
 
 140 
        public static StringBuffer constructRequest(String fileName) {
 141 
                StringBuffer request = new StringBuffer();
 142 
                request.append("<soapenv:Envelope
 143 
                        xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"
 144 
                        xmlns:get=\"http://www.stellent.com/GetFile/\">\r\n"
 145 
                                + "   <soapenv:Header/>\r\n"
 146 
                                + "   <soapenv:Body>\r\n"
 147 
                                + "      <get:GetFileByName>\r\n"
 148 
                                + "         <get:dDocName>"+ fileName +"</get:dDocName>\r\n"
 149 
                                + "         <get:revisionSelectionMethod>
 150 
                                LatestReleased</get:revisionSelectionMethod>\r\n"
 151 
                                + "         <get:rendition>primary</get:rendition>\r\n"
 152 
                                + "         <get:extraProps>\r\n"
 153 
                                + "            <get:property>\r\n"
 154 
                                + "               <get:name>soapResponseType</get:name>\r\n"
 155 
                                + "               <get:value>Multipart/Related</get:value>\r\n"
 156 
                                + "            </get:property>\r\n"
 157 
                                + "         </get:extraProps>\r\n"
 158 
                                + "      </get:GetFileByName>\r\n"
 159 
                                + "   </soapenv:Body>\r\n"
 160 
                                + "</soapenv:Envelope>");
 161 
 
 162 
                return request;
 163 
        }
 164 
 
 165 
        public static String getBoundary(HttpMethod httpMethod) {
 166 
                Header h = null;
 167 
                String boundary = null;
 168 
 
 169 
                h = httpMethod.getResponseHeader("Content-Type");
 170 
 
 171 
                if (h != null) {
 172 
                        HeaderElement[] elements = h.getElements();
 173 
 
 174 
                        for (HeaderElement element : elements) {
 175 
                                String name = element.getName().toUpperCase();
 176 
                                if (name.startsWith("MULTIPART/")) {
 177 
                                        NameValuePair parameter = element
 178 
                                                        .getParameterByName("boundary");
 179 
                                        if (parameter != null) {
 180 
                                                boundary = parameter.getValue();
 181 
                                        }
 182 
                                }
 183 
                        }
 184 
                }
 185 
 
 186 
                return boundary;
 187 
        }
 188 
       
 189 
        public static long getContentLength(HttpMethod httpMethod) {
 190 
                Header h = null;
 191 
                String contentLen = "0";
 192 
 
 193 
                h = httpMethod.getResponseHeader("Content-Length");
 194 
                if (h != null) {
 195 
                        contentLen = h.getValue();
 196 
                }
 197 
               
 198 
                long contentLength = Long.valueOf(contentLen).longValue();
 199 
               
 200 
                return contentLength;
 201 
        }
 202 
 
 203 
        public static void saveAttachment(BufferedInputStream responseStream,
 204 
                String boundary, long totalBytes, long totalBytesRead) {
 205 
                try {
 206 
                        if (responseStream != null) {
 207 
                                BufferedOutputStream out = new BufferedOutputStream(
 208 
                                                new FileOutputStream(
 209 
                                                                "c:\\temp\\test.txt"));
 210 
                                byte data[] = new byte[4096];
 211 
                                int bytesread = -1;
 212 
                                boolean boundaryEndHit = false;
 213 
                                int boundaryEnd = boundary.length()+6;
 214 
                               
 215 
                                System.out.println("Attachment Starts Here ========= ");
 216 
                               
 217 
                                while ((bytesread = responseStream.read(data)) != -1) {
 218 
                                        totalBytesRead += bytesread;
 219 
                                       
 220 
                                        if (totalBytesRead <= (totalBytes -
 221 
                                                boundaryEnd)) {
 222 
                                                out.write(data, 0, bytesread);
 223 
                                        } else if (totalBytesRead > (totalBytes -
 224 
                                                boundaryEnd)) {
 225 
                                                out.write(data, 0, bytesread -
 226 
                                                        (int) (totalBytesRead -
 227 
                                                                (totalBytes - boundaryEnd)));
 228 
                                        }
 229 
                                }
 230 
                               
 231 
                                responseStream.close();
 232 
                                out.close();
 233 
                        }
 234 
                } catch (Exception e) {
 235 
                        e.printStackTrace();
 236 
                }
 237 
        }
 238 
}

The example was tested with 10 concurrent thread each downloaded 160MB of file from web service and the JVM memory utilization was mere 20MB.