Tuesday, March 10, 2015

Download attachments from SOAP web service- streaming of contents

SOAP Client for Downloading Attachments - Streaming

We have seen uploading of larger data files using SOAP with attachments in the previous post(Upload attachments using SOAP). We will see how to download/stream larger files from SOAP web service.


Java Mail API Problem
The SOAP web service client for downloading attachments we will be discussing uses the MimeMessage structure but we will avoid using the JAVA MAIL package as it is not efficient while downloading the contents. If you use MimeMessage java api to get the MimeBodyPart then you'll be in trouble as MimeMessage loads the entire content as byte array into memory for calculating the size and number of MimeBodyParts. So we will avoid using it for larger files if you have small JVM memory.

How to stream larger files
The memory intensive operation can be avoided using our own custom code to parse the response from the SOAP web service. 

The following example shows the SOAP message response with the attachments. This example is for Oracle UCM's GetFileByName operation. 

The SOAP response message package is constructed using the  Multipart/Related  media type. The structure of the response stream is as follows:
  1. The primary SOAP message must be carried in the root body part of the Multipart/Related structure. Consequently the type parameter of the Multipart/Related media header will always equal the Content-Type header for the primary SOAP 1.1 message, i.e., text/xml.
  2. Referenced MIME parts must contain either a Content-ID MIME header.
Content-Type: multipart/related; type="text/xml"; start="rootContent"; boundary="----=_Part_12_21424854.1424891476173"
SOAPAction: "http://www.stellent.com/CheckIn/"
Content-Length: 1818


------=_Part_12_21424854.1424891476173
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-ID: rootContent

<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:get=\"http://www.stellent.com/GetFile/\">
   <soapenv:Header/>
   <soapenv:Body>
      <get:GetFileByName>
         <get:dDocName>Test.txt</get:dDocName>    <get:revisionSelectionMethod>LatestReleased</get:revisionSelectionMethod>
         <get:rendition>primary</get:rendition>
         <get:extraProps>
            <get:property>
               <get:name>soapResponseType</get:name>
               <get:value>Multipart/Related</get:value>
            </get:property>
         </get:extraProps>
      </get:GetFileByName>
   </soapenv:Body>
</soapenv:Envelope>
------=_Part_12_21424854.1424891476173
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Content-ID: test.log

Hello

------=_Part_12_21424854.1424891476173--

As you can see above the response stream starts with boundary (------=_Part_12_21424854.1424891476173). Each MimeBodyParts are separated by boundary. The content of each MimeBodyPart starts after blank line. The blank line is used to seperate the headers and content of each MimeBodyPart. Using apache LineInputStream we read each line until we encounter the blank line for the attachment content. Once the blank line preceeding the attachment body is read we will switch back to BufferInputStream for efficiency. Don't forget to ignore the boundary line which indicates the end of response and exclude it from writing into the file.


   1 
import java.io.BufferedInputStream;
   2 
import java.io.BufferedOutputStream;
   3 
import java.io.FileOutputStream;
   4 
import java.io.IOException;
   5 
import java.io.InputStream;
   6 
import java.util.Properties;
   7 
 
   8 
import javax.activation.DataHandler;
   9 
import javax.mail.Session;
  10 
import javax.mail.internet.MimeBodyPart;
  11 
import javax.mail.internet.MimeMessage;
  12 
import javax.mail.internet.MimeMultipart;
  13 
import javax.mail.internet.PreencodedMimeBodyPart;
  14 
 
  15 
import org.apache.axis.utils.StringUtils;
  16 
import org.apache.commons.httpclient.Header;
  17 
import org.apache.commons.httpclient.HeaderElement;
  18 
import org.apache.commons.httpclient.HttpClient;
  19 
import org.apache.commons.httpclient.HttpException;
  20 
import org.apache.commons.httpclient.HttpMethod;
  21 
import org.apache.commons.httpclient.NameValuePair;
  22 
import org.apache.commons.httpclient.methods.PostMethod;
  23 
 
  24 
import com.sun.mail.util.LineInputStream;
  25 
 
  26 
public class UCMDownloader {
  27 
 
  28 
        public static void main(String[] args) {
  29 
                HttpClient client = new HttpClient();
  30 
                PostMethod method = new PostMethod(
  31 
                                "http://www.oracle.com:16200/_dav/cs/idcplg");
  32 
                long start = System.currentTimeMillis();
  33 
                String fileName = "Test.txt";
  34 
 
  35 
                try {
  36 
 
  37 
                        method.addRequestHeader("SOAPAction",
  38 
                                        "\"http://www.stellent.com/GetFile/\"");
  39 
 
  40 
                        MimeMessage message = new MimeMessage(
  41 
                                        Session.getDefaultInstance(new Properties()));
  42 
                        MimeMultipart mp = new MimeMultipart();
  43 
 
  44 
                        // Create the first body part
  45 
                        StringBuffer requestXML = constructRequest(fileName);
  46 
                        long requestSize = requestXML.length();
  47 
                        MimeBodyPart rootPart = new PreencodedMimeBodyPart("8bit");
  48 
                        rootPart.setContentID("rootContent");
  49 
                        rootPart.setDataHandler(new DataHandler(new RequestXmlDataSource(
  50 
                                        requestXML.toString())));
  51 
 
  52 
                        mp.addBodyPart(rootPart, 0);
  53 
 
  54 
                        message.setContent(mp);
  55 
                        message.saveChanges();
  56 
 
  57 
                        String cType = message.getHeader("Content-Type")[0];
  58 
                        int idx = cType.indexOf("boundary");
  59 
                        int boundryLength = 0;
  60 
                        if (idx != -1) {
  61 
                                boundryLength = cType.substring(idx + 9).length();
  62 
                        }
  63 
 
  64 
                        long contentLength = (boundryLength * 2) + 2 + requestSize + 6;
  65 
 
  66 
 
  67 
                        method.setDoAuthentication(true);
  68 
                        method.setRequestBody(requestXML.toString());
  69 
 
  70 
                        method.addRequestHeader("Content-Length",
  71 
                                        String.valueOf(requestXML.length()));
  72 
                        method.addRequestHeader("Content-Type", "text/xml;charset=UTF-8");
  73 
                        method.getParams().setParameter("http.socket.timeout",
  74 
                                        new Integer(0));
  75 
 
  76 
                        // send the request
  77 
                        int status = client.executeMethod(method);               
  78 
                       
  79 
                        // process the response
  80 
                        contentLength = getContentLength(method);
  81 
                       
  82 
                        String boundary = getBoundary(method);
  83 
                        if (boundary != null) {
  84 
                                // Boundary foe each Body Part
  85 
                                boundary = "--" + boundary;
  86 
                        }
  87 
 
  88 
                        BufferedInputStream bin = new BufferedInputStream(
  89 
                                        method.getResponseBodyAsStream());
  90 
                        LineInputStream lin = new LineInputStream(bin);
  91 
 
  92 
                        boolean rootBoundaryEncountered = false;
  93 
                        boolean attachmentBoundaryEncountered = false;
  94 
 
  95 
                        String line = null;
  96 
                        long readSoFar = 0;
  97 
                        while ((line = lin.readLine()) != null) {
  98 
                                readSoFar += line.length() +2; // 2 for \r\n
  99 
                                if (line.equals(boundary)) {
 100 
                                        if (!rootBoundaryEncountered) {
 101 
                                                // boundary for root body part
 102 
                                                rootBoundaryEncountered = true;
 103 
                                                continue;
 104 
                                        } else { // root bodypart already read so we are ready to
 105 
                                                                // read next body part
 106 
                                                attachmentBoundaryEncountered = true;
 107 
                                        }
 108 
                                } else if (line.length() == boundary.length() + 2
 109 
                                                && line.startsWith(boundary) && line.endsWith("--")) {
 110 
                                                // End of response
 111 
                                        line = null;
 112 
                                        break;
 113 
                                } else if (!StringUtils.isEmpty(line)) {
 114 
                                        System.out.println(line);
 115 
                                        // header part and SOAP XML goes here
 116 
                                } else { // the attachment content always follows the blank line
 117 
                                        if (attachmentBoundaryEncountered) {
 118 
                                                // save the attachment content into file
 119 
                                                saveAttachment(bin, boundary, contentLength, readSoFar);
 120 
                                                break;
 121 
                                        }
 122 
                                }
 123 
                        }
 124 
 
 125 
                } catch (HttpException e) {
 126 
                        e.printStackTrace();
 127 
                } catch (IOException e) {
 128 
                        e.printStackTrace();
 129 
                } catch (Exception e) {
 130 
                        e.printStackTrace();
 131 
                } finally {
 132 
                        long end = System.currentTimeMillis();
 133 
                        System.out.println("Time taken for [" + fileName + "]: "
 134 
                                        + (end - start));
 135 
                        // release any connection resources used by the method
 136 
                        method.releaseConnection();
 137 
                }
 138 
        }
 139 
 
 140 
        public static StringBuffer constructRequest(String fileName) {
 141 
                StringBuffer request = new StringBuffer();
 142 
                request.append("<soapenv:Envelope
 143 
                        xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"
 144 
                        xmlns:get=\"http://www.stellent.com/GetFile/\">\r\n"
 145 
                                + "   <soapenv:Header/>\r\n"
 146 
                                + "   <soapenv:Body>\r\n"
 147 
                                + "      <get:GetFileByName>\r\n"
 148 
                                + "         <get:dDocName>"+ fileName +"</get:dDocName>\r\n"
 149 
                                + "         <get:revisionSelectionMethod>
 150 
                                LatestReleased</get:revisionSelectionMethod>\r\n"
 151 
                                + "         <get:rendition>primary</get:rendition>\r\n"
 152 
                                + "         <get:extraProps>\r\n"
 153 
                                + "            <get:property>\r\n"
 154 
                                + "               <get:name>soapResponseType</get:name>\r\n"
 155 
                                + "               <get:value>Multipart/Related</get:value>\r\n"
 156 
                                + "            </get:property>\r\n"
 157 
                                + "         </get:extraProps>\r\n"
 158 
                                + "      </get:GetFileByName>\r\n"
 159 
                                + "   </soapenv:Body>\r\n"
 160 
                                + "</soapenv:Envelope>");
 161 
 
 162 
                return request;
 163 
        }
 164 
 
 165 
        public static String getBoundary(HttpMethod httpMethod) {
 166 
                Header h = null;
 167 
                String boundary = null;
 168 
 
 169 
                h = httpMethod.getResponseHeader("Content-Type");
 170 
 
 171 
                if (h != null) {
 172 
                        HeaderElement[] elements = h.getElements();
 173 
 
 174 
                        for (HeaderElement element : elements) {
 175 
                                String name = element.getName().toUpperCase();
 176 
                                if (name.startsWith("MULTIPART/")) {
 177 
                                        NameValuePair parameter = element
 178 
                                                        .getParameterByName("boundary");
 179 
                                        if (parameter != null) {
 180 
                                                boundary = parameter.getValue();
 181 
                                        }
 182 
                                }
 183 
                        }
 184 
                }
 185 
 
 186 
                return boundary;
 187 
        }
 188 
       
 189 
        public static long getContentLength(HttpMethod httpMethod) {
 190 
                Header h = null;
 191 
                String contentLen = "0";
 192 
 
 193 
                h = httpMethod.getResponseHeader("Content-Length");
 194 
                if (h != null) {
 195 
                        contentLen = h.getValue();
 196 
                }
 197 
               
 198 
                long contentLength = Long.valueOf(contentLen).longValue();
 199 
               
 200 
                return contentLength;
 201 
        }
 202 
 
 203 
        public static void saveAttachment(BufferedInputStream responseStream,
 204 
                String boundary, long totalBytes, long totalBytesRead) {
 205 
                try {
 206 
                        if (responseStream != null) {
 207 
                                BufferedOutputStream out = new BufferedOutputStream(
 208 
                                                new FileOutputStream(
 209 
                                                                "c:\\temp\\test.txt"));
 210 
                                byte data[] = new byte[4096];
 211 
                                int bytesread = -1;
 212 
                                boolean boundaryEndHit = false;
 213 
                                int boundaryEnd = boundary.length()+6;
 214 
                               
 215 
                                System.out.println("Attachment Starts Here ========= ");
 216 
                               
 217 
                                while ((bytesread = responseStream.read(data)) != -1) {
 218 
                                        totalBytesRead += bytesread;
 219 
                                       
 220 
                                        if (totalBytesRead <= (totalBytes -
 221 
                                                boundaryEnd)) {
 222 
                                                out.write(data, 0, bytesread);
 223 
                                        } else if (totalBytesRead > (totalBytes -
 224 
                                                boundaryEnd)) {
 225 
                                                out.write(data, 0, bytesread -
 226 
                                                        (int) (totalBytesRead -
 227 
                                                                (totalBytes - boundaryEnd)));
 228 
                                        }
 229 
                                }
 230 
                               
 231 
                                responseStream.close();
 232 
                                out.close();
 233 
                        }
 234 
                } catch (Exception e) {
 235 
                        e.printStackTrace();
 236 
                }
 237 
        }
 238 
}

The example was tested with 10 concurrent thread each downloaded 160MB of file from web service and the JVM memory utilization was mere 20MB.


19 comments:

  1. Interesting Article

    Java Web Services Training Online Java Web Services Training Online Online Java Training Java Online Training Java EE Online Training Java EE Online Training

    | Java Web Services Training in Chennai Java Web Services Training in Chennai Java Online Training from India

    ReplyDelete
  2. The prevalent availability of high speed internet access has accelerated the adoption of streaming media services, further reducing the lag time in content delivery and thus improving the user's overall experience. moviebox download

    ReplyDelete
  3. When your website or blog goes live for the first time, it is exciting. That is until you realize no one but you and your. Touchless soap dispenser

    ReplyDelete
  4. PTV is defined as Internet Protocol television which is a system, where services for digital television are delivered through broadband IP link with the use of data communications. There are established organizations offering IPTV consulting to assist vendors and operators to safely enter the complex world of smart iptv

    ReplyDelete
  5. Which assault organizations to legitimize their own reality. Strangely enough these administration organizations catch fire half of all monies gathered in organization.soap box printing

    ReplyDelete
  6. I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. Voir film

    ReplyDelete
  7. This is my first time i visit here and I found so many interesting stuff in your blog especially it's discussion, thank you. Protocol

    ReplyDelete
  8. i'm grateful to you for sharing this plethora of beneficial trace. i found this aid utmost beneficial for me. thanks loads for difficult do some thing. https://totalsportek.news/f1-streaming

    ReplyDelete
  9. Im no expert, but I believe you just made an excellent point. You certainly fully understand what youre speaking about, and I can truly get behind that. ICAR Webmail

    ReplyDelete
  10. Wow, excellent post. I'd like to draft like this too - taking time and real hard work to make a great article. There are wonderful hashtag on tiktok, please go to site tiktok downloader to know more

    ReplyDelete
  11. Wow, What a Excellent post. I really found this to much informatics. It is what i was searching for.I would like to suggest you that please keep sharing such type of info.Thanks The useless web

    ReplyDelete
  12. The rapid growth in this field of science has resulted in the development of universities that have introduced different graduate programs related to data science. In this article, data science course syllabus

    ReplyDelete

  13. Everything is going to be alright. Are we really being honest
    frasesparaenamorarz.com
    fras-es.com

    ReplyDelete
  14. At the point when the iPad was first delivered, just Apple apps offered help for an outside screen hookup yet soon after dispatch, the Netflix app added this usefulness enabling supporters of utilization the iPod Dock Connector to VGA Adapter to watch Netflix on a TV, screen, projector or LCD show that can utilize a VGA link. https://twitchviral.com/

    ReplyDelete
  15. ทีมงาน pokerN8 พร้อมดูแลคุณ 24 ชม Poker เงิน พร้อมเล่นเกมส์ เติมเครดิตฝากถอน พร้อมรวมกลุ่มนักเล่นโป๊กเกอร์ทั่วประเทศไทย มาไว้ที่นี่โป๊กเกอร์ออนไลน์ Poker is the #1 poker game in the world. Play with friends and see who's got the best poker face or come and meet some new buddies. Help out in a poker

    ReplyDelete