JDebugger: March 2015

SOAP Client for Downloading Attachments - Streaming

We have seen uploading of larger data files using SOAP with attachments in the previous post(Upload attachments using SOAP). We will see how to download/stream larger files from SOAP web service.

Java Mail API Problem
The SOAP web service client for downloading attachments we will be discussing uses the MimeMessage structure but we will avoid using the JAVA MAIL package as it is not efficient while downloading the contents. If you use MimeMessage java api to get the MimeBodyPart then you'll be in trouble as MimeMessage loads the entire content as byte array into memory for calculating the size and number of MimeBodyParts. So we will avoid using it for larger files if you have small JVM memory.

How to stream larger files
The memory intensive operation can be avoided using our own custom code to parse the response from the SOAP web service.

The following example shows the SOAP message response with the attachments. This example is for Oracle UCM's GetFileByName operation.

The SOAP response message package is constructed using the Multipart/Related media type. The structure of the response stream is as follows:

The primary SOAP message must be carried in the root body part of the Multipart/Related structure. Consequently the type parameter of the Multipart/Related media header will always equal the Content-Type header for the primary SOAP 1.1 message, i.e., text/xml.
Referenced MIME parts must contain either a Content-ID MIME header.

Content-Type: multipart/related; type="text/xml"; start="rootContent"; boundary="----=_Part_12_21424854.1424891476173"

SOAPAction: "http://www.stellent.com/CheckIn/"

Content-Length: 1818

------=_Part_12_21424854.1424891476173

Content-Type: text/xml; charset=UTF-8

Content-Transfer-Encoding: 8bit

Content-ID: rootContent

<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:get=\"http://www.stellent.com/GetFile/\">

<soapenv:Header/>

<soapenv:Body>

<get:GetFileByName>

<get:dDocName>Test.txt</get:dDocName> <get:revisionSelectionMethod>LatestReleased</get:revisionSelectionMethod>

<get:rendition>primary</get:rendition>

<get:extraProps>

<get:property>

<get:name>soapResponseType</get:name>

<get:value>Multipart/Related</get:value>

</get:property>

</get:extraProps>

</get:GetFileByName>

</soapenv:Body>

</soapenv:Envelope>

------=_Part_12_21424854.1424891476173

Content-Type: text/plain

Content-Transfer-Encoding: 7bit

Content-ID: test.log

Hello

------=_Part_12_21424854.1424891476173--

As you can see above the response stream starts with boundary (------=_Part_12_21424854.1424891476173). Each MimeBodyParts are separated by boundary. The content of each MimeBodyPart starts after blank line. The blank line is used to seperate the headers and content of each MimeBodyPart. Using apache LineInputStream we read each line until we encounter the blank line for the attachment content. Once the blank line preceeding the attachment body is read we will switch back to BufferInputStream for efficiency. Don't forget to ignore the boundary line which indicates the end of response and exclude it from writing into the file.

1	import java.io.BufferedInputStream;
2	import java.io.BufferedOutputStream;
3	import java.io.FileOutputStream;
4	import java.io.IOException;
5	import java.io.InputStream;
6	import java.util.Properties;
7
8	import javax.activation.DataHandler;
9	import javax.mail.Session;
10	import javax.mail.internet.MimeBodyPart;
11	import javax.mail.internet.MimeMessage;
12	import javax.mail.internet.MimeMultipart;
13	import javax.mail.internet.PreencodedMimeBodyPart;
14
15	import org.apache.axis.utils.StringUtils;
16	import org.apache.commons.httpclient.Header;
17	import org.apache.commons.httpclient.HeaderElement;
18	import org.apache.commons.httpclient.HttpClient;
19	import org.apache.commons.httpclient.HttpException;
20	import org.apache.commons.httpclient.HttpMethod;
21	import org.apache.commons.httpclient.NameValuePair;
22	import org.apache.commons.httpclient.methods.PostMethod;
23
24	import com.sun.mail.util.LineInputStream;
25
26	public class UCMDownloader {
27
28	public static void main(String[] args) {
29	HttpClient client = new HttpClient();
30	PostMethod method = new PostMethod(
31	"http://www.oracle.com:16200/_dav/cs/idcplg");
32	long start = System.currentTimeMillis();
33	String fileName = "Test.txt";
34
35	try {
36
37	method.addRequestHeader("SOAPAction",
38	"\"http://www.stellent.com/GetFile/\"");
39
40	MimeMessage message = new MimeMessage(
41	Session.getDefaultInstance(new Properties()));
42	MimeMultipart mp = new MimeMultipart();
43
44	// Create the first body part
45	StringBuffer requestXML = constructRequest(fileName);
46	long requestSize = requestXML.length();
47	MimeBodyPart rootPart = new PreencodedMimeBodyPart("8bit");
48	rootPart.setContentID("rootContent");
49	rootPart.setDataHandler(new DataHandler(new RequestXmlDataSource(
50	requestXML.toString())));
51
52	mp.addBodyPart(rootPart, 0);
53
54	message.setContent(mp);
55	message.saveChanges();
56
57	String cType = message.getHeader("Content-Type")[0];
58	int idx = cType.indexOf("boundary");
59	int boundryLength = 0;
60	if (idx != -1) {
61	boundryLength = cType.substring(idx + 9).length();
62	}
63
64	long contentLength = (boundryLength * 2) + 2 + requestSize + 6;
65
66
67	method.setDoAuthentication(true);
68	method.setRequestBody(requestXML.toString());
69
70	method.addRequestHeader("Content-Length",
71	String.valueOf(requestXML.length()));
72	method.addRequestHeader("Content-Type", "text/xml;charset=UTF-8");
73	method.getParams().setParameter("http.socket.timeout",
74	new Integer(0));
75
76	// send the request
77	int status = client.executeMethod(method);
78
79	// process the response
80	contentLength = getContentLength(method);
81
82	String boundary = getBoundary(method);
83	if (boundary != null) {
84	// Boundary foe each Body Part
85	boundary = "--" + boundary;
86	}
87
88	BufferedInputStream bin = new BufferedInputStream(
89	method.getResponseBodyAsStream());
90	LineInputStream lin = new LineInputStream(bin);
91
92	boolean rootBoundaryEncountered = false;
93	boolean attachmentBoundaryEncountered = false;
94
95	String line = null;
96	long readSoFar = 0;
97	while ((line = lin.readLine()) != null) {
98	readSoFar += line.length() +2; // 2 for \r\n
99	if (line.equals(boundary)) {
100	if (!rootBoundaryEncountered) {
101	// boundary for root body part
102	rootBoundaryEncountered = true;
103	continue;
104	} else { // root bodypart already read so we are ready to
105	// read next body part
106	attachmentBoundaryEncountered = true;
107	}
108	} else if (line.length() == boundary.length() + 2
109	&& line.startsWith(boundary) && line.endsWith("--")) {
110	// End of response
111	line = null;
112	break;
113	} else if (!StringUtils.isEmpty(line)) {
114	System.out.println(line);
115	// header part and SOAP XML goes here
116	} else { // the attachment content always follows the blank line
117	if (attachmentBoundaryEncountered) {
118	// save the attachment content into file
119	saveAttachment(bin, boundary, contentLength, readSoFar);
120	break;
121	}
122	}
123	}
124
125	} catch (HttpException e) {
126	e.printStackTrace();
127	} catch (IOException e) {
128	e.printStackTrace();
129	} catch (Exception e) {
130	e.printStackTrace();
131	} finally {
132	long end = System.currentTimeMillis();
133	System.out.println("Time taken for [" + fileName + "]: "
134	+ (end - start));
135	// release any connection resources used by the method
136	method.releaseConnection();
137	}
138	}
139
140	public static StringBuffer constructRequest(String fileName) {
141	StringBuffer request = new StringBuffer();
142	request.append("<soapenv:Envelope
143	xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"
144	xmlns:get=\"http://www.stellent.com/GetFile/\">\r\n"
145	+ " <soapenv:Header/>\r\n"
146	+ " <soapenv:Body>\r\n"
147	+ " <get:GetFileByName>\r\n"
148	+ " <get:dDocName>"+ fileName +"</get:dDocName>\r\n"
149	+ " <get:revisionSelectionMethod>
150	LatestReleased</get:revisionSelectionMethod>\r\n"
151	+ " <get:rendition>primary</get:rendition>\r\n"
152	+ " <get:extraProps>\r\n"
153	+ " <get:property>\r\n"
154	+ " <get:name>soapResponseType</get:name>\r\n"
155	+ " <get:value>Multipart/Related</get:value>\r\n"
156	+ " </get:property>\r\n"
157	+ " </get:extraProps>\r\n"
158	+ " </get:GetFileByName>\r\n"
159	+ " </soapenv:Body>\r\n"
160	+ "</soapenv:Envelope>");
161
162	return request;
163	}
164
165	public static String getBoundary(HttpMethod httpMethod) {
166	Header h = null;
167	String boundary = null;
168
169	h = httpMethod.getResponseHeader("Content-Type");
170
171	if (h != null) {
172	HeaderElement[] elements = h.getElements();
173
174	for (HeaderElement element : elements) {
175	String name = element.getName().toUpperCase();
176	if (name.startsWith("MULTIPART/")) {
177	NameValuePair parameter = element
178	.getParameterByName("boundary");
179	if (parameter != null) {
180	boundary = parameter.getValue();
181	}
182	}
183	}
184	}
185
186	return boundary;
187	}
188
189	public static long getContentLength(HttpMethod httpMethod) {
190	Header h = null;
191	String contentLen = "0";
192
193	h = httpMethod.getResponseHeader("Content-Length");
194	if (h != null) {
195	contentLen = h.getValue();
196	}
197
198	long contentLength = Long.valueOf(contentLen).longValue();
199
200	return contentLength;
201	}
202
203	public static void saveAttachment(BufferedInputStream responseStream,
204	String boundary, long totalBytes, long totalBytesRead) {
205	try {
206	if (responseStream != null) {
207	BufferedOutputStream out = new BufferedOutputStream(
208	new FileOutputStream(
209	"c:\\temp\\test.txt"));
210	byte data[] = new byte[4096];
211	int bytesread = -1;
212	boolean boundaryEndHit = false;
213	int boundaryEnd = boundary.length()+6;
214
215	System.out.println("Attachment Starts Here ========= ");
216
217	while ((bytesread = responseStream.read(data)) != -1) {
218	totalBytesRead += bytesread;
219
220	if (totalBytesRead <= (totalBytes -
221	boundaryEnd)) {
222	out.write(data, 0, bytesread);
223	} else if (totalBytesRead > (totalBytes -
224	boundaryEnd)) {
225	out.write(data, 0, bytesread -
226	(int) (totalBytesRead -
227	(totalBytes - boundaryEnd)));
228	}
229	}
230
231	responseStream.close();
232	out.close();
233	}
234	} catch (Exception e) {
235	e.printStackTrace();
236	}
237	}
238	}

The example was tested with 10 concurrent thread each downloaded 160MB of file from web service and the JVM memory utilization was mere 20MB.

Tuesday, March 10, 2015

Download attachments from SOAP web service- streaming of contents

SOAP Client for Downloading Attachments - Streaming