Tuesday, April 7, 2009

Handling Big XML Files on iPhone

Jim Dovey (aka @ alanQuatermain) has come up with what looks like a tremendous solution for handling very large XML files on the iPhone. Jim tested with a large (22 megabyte) XML file. Parsing the file with NSXMLDocument took about 123 megabytes of virtual memory to parse (I'm assuming this was run on the simulator since there's no way you could get that much heap allocated on the phone itself).

Running the same exact file through his own parser used only 70 kilobytes of virtual memory. That's a dramatic, several-order-of-magnitude difference. So, if large XML files are the bane of your existence, you might want to check his solution out.



9 comments:

alanQuatermain said...

Many thanks for the linkage :o)

It was all run on OS X; the test app is just a command-line application. I'm going to be doing some tests within Outpost soon— in the process of porting the codebase over to this one from NSXMLParser, which means changing all the HTTP stuff to return streams rather than data, etc. But I'm very excited to see the results there, to say the least…

nskboy said...

Alan, why just not to use libxml in this case? Is there any advantages between your own parser and libxml?

Ole Begemann said...

I haven't looked at Alan's code yet, but Apple's XMLPerformance sample app also addresses this and uses xmlParseChunk() for the solution. Might be worth a look.

alanQuatermain said...

@nskboy:
AQXMLParser is a version of NSXMLParser, and thus uses libxml internally to do the actual parsing. The big difference between this & NSXMLParser is that Apple's version takes the complete chunk of XML data and calls xmlParseChunk() exactly once, while mine pulls data in from a stream, calling xmlParseChunk() for each 1KB of data. Another advantage is that it's effectively compatible with NSXMLParser: you can use the same delegate class for both, since they use the same delegation methods.

@Ole
Interesting, I'd not seen that one. A Spotlight search here just turned up this web page, so I'll have a dig around & see what I can find.

Ole Begemann said...

Alan, here's a link to the sample at Apple's site: http://developer.apple.com/iphone/library/samplecode/XMLPerformance/index.html (iPhone Dev Account required). It shows how to improve XML parsing performance and UI responsiveness by directly using libxml and parsing chunks. It would be interesting to compare the performance of Apple's code to yours.

alanQuatermain said...

@Ole

I just looked at their code, and it seems to be doing much the same. The chief optimisation, as far as memory is concerned, is just to call xmlParseChunk() with multiple small chunks of data, which both versions will do. Mine is explicitly using 1K chunks while Apple's uses whatever is handed out by NSURLConnection.

In terms of general memory usage, however, it's been noted that NSURLConnection is a little memory-intensive in itself, which is why I've started using CFHTTPMessage and streams directly (hence the HTTPMessage folder in the toolkit there).

dontcare said...

You may also want to look at vtd-xml, the latest/greatest open source parsing/indexing/xpath technology

Edwin said...

scrub m65 kamagra attorney lawyer body scrub field jacket lovegra marijuana attorney injury lawyer

h4ns said...

What youre saying is completely true. I know that everybody must say the same thing, but I just think that you put it in a way that everyone can understand. I also love the images you put in here. They fit so well with what youre trying to say. Im sure youll reach so many people with what youve got to say.

Arsenal vs Huddersfield Town live streaming
Arsenal vs Huddersfield Town live streaming
Wolverhampton Wanderers vs Stoke City Live Streaming
Wolverhampton Wanderers vs Stoke City Live Streaming
Notts County vs Manchester City Live Streaming
Notts County vs Manchester City Live Streaming
Bologna vs AS Roma Live Streaming
Bologna vs AS Roma Live Streaming
Juventus vs Udinese Live Streaming
Juventus vs Udinese Live Streaming
Napoli vs Sampdoria Live Streaming
Napoli vs Sampdoria Live Streaming
Fulham vs Tottenham Hotspur Live Streaming
Fulham vs Tottenham Hotspur Live Streaming
AS Monaco vs Marseille Live Streaming
AS Monaco vs Marseille Live Streaming
Alajuelense vs Perez Zeledon Live Streaming
Alajuelense vs Perez Zeledon Live Streaming
Technology News | News Today | Live Streaming TV Channels