Monday, October 20, 2008

iPhone "Optimized" PNGs

For the most part, Apple's developer documentation for both the Mac and iPhone is excellent. They are well-written, accurate, and provide excellent code samples. There are, of course, always gaps, especially with newer areas. Sometimes these gaps are simply a matter of resource constraints; the volume of documentation the documentation teams have to maintain is staggering, and frameworks are constantly being added and updated.

At other times, Apple is tight-lipped about stuff not because of resource constraints, but rather intentionally, though it's not always clear why. Do you want to know how fast the processor in your iPhone is, or how much memory is available to your application? Both of these are obviously relevant to developing for the iPhone, but don't look to Apple for that information. The information is available (processor | memory), just not from Apple.

Here's another area where Apple is less than fully forthcoming: When it comes to the iPhone's use of PNG images, Apple tells you that Xcode "optimizes" PNGs as part of the build process and, as a result, you should always use PNG images in your iPhone projects. PNG images will usually be bigger than JPEGs, causing your application to be a little larger, but you'll be rewarded by some mysterious kind of improvement.

Since this information is not documented, what follows is something of an educated guess based on research and observations. I will gladly be corrected on any inaccuracies, so don't hesitate to ping me if you see something wrong.

So, just what is this magical, mystery "optimization"?



During the build, Xcode basically screws up your PNG image, turning it into something that's technically no longer a PNG image. Xcode doesn't change your original file, but when it copies it to the Resources folder of your application bundle, what it does is it byte-swaps the red and blue octets and also "premultiplies the alpha" (if that doesn't make sense, don't worry, we'll talk about it in a moment). The PNG specification does not support either of these changes, which is why the image is no longer technically a PNG and can't be loaded by most image editing software. Not that it really matters, since the new version only exists inside your application bundle, but it does mean that any PNG file that you include in your app bundle should not be, for example, FTPed or e-mailed from your application directly, because the file will be corrupt as far as most software is concerned.

Byte Swapping


Byte swapping is exactly what it sounds like. Uncompressed computer graphic images are most-often stored as sequences of three or four bytes or "octets" representing each pixel in the image. Each byte in the pixel represents one of the three additive primaries (red, blue, and green), plus there's often another byte called "alpha", which we'll look at in a moment. There are other ways to store images, but this is the most common technique. The most common byte-ordering used in this technique is RGB (or RGBA) which stands for Red-Green-Blue(-Alpha). This means that a single pixel is represented in memory by four bytes, the first representing the intensity of red, the second representing the intensity of green, and the third representing the intensity of blue (we'll ignore alpha for now). The PNG specification uses this byte-ordering, as do many other image formats.

However, the iPhone's video memory uses a non-standard byte-ordering of BGR or Blue-Green-Red (I don't believe there's an alpha component in the iPhone's vRAM). In order to copy from an image in memory using RGBA to the video memory using GRB is more computationally expensive than just copying from, for example, BGR to BGR, which can be done with a single, fast memory move (or "blit") operation. By doing this byte-swap in the Copy Files Build Phase, your application is (in some situations) able to ignore the alpha component when loading the file into memory so it can use the faster memory move operation to get the image into video memory (and hence onto the screen).

Premultiplied Alpha


Okay, that makes sense, but what about this "premultiplying the alpha" thing? That one sounds kind of mysterious, doesn't it? As mentioned in the previous section, the iPhone's vRAM has no alpha component, so if we're going to ignore that component, we still need to take it into account somehow. Remember that PNG Images that have been optimized for the iPhone are stored as four values, representing the Blue, Green, Red, and Alpha components (in that order). Although it's called a "component", Alpha isn't really a color component at all. It's more like a modifier that acts on the other three values, and it represents how much of whatever is beneath the color will show through. So, a color that looked like this:

B:1.0
G:0.0
R:0.0
A:0.5

Would represent the color blue, but at 50% opacity. In order for the computer to use the alpha component, it has to multiply the alpha times the other three components (and possibly by other values) before putting them into video memory. This multiplication process is more computationally expensive than just copying the value into video memory. So, Xcode also premultiplies the alpha by the three components and stores the multiplied value. As a result, the color above would look like this after the Copy Files Build Phase:

B:0.5
G:0.0
R:0.0
A:0.5

Notice that the Blue component in this new pixel is equal to the Blue component of the previous version multiplied by the alpha. The result of this premultiplication, is that the alpha component can be ignored when loading the image into memory and the image can then be blitted directly to video memory without having to do the expensive floating-point multiplications to calculate the alpha. Well, at least, sometimes. This pixel is what the pixel would look like when drawn over white.

So, what happens if there's a color beneath that needs to show through? This is where things can get a little confusing. If your application is drawing the image on top of something else, or if you're using the alpha channel in the image as a mask, then the iPhone can't use this optimization. It has to do the alpha calculations (which is why the alpha channel is left intact in the pre-multiplication process), the the byte-swapping still offers some improvement. For the most part, you don't have to worry about this - it all happens under the hood. The thing you should take away from it though, is that you can help your iPhone know when it is safe to use the optimization. The way to do that is to make sure you check the "opaque" checkbox in Interface Builder for your image views, or other views that contain images, or set the opaque property of your UIImageView to YES in code. This tells the iPhone that it's safe to do the faster blit operation and ignore the alpha channel because nothing below shows through an opaque object and there's no need to do any expensive floating point alpha calculations for that object. Of course, if you are using the alpha channel in your image, or if you are doing complex image compositions in your app at run-time, you shouldn't make the view opaque. But, if you're just displaying an image in an UIImageView, checking that opaque checkbox is a really, really good idea.

An awful lot of the time, PNG images have 1.0 for the Alpha value in every pixel. As a result, more often than not, program runs faster and use less memory because of the PNG "optimizations" done by Xcode.

What happens when you use different file types


When you use any other file type (or if you load a non-optimized PNG files), your iPhone has to do the byte-swapping and alpha premultiplication at load-time (and possibly re-do the alpha multiplication at display time). Your application basically has to do the same processing that Xcode does, but it's doing it at run-time instead of at build-time. This is going to cost you both in terms of processor cycles and memory overhead. One of the reasons why Mobile Safari is the biggest memory hog of the built-in iPhone applications is because the images it has to load in order to display web-pages are all non-optimized images, mostly JPEGs. Since JPEG is a compressed format, it has the added extra step of having to decompress the image into memory before it can do the premultiplication and byte-swapping.



18 comments:

David said...

Do you know if anyone has written Mac-based Cocoa code so that these optimized PNG's are loadable/viewable in a Mac application?

I'm writing a level design tool for my client, and for workflow reasons, it would be much easier to load the iPhone.app bundle, and load data from within that app bundle.

stuart said...

@David: it appears that when compiling your app for the Simulator, the images are not iPhone "optimized" and so are viewable in standard image editors.

Christopher Atlan said...

iphone .png decoding

Dan Bourque said...

Thanks for the great article. I wonder if you might be able to help me with a problem.

My iPhone application downloads its PNGs from an external source at runtime (from Picasa Web albums, using the Google Data APIs). Unfortunately, these images' performance is quite bad, because they haven't been optimized. When I do custom rendering on top of the image, it seems 100x slower than if I had added it as a resource.

Do you know how I can optimize an externally downloaded PNG at runtime on the iPhone? I'm hoping for a class that does this. I even considered adding pngcrush's source code to my app.

Frédéric said...

I would partially disagree with what you're saying about JPEG. When you uncompress a JPEG file, the output is not in an RGB format but rather in an YCbCr format. It means that you have to convert it to RGB or BGR which has the same cost. Apple took libjpeg and modified the conversion function so that the output is BGR (you can see it if you run the profiler).

Note that JPEG does not have transparency and you would have to create an alpha channel afterward, which could be costly.

Clint said...

Incredibly helpful--thanks for taking the time to write up such a thorough and easy-to-understand explanation! Appreciating a post like this = taking time to check out the ads on the page. ;)

Druva said...

Thanks for sharing the information, i have a question now. i have developed an gallery application which contains around 100 png file,

after launching the application, thumbs will be displayed at the bottom and when clicking on thumb respective image will displayed on the top.
It goes well for few (50)clicks after that it crashes. After strugling for few days i found it as memory problem.

Can you please tell how to handle image gallery applications

rave3000 said...

Sorry, but IO'm not very technical - I'm a designer. I was wondering if you have encounter this problem with png's ... I'm designing a game, and we are using cocos 2d - and my images all look georgous before they enter cocos 2s. The iPhone shows colors and images a bit washed out in the application. Is it because the png's are in the wrong bit 8, 16, 24 or something else I'm missing.

marc said...
This comment has been removed by the author.
marc said...

We (atPurpose Technologies) have created 2 QuickLook plugins that can decode crushed PNG images and PVR textures. They are part of atPeek tool we have developed, but can also be downloaded separately at: http://atpeek.download.atpurpose.com/PngUncrush.zip and http://atpeek.download.atpurpose.com/PvrUncrush.zip

atPeek tool, which uses them, is available at: http://www.atpurpose.com/atPeek

Sergei said...

"This pixel is what the pixel would look like when drawn over white.”

No, this is wrong. The pixel is what the pixel would look like when drawn over BLACK. And premultiplied alpha is not just for black backgrounds, it’s more interesting than that. See, the equation for, say, red component is this (assuming ‘a' changes from 0 to 1):

r_result = r_back * (1 - a_front) + r_front * a_front

Since r_front and a_front are known beforehand, we can multiply them beforehand and store into r_front_pre. And let’s store (1 - a_front) into a_front_pre. Now the compositing equation becomes:

r_result = r_back * a_front_pre + r_front_pre

which is one multiplication less, and one addition less. It’s obviously faster, but it can be made even faster because many processors have a special A * B + C instruction, which is also sometimes vectorized.

And yes, if r_back = 0 (black background), you can just blit, no need to even multiply.

asad_ddos said...

@Dan Bourque why don't you write an intermediate php script tht does this byteswapping optimization by first downloading the image via curl and caching them for later use on the server

CJ said...

Could this optimization issue be the reason I can not see images which should be animated on the iPhone?

I've swapped the problem images with different files and it animates fine in my build.

This coulees solve an issue both my graphic designer and I have been puzzled about.

Thanks!

Hector Tafoya said...

Interesting, but actually if you are carefull you may have a PNG of better quality and less weight than a JPEG image.

If your image uses few colors (like in a logo or something) and you don't really want transparency, you can choose those colors to make a 8-bit or less png that looks just the same.

E}I{ said...

Great post overall, but not without small technical mistakes.
First, there is a problem with your assumption that iphone framebuffer doesn't support alpha. It does, or at least can be configured this way.
Assumption about reason to byte-swap and alpha pre-multiplying is probably correct. I only must add that BGRA format is preferred by graphics hardware (and/or its drivers) both on mac and iPhone platforms. Hardware usually doesn't care about extra multiplication step, but at least texture upload is faster with BGRA. Also, if alpha-blending still has to be done by CPU-based Quartz, pre-multiplying will save significant amount of CPU time despite blending not usually being done in floating-point format. It is usually done with help of fixed-point integer operations or SIMD instructions.

Edwin said...

scrub m65 kamagra attorney lawyer body scrub field jacket lovegra marijuana attorney injury lawyer

JeansPilot said...

JeansPilot offers the chance to buy a large variety of men’s and women’s jeans clothing from the world famous Italian Brands.
Online jeans clothing store looks for original fashion clothing sales and clearances of worldwide known designers. We participate in fashion auctions to get the lowest possible price for Top quality Clothes, Shoes and Accessories.
Buy Jeans

h4ns said...

What youre saying is completely true. I know that everybody must say the same thing, but I just think that you put it in a way that everyone can understand. I also love the images you put in here. They fit so well with what youre trying to say. Im sure youll reach so many people with what youve got to say.

Arsenal vs Huddersfield Town live streaming
Arsenal vs Huddersfield Town live streaming
Wolverhampton Wanderers vs Stoke City Live Streaming
Wolverhampton Wanderers vs Stoke City Live Streaming
Notts County vs Manchester City Live Streaming
Notts County vs Manchester City Live Streaming
Bologna vs AS Roma Live Streaming
Bologna vs AS Roma Live Streaming
Juventus vs Udinese Live Streaming
Juventus vs Udinese Live Streaming
Napoli vs Sampdoria Live Streaming
Napoli vs Sampdoria Live Streaming
Fulham vs Tottenham Hotspur Live Streaming
Fulham vs Tottenham Hotspur Live Streaming
AS Monaco vs Marseille Live Streaming
AS Monaco vs Marseille Live Streaming
Alajuelense vs Perez Zeledon Live Streaming
Alajuelense vs Perez Zeledon Live Streaming
Technology News | News Today | Live Streaming TV Channels