adaptive multiframe image averaging – Don't hold your breath

Midweek Mobile 16

Sunlight on pines and grassland, mountains behind. It was a lovely scene which I captured with my phone. The phone used a lens which is 4.7 mm wide and has a fixed aperture f/1.7. It reported an exposure of 1/1043 seconds and an ISO of 100. The sensor on my phone has 4608 x 3456 pixels. This is an aspect ratio of 4:3, which I’ll retain in all the experiments I show here. The original jpeg image the phone gave me had 9248 x 6936 pixels. I compressed the image down to 1250 x 938 pixels in the header photo. It looks rather nice on my phone, and also on my laptop screen. The image has areas of bright illumination and areas of pretty heavy shadow. It also has some sharp colour contrasts. I was interested in how well the images look when I zoom in so that one pixel in the photo shows up as a single pixel on my laptop’s display.

Here is a zoom into brightly lit pine needles. I took a section which had 832 x 624 pixels and reduced it 640 x 480 pixels to show you in this post. All the following images also do exactly the same. You can see lots of digital artifacts. The most noticeable is aliasing: smooth lines and curves appearing jagged in the image. The software has teased out a lot of detail both in shadow and in full light, but the jaggedness makes it look somewhat artificial.

Next I zoomed into a portion of the dappled shadows. This is where your own eyes will play tricks. The camera captured almost nothing in the deepest shadow and the brightest light, but it does quite a good job even in the lighter shadows, apart from the aliasing problem. The best parts of this zoom are the portions where there is a strong contrast of illumination: bright details against dark background. But where a dark portion is seen against a bright background you see strange curves and squiggles. This is due to aliasing.

This zoom shows you a situation where the contrast is in colours, not so much in the level of illumination. Both the sky and the leaves are bright. I’m surprised by the amount of digital noise in the sky, in spite of the ISO being 100. Apart from the aliasing problems, I’m surprised by how soft the pine needles look. This is caused by a problem I’d written about earlier. The image is created by adding together a very large number of separate exposures (a technique called adaptive multiframe image averaging), and the breeze at that height causes the pine needles to move. The softness is due to the motion between different exposures. This is not a problem that a DSLR has; not does it ever have this digital noise in the sky.

I also found incredibly bad digital artifacts in a portion of the photo which looked pretty easy to take. The squiggles in the far slopes are due to aliasing. The strange halo around the shadows is another weird algorithmic effect. The light on the branches is like little bits of paint dabbed on by a bad painter trying to emulate the impressionists. The pine needles are just masses of colour. This zoom makes me think I should never again look at a phone photo blown up to see it pixel for pixel.

If I want sharp details, I should use a DSLR. A phone is what I would use if I wanted a quick snapshot which I would look at only on a little screen which fits in my palm. Conversely, if you want to see the defects in the phone photo, look at these examples on a big screen, not a phone.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek mobile 15

Modern phone cameras get sharp and bright images with awful lenses and jokes of sensors. The most important aspect of the images is that they are usually viewed on the small screen of a phone. A quick search led me to an estimate that people take 4.7 billion photos every day. Be suspicious of such facile estimates. But it is clear that far less than a percent of a percent would be viewed on a large screen, where defects can show.

I stress tested my phone camera in exactly this way. My phone has a sensor with 4608 x 3456 pixels. I reduced it to 1667 x 1250 pixels for the leader photo. That looks good. But I asked what it I looked at it pixel for pixel: one pixel of the sensor for every pixel on the screen (1:1). I did that in the most detailed photo in the slideshow above. The next one compressed 4 pixels of the photo into one on the display (4:1), the next 16 pixels of the photo for one on the display (16:1), and the next (the featured photo) is shown 32 pixels per pixel of display (32:1). But for the post I compressed these views a little more; the closest is at 9:1, the rest are 36:1, 144:1 and 288:1. The result begins to show digital artifacts in the 9:1 view, although they are not overwhelming (at 1:1 they are unmistakable). Of course, I can’t predict what screen you’ll see them on, but if you have a choice looking at them on the biggest screen you have would be interesting.

On a whim I took a photo of a beetle and gave it the same treatment. Here you see the views in the ratios 9:1 (nine pixels of the photo to one of the display), with the successive frames showing 36:1, 144:1 and 288:1 compressions. It is only the last which looks sharp. On my phone the display is even smaller, so the image looks much sharper. But why this big difference between flora and fauna? I compared the exposure first. The flowers are taken with an equivalent exposure of 1/100 seconds and ISO of 100; the beetle with 1/50 seconds and ISO of 223. This means that the number of frames which are superimposed to give the final image is twice as many in the second. Slight hand movements could create the effect that you see, but the phone must compensate for that. But the ISO is also a factor; you can see more “grain” in the image of the beetle. I think another important factor must be the contrast between the object and background. That’s much smaller in the second photo. I’ll try to explore this further.

If you want a moral, I would say “Don’t look a gift horse in the mouth.” Your phone does not replace a good DSLR in image quality. Be happy with what it shows on its small display.

Midweek mobile 8

Puri, an ancient temple town, is the perfect place for street photos. No camera can be more discreet these days than a phone; bystanders can seldom tell whether you are taking a selfie or a photo of the street. Gone are the days when you saw a photographer and you would walk around them. These days you could land up photo-bombing a selfie. I walked about taking more shots than I could ever use. I had a few destinations in mind, and since these small lanes are a little confusing, I had my maps and location service on. I knew that all this can eat charge like a hungry spider. This time I was going to track exactly how much.

Normally I charge the battery fully, and it gives me a low battery alert when the charge has fallen to 15% of its capacity. On the average I have to charge my phone every three days. That means in an average hour I use 2.3% of the charge. After an hour of walking, I saw that maps and camera had each been on for the hour. Maps had eaten 3% of charge, but the camera had eaten just over 10%. This was just the camera software, since the display is counted separately. This agreed with my previous experience, that I would need to charge my camera after a day’s shooting.

To understand why, back up a little. These photos are zoomed in by a factor of about 4 to 8. With a DSLR setup you would not expect to capture the details of the old man’s mustache using a 15 mm diameter lens which has a focal length of 26 mm. The sensor size on my phone is almost 8 times smaller than that on most DSLRs, and therefore catches that much less light. The sharpness that you see comes from the number of output pixels in the image. That pixel density is due to intense computation, including two components that I’ve explored before: computational super-resolution and image averaging over a very large number of images. Driving the software that tries to compensate for the hardware limitations is what uses up so much charge. Algorithms will improve in future, mass market hardware will become better, and processors will run cooler. But until then, the carbon footprint of phone photography will remain several times larger than that of communication.

Midweek Mobile 4

Does AI have limits? You come to this question very quickly when you begin to think about phone cameras. They have tiny lenses which would once have limited both the amount of light and the resolution of photos. Once upon a time, the limited aperture would have meant long exposure times (and camera shake). It would also have created a resolution problem: that you could not get distant details with limited aperture (that’s bokeh for you). How does a phone camera wriggle out of this problem and produce photos as sharp as these two?

There are two parts to the answer. One is physics: the chemistry of silver halide crystals is replaced by the electronics of CCDs. The pixels can be made smaller, and there are clever constructions for noise reduction. As a result, you get closer to the ideal of “one photon, one pixel”, although not very close, at least on the cheap phone that I use. The other is mathematics: there is a lot of computation between what you see and what the CCD gives. First there is the subtraction of the background noise. Then there is the AI which begins to make statistical inferences about the image. Earlier I’d mentioned computational super-resolution: the improvement of lens resolution by making assumptions about the nature of the lighting. In both the photos above I looked at another technique that the AI uses: image averaging.

When I looked at this scene of the Sahyadris, there was a wall of dragonflies between me and the far cliffs. Each hovered in the air to find prey, then quickly darted to it. The light was not too bad, and on a normal camera, many would be blurred, but some would perhaps be sharp because they would be caught hovering. I wondered what the 65 megapixels of my phone camera would catch. Surprise! It caught nothing, although the EXIF told me that the exposure was 1/1912 seconds. Nothing at all, as you can see in this full size zoomed segment of the photo. I went over the photo segment by segment at full size. Nothing! What happened?

The phone camera took multiple readouts (frames) from the CCD sensor and then added them together to form the final image. This image averaging give noise reduction: pixels are averaged over frames and random noise is cancelled. But the random darting of the dragonflies also mimicked noise, and was removed. The exposure time written on the EXIF is probably a sum over the exposure times of the frames. The shorter reported exposure perhaps means that a smaller number of frames is averaged over.

Do I have an image that tells me that the camera is doing image averaging? Yes, the image comparator above tells me that. The “original image” (compressed for the purposes of this blog to 640×480 pixels) is shown on the left. The photo was taken from the car as it slowed for a speed bump. The EXIF data tells me that this was shot at f/1.7 with an exposure of 1/2422 seconds. In that time I estimate that the car must would have moved by a hair over 1/2 mm. The original looks sharp here, and looked even sharper on my phone. But the full size zoom shows strange artifacts. The lettering on the signboard is very blurred, as it would be if multiple images were added together. But the narrow leftmost pole supporting the roof of the shack is perfectly clear. Similarly, the edges of the sun umbrella are clean. This is clear evidence that the AI has selectively added parts of images. Even more than image averaging, there is clearly adaptive multiframe image averaging at work.

A 1450×1088 pixel section of two photos reduced to 640X480 pixels are shown here for comparison. The left from a phone camera, the right with a macro lens.

Now let’s get back to the photo of moss on a brick wall to see how much detail I could get from it. It was taken in full sunlight. At f/3.2 my macro lens required an exposure of 1/200 of a second to capture the moss in the comparison photo on the right. The phone camera lens has about 1/25 of the area, so if I’d masked my macro lens to keep only a phone camera sized area free, the exposure would have climbed to 1/8 of a second. But the phone camera reported f/1.7 (the lens is fixed), with an exposure of 1/264 seconds. Yet when I looked at the phone camera output at full size, I saw the blur on the left! Why?

First, keep in mind that the exposure time of the photo of moss implies averaging about 7 times as many frames as that of the cliff. You might expect so much averaging to remove blur. But I suspect that the blur in this second photo is an due to image averaging interacting with computational super-resolution: the improved lens resolution that AI gives. Since the small details in the zoomed view is almost entirely due to computation, little changes in the illumination can change the inferred image. Then averaging over the result can give the blurred details that you see. In the second zoom into the same photo you can see that the deep shadows look more noisy and equally blurred. This is also what you might expect from the averaging over super-resolved frames: CCD noise is removed, but inferred details are blurred by averaging over inferences.