Midweek Mobile 7

When do you switch modes on your phone? The featured photo of a statue of a lion was done in portrait mode. On my old phone, with a single lens on the back, going to portrait mode would allow me to move closer to the object. Under the hood what it did was to computationally undo the barrel distortion that mars photos when you move the lens too close to the subject. My current phone has three lenses on its back, and it always merges the images from them, correcting distortions on the fly. So the portrait mode is used for something else: it calls on an AI to edit the photo.

This image comparison shows two photos I took of The Family from across a table as we waited for our desserts to be delivered. The lighting was low, and I decided to use the rim of a glass to provide a bit of support to steady the phone. The main effect of portrait mode seems to be to blur the background. The other photo is taken with the default setting. I’d expected the portrait mode to smooth the shadows on the skin, but instead it seems to have given it a colour that looks mottled if I see it pixel for pixel (full size). Most of the skin looks smoother in the default. We’d just walked in from outside, and she had some water on her arms. The default sees the light reflect off the edge of her wet arm as a blue streak. The portrait tries to remove the blue. The result is a strange distortion of the single drop of water close to her wrist. In both photos you see that the computational image processing has made a hash of the outline of the arm around where the background illumination is high. I must say that looking at the photos at full size I’m very disappointed with the rendering of the image in both modes. I don’t have any of these artifacts with an older camera where computation plays a smaller role.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek Mobile 6

You can completely lose control of what you are doing when you take a photo with a mobile phone. Sure, there are all those presets which make you think you are in control: portrait mode, night photo, food photo, aspect ratio, quality, HDR. So many settings, so little control. Every setting instructs your resident AI to switch on some effect. That’s why it’s such a mass-market rage: you can get seriously good results with no effort at all. And that’s why serious photographers have still not taken to it. How to do you get the instrument to render your vision?

So here was one little experiment I did. I noticed that it computed the exposure (that’s a loaded word, see my previous field notes on exposure) through centered weighted averaging. The camera estimated the amount of illumination in the scene by taking an average over the frame with more weight given to the center. I took a photo first of the artfully aged wooden table in the dimly lit bar, with only my reading glasses and its case. The camera declared that it had used f/2.8 and an effective exposure of 1/17 second. Then, when my wine was delivered, I plonked it in the center of the same frame and took another photo. Now the AI decided to use a bigger lens from the cluster of lenses that phones are now studded with. It reported an aperture of f/1.7 with the same exposure as before. Notice how the light on the table changed?

I can’t carry red wine with me on every photo shoot (sigh!), but I can point the center of the field at different places in the frame to influence the quality of light. With 65 megapixels to play with, I can often sacrifice part of the frame to get the light that I want in the rest of the frame. A kludge, to be sure, but if you want a modicum of control in a mass-market device that gives you none, this is a workable hack.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek mobile 5

Panoramas have had a cult following since before the early days of photography. Special cameras and odd format film plates were developed for this purpose even in the 19th century CE. I came to the field in the early days of commercial digital photography when enthusiasts would spend days stitching together individual exposures with the early versions of photo-editors. When I tried my hand at it I rapidly realized two important points. First, as you rotate the camera, the illumination changes, and you have to compensate for the different quality of light in the editor. Second, as you image the same object from slightly different angles, its shape on the film changes slightly, and you come across mismatched edges when you try to stitch the images together. You can understand this as a simple problem in perspective, but it was hard to compensate for it with the photo editor tools that were then available.

Now, a smart camera does all this “in the box” for you. On my drive in the Sahyadris, I stopped near a village and took a panorama of a wooded cliff above rice fields. All I had to do was stand firm, hold the phone in my hands and twist my torso smoothly. The demon in the box took care of the rest: the shading of light across the image, and the automatic adjustment of the distortions of objects due to changing angles. The second step, the one which was hard to do by hand, has a simple mathematical representation (three-point perspective) which the AI solves rapidly. The result seems to be good enough that you can wrap it around the walls of your home.

But my phone records the panoramas only in 40 megapixel images, not the 65 megapixels which it uses for other photos. This is due to the physical limitations of the small aperture and sensor which the phone camera uses. I’ve discussed earlier how multiple frames are read out from the CCD and averaged to produce a single image. The same thing is being done for panoramas. But since the camera is moving while the image is synthesized, the number of frames available for a single segment is limited. When you examine a regular image and a panorama at the same scale, you can see this clearly. In the image comparison above, both photos use the same number of pixels from the original image. I can zoom less when I use a panorama. This is the result of using a smaller number of frames for image averaging in the panorama, and also the restriction on computational super-resolution imposed when using the smaller number of frames. So really, you cannot wrap the panorama from a cheap phone camera around the walls of your home. At least not until the sensor and frame-readouts, or the processor and algorithms improve.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek Mobile 4

Does AI have limits? You come to this question very quickly when you begin to think about phone cameras. They have tiny lenses which would once have limited both the amount of light and the resolution of photos. Once upon a time, the limited aperture would have meant long exposure times (and camera shake). It would also have created a resolution problem: that you could not get distant details with limited aperture (that’s bokeh for you). How does a phone camera wriggle out of this problem and produce photos as sharp as these two?

There are two parts to the answer. One is physics: the chemistry of silver halide crystals is replaced by the electronics of CCDs. The pixels can be made smaller, and there are clever constructions for noise reduction. As a result, you get closer to the ideal of “one photon, one pixel”, although not very close, at least on the cheap phone that I use. The other is mathematics: there is a lot of computation between what you see and what the CCD gives. First there is the subtraction of the background noise. Then there is the AI which begins to make statistical inferences about the image. Earlier I’d mentioned computational super-resolution: the improvement of lens resolution by making assumptions about the nature of the lighting. In both the photos above I looked at another technique that the AI uses: image averaging.

When I looked at this scene of the Sahyadris, there was a wall of dragonflies between me and the far cliffs. Each hovered in the air to find prey, then quickly darted to it. The light was not too bad, and on a normal camera, many would be blurred, but some would perhaps be sharp because they would be caught hovering. I wondered what the 65 megapixels of my phone camera would catch. Surprise! It caught nothing, although the EXIF told me that the exposure was 1/1912 seconds. Nothing at all, as you can see in this full size zoomed segment of the photo. I went over the photo segment by segment at full size. Nothing! What happened?

The phone camera took multiple readouts (frames) from the CCD sensor and then added them together to form the final image. This image averaging give noise reduction: pixels are averaged over frames and random noise is cancelled. But the random darting of the dragonflies also mimicked noise, and was removed. The exposure time written on the EXIF is probably a sum over the exposure times of the frames. The shorter reported exposure perhaps means that a smaller number of frames is averaged over.

Do I have an image that tells me that the camera is doing image averaging? Yes, the image comparator above tells me that. The “original image” (compressed for the purposes of this blog to 640×480 pixels) is shown on the left. The photo was taken from the car as it slowed for a speed bump. The EXIF data tells me that this was shot at f/1.7 with an exposure of 1/2422 seconds. In that time I estimate that the car must would have moved by a hair over 1/2 mm. The original looks sharp here, and looked even sharper on my phone. But the full size zoom shows strange artifacts. The lettering on the signboard is very blurred, as it would be if multiple images were added together. But the narrow leftmost pole supporting the roof of the shack is perfectly clear. Similarly, the edges of the sun umbrella are clean. This is clear evidence that the AI has selectively added parts of images. Even more than image averaging, there is clearly adaptive multiframe image averaging at work.

A 1450×1088 pixel section of two photos reduced to 640X480 pixels are shown here for comparison. The left from a phone camera, the right with a macro lens.

Now let’s get back to the photo of moss on a brick wall to see how much detail I could get from it. It was taken in full sunlight. At f/3.2 my macro lens required an exposure of 1/200 of a second to capture the moss in the comparison photo on the right. The phone camera lens has about 1/25 of the area, so if I’d masked my macro lens to keep only a phone camera sized area free, the exposure would have climbed to 1/8 of a second. But the phone camera reported f/1.7 (the lens is fixed), with an exposure of 1/264 seconds. Yet when I looked at the phone camera output at full size, I saw the blur on the left! Why?

First, keep in mind that the exposure time of the photo of moss implies averaging about 7 times as many frames as that of the cliff. You might expect so much averaging to remove blur. But I suspect that the blur in this second photo is an due to image averaging interacting with computational super-resolution: the improved lens resolution that AI gives. Since the small details in the zoomed view is almost entirely due to computation, little changes in the illumination can change the inferred image. Then averaging over the result can give the blurred details that you see. In the second zoom into the same photo you can see that the deep shadows look more noisy and equally blurred. This is also what you might expect from the averaging over super-resolved frames: CCD noise is removed, but inferred details are blurred by averaging over inferences.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

The rain in Spain

Flying in to Madrid from Delhi a few years ago, our plane had taken a southerly course over the Mediterranean. It was morning when we came in over the Spanish coast somewhere in the province of Valencia. A half hour of flight remained, and most of it was over the countryside of Castilla-La Mancha.

The portholes of Dreamliners have complex optics, with a gel filled layers between the stressed outer layers and the innermost clear layer. All portholes on airliners create a bit of chromatic aberration, but I had the feeling that this was much worse than normal. It was very obvious in this photo, which I took just as we hit the coast of Spain. The best I could do was to avoid the edges of the porthole. Due to a Dreamliner’s larger portholes, this is not as hard a constraint as it can be on some other planes. There are a lot of breakwaters and docking areas in that port. I wonder which one it is.

We were to discover later that we had landed during one of the hottest summers that Spain had had in recent years. But even before we knew that, it was clear that we were flying over some pretty parched lands. This green low-land with its lakes was the last bit of green we saw in quite a long while. I took a lot of photos from the air, but seeing anything in them requires quite a lot of work. The photos that you see here required fiddly adjustment of contrast and brightness. Still learning!

The parched landscape around this town is more typical of the area we flew over. I found later that a large part of the flat country that we flew over has average annual rainfalls between 400 and 600 mm. No wonder the fields that you can see in the photo above look so dry. Eliza Doolittle should have fact-checked the sentence she was taught. The rain in Spain does not stay mainly in the plain. Not at all. Most of it is in the north or along the mountains. I wonder how quickly, and by how much, these patterns will change in the coming decades.