Midweek Mobile 7

When do you switch modes on your phone? The featured photo of a statue of a lion was done in portrait mode. On my old phone, with a single lens on the back, going to portrait mode would allow me to move closer to the object. Under the hood what it did was to computationally undo the barrel distortion that mars photos when you move the lens too close to the subject. My current phone has three lenses on its back, and it always merges the images from them, correcting distortions on the fly. So the portrait mode is used for something else: it calls on an AI to edit the photo.

This image comparison shows two photos I took of The Family from across a table as we waited for our desserts to be delivered. The lighting was low, and I decided to use the rim of a glass to provide a bit of support to steady the phone. The main effect of portrait mode seems to be to blur the background. The other photo is taken with the default setting. I’d expected the portrait mode to smooth the shadows on the skin, but instead it seems to have given it a colour that looks mottled if I see it pixel for pixel (full size). Most of the skin looks smoother in the default. We’d just walked in from outside, and she had some water on her arms. The default sees the light reflect off the edge of her wet arm as a blue streak. The portrait tries to remove the blue. The result is a strange distortion of the single drop of water close to her wrist. In both photos you see that the computational image processing has made a hash of the outline of the arm around where the background illumination is high. I must say that looking at the photos at full size I’m very disappointed with the rendering of the image in both modes. I don’t have any of these artifacts with an older camera where computation plays a smaller role.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek mobile 5

Panoramas have had a cult following since before the early days of photography. Special cameras and odd format film plates were developed for this purpose even in the 19th century CE. I came to the field in the early days of commercial digital photography when enthusiasts would spend days stitching together individual exposures with the early versions of photo-editors. When I tried my hand at it I rapidly realized two important points. First, as you rotate the camera, the illumination changes, and you have to compensate for the different quality of light in the editor. Second, as you image the same object from slightly different angles, its shape on the film changes slightly, and you come across mismatched edges when you try to stitch the images together. You can understand this as a simple problem in perspective, but it was hard to compensate for it with the photo editor tools that were then available.

Now, a smart camera does all this “in the box” for you. On my drive in the Sahyadris, I stopped near a village and took a panorama of a wooded cliff above rice fields. All I had to do was stand firm, hold the phone in my hands and twist my torso smoothly. The demon in the box took care of the rest: the shading of light across the image, and the automatic adjustment of the distortions of objects due to changing angles. The second step, the one which was hard to do by hand, has a simple mathematical representation (three-point perspective) which the AI solves rapidly. The result seems to be good enough that you can wrap it around the walls of your home.

But my phone records the panoramas only in 40 megapixel images, not the 65 megapixels which it uses for other photos. This is due to the physical limitations of the small aperture and sensor which the phone camera uses. I’ve discussed earlier how multiple frames are read out from the CCD and averaged to produce a single image. The same thing is being done for panoramas. But since the camera is moving while the image is synthesized, the number of frames available for a single segment is limited. When you examine a regular image and a panorama at the same scale, you can see this clearly. In the image comparison above, both photos use the same number of pixels from the original image. I can zoom less when I use a panorama. This is the result of using a smaller number of frames for image averaging in the panorama, and also the restriction on computational super-resolution imposed when using the smaller number of frames. So really, you cannot wrap the panorama from a cheap phone camera around the walls of your home. At least not until the sensor and frame-readouts, or the processor and algorithms improve.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.

Midweek mobile 2

A mobile camera is not a good camera in ways that photographers were used to thinking of. The lens is a toy. Four centwp-admin/wp-admin/wp-admin/uries worth of lens technology have been junked by two related developments. The most important? That about 95% of the world looks at photos on tiny screens when distributing likes. So you don’t need the sharpness that photographers of old wanted; sell megapixels instead. That translates to about 10 Mbytes for the featured photo when my camera saves it. I know from experience that even on my large screen I can easily compress it down to about 200 kbytes and most people would not be able to tell the difference. That means I can retain only 2% of what is recorded. And on my phone I could easily throw away another 90% of the information (retain just 0.2% of the original) and no one would be able to tell. Then why so many megapixels? Because when you start from a large format photo and compress it down to a small screen, everything looks sharp.

You might remember that when you last changed your phone the picture quality changed a lot. Is that all due to more pixels? In a big part, yes. I dropped my old phone too often and was forced to change it quicker than I normally do. In three years the number of pixels in photo from a less-than-mid-range phone had gone up from around 10 million to about 65 million. Now look at the featured photo. The architectural details look sharp, considering that the subject is more than 300 meters away, and it was taken from a car that was making a sharp turn at a reasonable speed. But look at the near-full size blow-up in the photo above. You can see that at this zoom, details are pretty blurred. I have gained the clarity of the featured photo purely by not looking at it at full scale.

But that’s not the only change when you get a new phone. You also get a different AI translating the sensor output into an image. And this technology, which is a guess at what is being seen, is improving rapidly. As a result, the distortions of a bad lens can still be interpreted better, and result in a reasonable image. Note that this phone can remove digital noise much better than a five years-old phone would have done. The darker areas of the photo are much more clean (the detailed view above has been cropped out in the featured photo). Also, notice that the new generation AI deals with non-white faces better than before, getting an impressive image for the man walking towards the camera. This improvement is a response to accusations of biased training of AI.

But another detail is technically very impressive. Notice the level of detail? I can see very clearly that he is not wearing a mask. This resolution is better than a fundamental limitation which is imposed on lenses due to the wave nature of light (something called Rayleigh’s resolution limit). This computational super-resolution is a statistical trick which improves the image by making a guess about the nature of the ambient light. The down side of all this AI? This much of computation has a carbon cost. When I use my phone only for communication, the batteries last three and a half days. Street photography can drain the charge in a few hours.

Phone photography changes our expectation of the interaction of camera hardware and image so dramatically that it is worth rethinking what photography means. I intend to explore this a bit in this series.