In smartphones, the ISP or image signal processor is responsible for basic image processing tasks, such as white balance, autofocus, and HDR. Over the years, these processors have become more advanced and sophisticated, gaining the ability to process more data at a time. This is the reason for the many improvements we have seen in the quality of smartphone cameras in recent years.
Last year, Spectra 480 ISP in Qualcomm’s Snapdragon 865 broke through the barrier of 2G pixels. In other words, it can process 2 billion pixels in one second, which unlocks some very incredible features in smartphone cameras.
Processing so much data at a time can support 200MP image capture, 8K video recording, and semantic segmentation to achieve better AI photo enhancement. The overall speed has increased by 40%, which is a considerable upgrade compared to the Spectra 380 in the Snapdragon 855.
This year, Qualcomm hopes to surpass last year’s ISP to a large extent, using its new processor Qualcomm Snapdragon 888 to increase performance by 35%. But how does Qualcomm perform at a throughput of 2.7 gigapixels?
Android Authority had the opportunity to talk with Judd Heape, vice president of camera product management at Qualcomm, to find out.
One of the biggest changes in Qualcomm’s Snapdragon 888 is the addition of a third image signal processor. Since the first introduction of Spectra ISP, the previous Snapdragon SoC has encapsulated two of them. This means that smartphones using Qualcomm chips can simultaneously use two cameras for recording, but it also means that Qualcomm can provide higher throughput for a single camera.
The company switched to using Snapdragon 865 to process four pixels at a time, which significantly improved the performance of its previous generation ISP by 40%. However, this year Qualcomm is taking a simpler approach: batch transactions.
With Snapdragon 888, the company is adding another image signal processor to its SoC. This makes the total number of Qualcomm chips tripled for the first time. This unlocks a series of new features, but one of the most interesting features is the support for recording with three cameras at once.
Considering that many flagship smartphones now have at least three rear cameras, the third ISP allows users to record three focal lengths at the same time, shooting up to 4K 10-bit video at 30fps.
If you are recording a scene and want to jump between three different focal lengths while editing, then Snapdragon 888 provides support for this. OEMs must implement it into their equipment.
Using three ISPs can also improve smartphone camera zoom. Last year’s Snapdragon 865 enabled smooth zoom between sensors, giving the impression that you are using a telephoto zoom instead of three separate lenses. But with only two ISPs, it is not easy for chip manufacturers to know which lens to switch to at any given time.
Normally you will see smooth zoom between lenses, but if you accidentally swap focal lengths, there will usually be a delay when switching from one ISP to another sensor.
Assuming your smartphone has only three rear cameras, then through three ISPs, Qualcomm can activate all three sensors at the same time.
Superfast capture and playback
Thanks to the third ISP, a device running Snapdragon 888 can capture three 10-bit 4k HDR video streams at 30fps, but Qualcomm considered what would happen if all the throughput was concentrated in one place.
After all, many modern smartphones have 90Hz, 120Hz, or even 144Hz displays. Shouldn’t you record video at the same rate?
Using three Spectra 580 ISPs in Snapdragon 888, the chip can achieve a throughput of 2.7Gpix/sec (Gigapixels per second). With so much data, Qualcomm can capture and playback a single video stream at 4K 120fps.
This means that if your smartphone has a 120Hz display, you can capture and watch videos at the frame rate of the display. If you think a 60fps video is smooth, then this is the next step.
Of course, just like HDR 10 and Dolby HDR video, the device and viewing platform you watch the video on must support 120fps playback. For example, YouTube currently only supports 4K video playback at 60fps.
If you share directly with another device, the device also needs to have a 120Hz display screen to experience how the video is captured. Fortunately, the more popular the format, the more likely devices and hosting services will support the format.
Calculate HDR video
Now, thanks to the use of HDR processing technology, smartphone cameras can produce photos with amazing dynamic range, but smartphone videos often cannot benefit from the same improvements.
In fact, this is the main reason why the video recording of Google Pixel 5 is not as good as its photo function. Despite the rapid development of computational photography in recent years, video is usually an afterthought.
Now, Qualcomm hopes to change this situation through interlaced HDR video recording. First, I should note that interlaced HDR is very different from things like HDR 10 and Dolby HDR video.
Although these standards define the brightness of the display and how much color detail can be captured in the highlights and shadows, interlaced HDR video aims to achieve almost the same goals as HDR photos. It uses multiple exposure processing to preserve as much highlight and shadow detail as possible.
Interlaced HDR video captures multiple exposures for each frame at different shutter speeds. Therefore, the sensor can retain details in the highlights and shadows and then fuse the exposure to achieve a more balanced video stream.
Currently, Snapdragon 888 will merge two exposures per frame, but Heape has not ruled out fusing three or more exposures per frame in future Snapdragon models to obtain better dynamic range.
84MP image with zero Lag
On a smartphone, capturing video is almost the same as displaying a constant stream from the camera to the display. The only difference is actually storing these images.
Even so, most smartphones still temporarily store frames in RAM to capture the moments before, during, and after the shutter button is pressed. This is the technology that enables HDR photos, real-time photos, and other functions.
Last year, Qualcomm Snapdragon 865 unlocked the ability to display and capture 64MP images with zero shutter lag. This means that smartphones running the chip can display 64MP image streams at 30fps, and there is no lag or power outage between lightly pressing the shutter and storing the image in the memory.
Now you may be thinking, “Wait a minute, if it only supports 64MP images, how can a phone like the Samsung Galaxy Note 20 Ultra capture 108MP photos?”
What Heape has to say is: “For phones running Snapdragon 865, 108MP images were actually previewed at 1/4 resolution. When the user lightly presses the shutter button, the sensor will switch to 108MP mode for one frame, dump the frame into the memory, and then resume the display scene with 1/4 resolution. Therefore, Qualcomm does not consider the shutter delay of 108MP images to be zero.”
In other words, a 35% increase in the speed of the Snapdragon 888 ISP can unlock 84MP images with zero lag. This means that the user will see the full resolution image before clicking the shutter button, up to 84MP.
Although Snapdragon 888 has not reached the 3.2Gpix/sec quota, it needs to display 108MP images at full resolution, but the recent two generations of improvements make 108MP ZSL images look promising shortly.
Qualcomm set out to improve the so-called “three A’s” in Snapdragon 888. These are auto exposure, auto white balance, and autofocus. To this end, the company used machine learning to train a large number of images to train Snapdragon 888 on the appearance of images under specific conditions. However, in the case of autofocus, Qualcomm goes further. It uses real people to train the chip.
Qualcomm uses VR headsets with eye-tracking technology to track the person’s line of sight when displaying a given image. Even if the object is in the foreground, it does not mean that it is the subject of the photo.
Humans are very good at distinguishing real objects in a given scene, so using people’s eye movement training chip can make Snapdragon 888 focus on real objects with higher precision. Qualcomm said they used hundreds of topics and thousands of images to create this dataset. It is expected to be further improved in the future Snapdragon model.
Devices running Snapdragon 888 have improved the camera even more. However, many of these changes are related to other components such as the Hexagon AI engine. The third ISP alone has given Qualcomm a huge leap in processing power in Snapdragon 888, and we hope to see similar improvements in future Snapdragon generations.