SAMSUNG

Q: What is Under Display Camera (UDC)?

Seok-Jun: The Under Display Camera (UDC) is an innovative camera solution that hides the camera lens beneath the display panel, completely freeing up the screen during application execution for an uninterrupted mobile device viewing experience. Placing the camera under the display panel requires hardware technology and algorithmic innovation to restore image quality, which is a highly complex image restoration problem.

Q: How does the Under Display Camera work?

Alok: In terms of the UDC architecture, the front camera is located at the bottom of the display panel. The screen pixels in this area are reduced in density, allowing the screen to display images without obstruction, while still allowing light transmission when the camera is active. Because the camera is placed under the display panel, UDC can cause diffraction, leading to flares, light saturation, and blur, resulting in poor image quality. Therefore, while UDC offers a better viewing experience, it also affects camera quality and other downstream visual tasks. These complex and diverse distortions make restoring UDC image quality highly challenging.

Q: Why is UDC a key technology for the Galaxy Z Fold4? What value does it create for consumers?

Seok-Jun: Samsung's foldable phones not only boast innovative design but also combine smartphone functionality with an immersive viewing experience through groundbreaking display technology, all within a single mobile device. When folded, it serves the purpose of a regular smartphone; when unfolded, its display offers an elevated immersive experience and high productivity. According to Samsung consumer research, Galaxy Z Fold series users prefer using the cover screen camera for photos and selfies, while opting for the immersive inner screen for high productivity tasks like watching videos, gaming, and document processing. Therefore, we believe UDC technology meets the user's demand for an unobstructed visual experience and enables tasks like video calls. Samsung first introduced the foldable UDC solution with the Galaxy Z Fold3, and its image quality has been further enhanced with the arrival of the Galaxy Z Fold4.

Alok: The Galaxy Z Fold4 ushers in a new era of immersive displays, with UDC technology serving as a crucial cornerstone for the full-screen display experience. The Galaxy Z Fold4 combines advanced innovation in display technology with AI in the field of image restoration, achieving a perfect balance between screen quality and camera image quality.

Q: How has the image quality of the UDC evolved since the Galaxy Z Fold3?

Alok: Samsung was a pioneer in adopting UDC technology for foldable phones with the Galaxy Z Fold3. One of the key feedback points from consumers after the Galaxy Z Fold3 launch was the desire for the UDC to be less noticeable. The Galaxy Z Fold4 optimizes the arrangement of sub-pixels to achieve better visual invisibility for the Under Display Camera (UDC).

As mentioned earlier, when the camera is placed under the display panel, it can cause diffraction, leading to flares, light saturation, and blur, resulting in poor image quality.

For this reason, for the Galaxy Z Fold4 product design, we leveraged AI, backed by multi-frame processing technology, to develop a new computational imaging pipeline for UDC image restoration, addressing these issues. Simultaneously, Samsung's SRA MPI lab evolved its multi-frame computational photography algorithms. The UDC adopted by the Galaxy Z Fold4 resolves key technical challenges in image quality, providing a unique experience for end consumers.

Q: Please guide us through the SRA-SRI collaboration history. Why did these two teams collaborate?

Seok-Jun: Since Samsung established the MPI lab, SRA MPI has maintained a close collaborative relationship with SRI-B, evident in the commercialization of the Galaxy Note4 camera in 2014, and continuing through the Galaxy Z Fold4 and Z Flip4 in 2022. SRA MPI has consistently taken the lead in developing MX Multi-Frame Processing (MFP) technology, Night Mode, Expert RAW, and other technologies. SRI-B has made significant contributions by developing impressive camera software, image tuning, commercialization resources, and AI camera R&D, enabling the commercialization of all commercialized SRA MPI MFP technologies. The seamless collaboration among the MX R&D team, SRA MPI, and SRI-B has led to successful milestones, especially the global teams efficiently utilizing time differences to achieve this complex project's milestones. It is a pleasure to work alongside them towards the same goal, reaching the finish line, and creating the world's most outstanding smartphone cameras.

"The seamless collaboration among the three teams has led to successful milestones, especially the global teams efficiently utilizing time differences to achieve this complex project's milestones. It is a pleasure to work alongside them towards the same goal, reaching the finish line, and creating the world's most outstanding smartphone cameras."

- John Seok-Jun Lee

Q: Please describe your role and your team's contribution to UDC. What efforts has your team made to provide the best user experience?

Alok: The Galaxy Z Fold4 UDC is a complex R&D project, and SRI-B's role is to think end-to-end to achieve the best consumer experience. This requires cross-domain expertise and deep collaboration across teams. The experts from SRI-B, SRA-MPI lab, and the MX Visual Software R&D team have collaborated seamlessly to achieve the end-to-end UDC experience.

SRI-B also plays a crucial role in various aspects, including utilizing Neural Processing Units (NPUs) for AI models to achieve real-time performance for previews, significantly reducing UDC image processing time compared to the Galaxy Z Fold3. Real-time processing performance is essential for camera preview, video recording, and video calls to avoid missing any frames. AI UDC image restoration accelerated by NPU is being used for the first time in the Samsung Galaxy Z Fold4. While photo capture does not require real-time processing, the demand for higher image quality, achieved with shorter processing times, is what we strive for in user experience.

"We leverage AI to develop a new computational imaging pipeline that supports UDC image restoration. Consequently, the UDC adopted by the Galaxy Z Fold4 addresses key technical challenges in image quality, delivering a unique experience for consumers by providing better detail and sharper, clearer images."

- Alok Shukla

Q: What challenges did you face during the development of this technology? How did you overcome them?

Seok-Jun: Low light transmission and light diffraction are the biggest challenges for UDC camera technology. Another challenge, different from punch-hole camera imaging pipelines, is that the image quality captured by UDC is affected not only by the camera module but also by the display panel. SRA MPI's AI 3D ISP algorithms are designed with the necessary flexibility to adjust the output image quality based on different display characteristics. This flexibility of the 3D ISP helps overcome the aforementioned challenges.

Alok: SRI-B's key task is to develop AI image restoration modules that support the Samsung Galaxy Z Fold4 UDC. These restoration modules are essential components for both photo and video modes. Use cases such as video recording and video calls impose strict requirements for real-time image processing within limited power consumption.

Q: What are your aspirations for future iterations of UDC technology?

Alok: The UDC technology in the Samsung Galaxy Z Fold4 opens up new dimensions for smartphone form factors. Future UDC technology may enable larger camera sensors and flexible multi-camera configurations without compromising the viewing experience. Another significant challenge lies in the large-scale adoption of UDC for front cameras across more products. To achieve this goal, we must make substantial upgrades in camera image quality, processing time, and display experience.

Seok-Jun: Given the significant enhancement in user experience with the Galaxy Z Fold4 UDC, including hiding the camera under the screen and improving image quality, the development direction for future flagship devices will focus on making the UDC completely invisible. This will provide an uninterrupted, immersive mobile display experience with image quality comparable to punch-hole cameras.

Note 1: Samsung Research (SRA) serves as the forward-looking R&D hub for Samsung's DX (Device eXperience) business unit.