You keep hearing about AI-powered cameras, but it often seems like the Wizard of Oz is working behind the curtain. Now Samsung has released a pretty detailed explanation of how the Galaxy S21 series uses AI to create impressive portraits.
The processing of the selfie camera is quite simple. First, the software identifies the faces in the image and marks them for further processing (this is called segmentation). Second, the details of the hair, eyes and facial features are enhanced. The camera also adjusts the white balance to achieve natural skin tones in any ambient light.
Portrait mode is much more complex. It starts with segmentation so that later steps know which parts of the image are humans, which are pets, and which are just the background. This way, the correct treatment can be applied if necessary. This segmentation map is used to create an approximate “starting map”.
Next comes the “tri-map”, which is important because it highlights the border between the subject and the background. Then the matte card draws the fine details inside that border – it prevents hair and facial features from blending into the background. Finally, the Depth Estimate pass calculates the distance to objects, which will be used to create the shallow depth of field effect.
If you’ve come across less than perfect portrait modes, you’ll know how often strands of hair and other fine details are blurry in the background, making the image feel very artificial. It’s the precision of the matte card that sets Samsung portraits apart from the rest.
Below is an example of three maps (center image), white for the subject, black for the background, and gray for the uncertain areas. On the right is the resulting matte map, which refines the subject / background separation.
The phone takes the starting cards to apply image enhancements to the subject and blur the background. Then he adds the matte map to create a sharp divide between the subject and the background. All of these layers are processed and combined into the final image in 3 seconds.
Source
