Canon Develops A New Wide Dynamic Range Sensor With “area-specific exposure”

It seems Canon is actively developing a new imaging sensor with wide dynamic range, according to news coming from Japan.

Newswitch (Nikkan Kogyo Shimbun) posted an interesting article about a new Canon imaging sensor with wide dynamic range and featuring a technology called “area-specific exposure”.

Canon has developed a new complementary metal-oxide-semiconductor (CMOS) sensor product (pictured) suitable for surveillance applications. The dynamic range, which is the range of brightness and darkness that can be expressed, is 148 decibels, which is the highest level in the industry for surveillance cameras, according to the company. High-quality images can be obtained even in environments with a large difference in brightness. Aiming for early commercialization.

The developed CMOS sensor is a layered type with two layers: pixels that convert external light into electrical signals and a central processing unit (CPU) that performs calculations. Compared to when pixels and CPUs are placed on the same substrate, more CPUs and dedicated circuits can be mounted. Processing power has increased, making it easier to shoot high-precision images even when the subject is moving.

When shooting in an environment with a large difference in brightness, normally multiple images with different exposure times are combined and processed to correct the difference in brightness. However, since images with different exposure times are combined, there was the problem that the moving subjects overlapped several times and appeared blurred.

The CMOS sensor developed by Canon this time divides the screen finely and adopts “exposure by area” that changes the exposure conditions for each area. Since no compositing process is required, there is no need to worry about blurring the subject. It can contribute to improving the performance of surveillance cameras by, for example, simultaneously recognizing the vehicle license plate and driver’s face at the entrance/exit of an underground parking lot.

The metal oxide semiconductor (CMOS) imaging sensor we are talking here is first and foremost for surveillance applications. Don’t expect it in your next Canon consumer mirrorless camera. The imaging sensor has two layers: pixels that convert external light into electrical signals and a central processing unit (CPU) that performs calculations. And here comes the trick used by Canon:

When shooting in an environment with a large difference in brightness, normally multiple images with different exposure times are combined and processed to correct the difference in brightness. However, since images with different exposure times are combined, there was the problem that the moving subjects overlapped several times and appeared blurred. The CMOS sensor developed by Canon this time divides the screen finely and adopts “exposure by area” that changes the exposure conditions for each area. Since no compositing process is required, there is no need to worry about blurring the subject.

Cool, kudos Canon.

And here is the press release:

Canon develops CMOS sensor for monitoring applications with industry-leading dynamic range, automatic exposure optimization function for each sensor area that improves accuracy for recognizing moving subjects

TOKYO, January 12, 2023—Canon Inc. announced today that the company has developed a 1.0-inch, back-illuminated stacked CMOS sensor for monitoring applications that achieves an effective pixel count of approximately 12.6 million pixels (4,152 x 3,024) and provides an industry-leading¹ dynamic range of 148 decibels² (dB). The new sensor divides the image into 736 areas and automatically determines the best exposure settings for each area. This eliminates the need for synthesizing images, which is often necessary when performing high-dynamic-range photography in environments with significant differences in brightness, thereby reducing the amount of data processed and improving the recognition accuracy of moving subjects.

With the increasingly widespread use of monitoring cameras in recent years, there has been a corresponding growth in demand for image sensors that can capture high-quality images in environments with significant differences in brightness, such as stadium entrances and nighttime roads. Canon has developed a new sensor for such applications, and will continue to pursue development of sensors for use in a variety of fields.

The new sensor realizes a dynamic range of 148 dB—the highest-level performance in the industry among image sensors for monitoring applications. It is capable of image capture at light levels ranging from approximately 0.1 lux to approximately 2,700,000 lux. The sensor’s performance holds the potential for use in such applications as recognizing both vehicle license plates and the driver’s face at underground parking entrances during daytime, as well as combining facial recognition and background monitoring at stadium entrances.

In order to produce a natural-looking image when capturing images in environments with both bright and dark areas, conventional high-dynamic-range image capture requires taking multiple separate photos under different exposure conditions and then synthesizing them into a single image. Because exposure times vary in length, this synthesis processing often results in a problem called “motion artifacts,” in which images of moving subjects are merged but do not overlap completely, resulting in a final image that is blurry. Canon’s new sensor divides the image into 736 distinct areas, each of which can automatically be set to the optimal exposure time based on brightness level. This prevents the occurrence of motion artifacts and makes possible facial recognition with greater accuracy even when scanning moving subjects. What’s more, image synthesizing is not required, thereby reducing the amount of data to be processed and enabling high-speed image capture at speeds of approximately 60 frames-per-second³ (fps) and a high pixel count of approximately 12.6 million pixels.

Video is comprised of a series of individual still images (single frames). However, if exposure conditions for each frame is not specified within the required time for that frame, it becomes difficult to track and capture images of subjects in environments with subject to significant changes in brightness, or in scenarios where the subject is moving at high speeds. Canon’s new image sensor is equipped with multiple CPUs and dedicated processing circuitry, enabling it to quickly and simultaneously specify exposure conditions for all 736 areas within the allotted time per frame. In addition, image capture conditions can be specified according to environment and use case. Thanks to these capabilities, the sensor is expected to serve a wide variety of purposes including fast and highly accurate subject detection on roads or in train stations, as well as stadium entrances and other areas where there are commonly significant changes in brightness levels.

Example use case for new sensor

• Parking garage entrance, afternoon: With conventional cameras, vehicle’s license plate is not legible due to whiteout, while driver’s face is not visible due to crushed blacks. However, the new sensor enables recognition of both the license plate and driver’s face.
• The new sensor realizes an industry-leading high dynamic range of 148 dB, enabling image capture in environments with brightness levels ranging from approx. 0.1 lux to approx. 2,700,000 lux. For reference, 0.1 lux is equivalent to the brightness of a full moon at night, while 500,000 lux is equivalent to filaments in lightbulbs and vehicle headlights.

Technology behind the sensor’s wide dynamic range

• With conventional sensors, in order to produce a natural-looking image when capturing images in environments with both bright and dark areas, high-dynamic-range image capture requires taking multiple separate photos under different exposure conditions and then synthesizing them into a single image. (In the diagram below, four exposure types are utilized per single frame).
• With Canon’s new sensor, optimal exposure conditions are automatically specified for each of the 736 areas, thus eliminating the need for image synthesis.

Technology behind per-area exposure

• Portion in which subject moves is detected based on discrepancies between first image (one frame prior) and second image (two frames prior). ((1) Generate movement map).
• In first image (one frame prior) brightness of subject is recognized for each area⁴ and luminance map is generated (2). After ensuring difference in brightness levels between adjacent areas are not excessive ((3) Reduce adjacent exposure discrepancy), exposure conditions are corrected based on information from movement map, and final exposure conditions are specified (4).
• Final exposure conditions (4) are applied to images for corresponding frames.

[via digicame-info]