Another pretty interesting patent that shows how much Canon is investing in research. Canon patent application US20180077345 discusses a predictive camera control system that takes viewers interests into account, for instance during a sport event.

Such a system is capable to recognise where people in a stadium is looking, what captures their interest most at a given time, and then can adjust its cameras to point in the same direction. A system that learns from user behaviour what gets the most attention by the viewers. This AI powered system (using neural networks) tracks the eyes of the viewers to learn where they are looking at, and is able to recognise which player attracts most looks from viewers. The system’s cameras are then set to capture what captures the interest of the crowd. I guess this may change the rules for future coverage of big sport events (but the system can virtually be used also for other purposes).

From the patent literature:

A computer-implemented method and system of selecting a camera angle is described. The method comprises determining a visual fixation point of a viewer of a scene using eye gaze data from an eye gaze tracking device; detecting, from the eye gaze data, one or more saccades from the visual fixation point of the viewer, the one or more saccades indicating a one or more regions of future interest to the viewer; selecting, based on the detected one or more saccades, a region of the scene; and selecting a camera angle of a camera, the camera capturing video data of the selected region using the selected angle.

[…] A system, comprising: an eye gaze tracking device for detecting eye gaze data of a viewer of a scene; a multi-camera system configured to capture video data of the scene; a memory for storing data and a computer readable medium; and a processor coupled to the memory for executing a computer program, the program having instructions for: detecting, using the eye gaze tracking data, a visual fixation point of the viewer and one or more saccades of the viewer relative to the visual fixation point; determining an object of interest in the scene based on at least the detected one or more saccades of the viewer, the object of interest being determined to have increasing relevance to the viewer of the scene; and selecting a camera of the multi-camera system, the selected camera having a field of view including the determined object of interest in the scene, the second camera capturing video data of the determined object of interest.

More Canon patents.