Faster Depth-Map Creation Using Depth Propagation Tool

Depth Propagation (DP) is an essential tool in YUVsoft’s 2D to 3D Suite that is designed to accelerate the conversion process and simplify depth-map compositing.

The “Juneau” case study offers an example application of YUVsoft’s DP tool for 2D-to-stereo-video conversion with minimized manual work. The input data is a complex non-static shot (see the source video). Using DP to process background and foreground objects decreases the required number of roto masks by more than 6 times without noticeably sacrificing quality. Here’s a look at how it was done.

Input data for depth propagation: 100 source frames (source video; see Image 1) 4 key depth frames for background 16 key depth images for foreground objects (1.6 key images per object on average)

Image 1: Source frame #63.

Depth maps must be created separately for the background and foreground:

1. Background: Key depth frames were hand painted for the entire image (see Images 2–5). Background objects can be painted without any rotoscoping, but to create four key depth frames using roto, 184 masks are required.

Four key depth frames for the background:

Image 2: Key depth frame #00.	Image 3: Key depth frame #33.
Image 4: Key depth frame #67.	Image 5: Key depth frame #97.

2. Foreground: Key depth images (see Image 6) were hand painted for the 10 most important objects.

Image 6: Key image example for an object. Contrast is increased for detail visibility. Input for the DP tool included four such key depth images for this object.

Roto masks prevent boundary leakage from the background, but the number of necessary roto masks is lower: practically, just a single mask per object. All together, this example involved creation of 10 isolating roto masks (see Images 7–8) per frame to merge the propagated background and foreground depth.

Image 7: Isolating roto masks for several objects.	Image 8: Isolating roto mask for a single object, frame #63.

The traditional approach requires rotoscoping of 78 masks for each frame to achieve a depth map of the same quality, totaling 7,800 roto masks for 100 source frames.

Note that propagation preserves depth details, and it also takes into account depth changes over time (see Images 9–10)

Picture 9: Propagated depth for a single object that is cut out using the respective isolating mask, frame #63. Contrast is increased for detail visibility	Picture 10: Propagated background depth, frame #63.

The table below compares the amount of work needed for each approach, where each manually drawn object contour for a given frame is termed a “mask”:

Layer	Traditional Approach	DP-Tool Approach
FG	32 * 100 = 3,200 masks	10 * 100 = 1,000 isolating masks; also, 31 masks to make key frames
BG	46 * 100 = 4,600 masks	46 * 4 = 184 masks to make key frames
Total	7,800 masks	1,215 masks

As a result, the DP tool greatly reduces the rotoscoping workload by 6.4 times, even for nontrivial shots, saving important resources: time and money.