Paradigm

For 2D to stereo 3D conversion we automate approaches that explicitly use depth maps. For a given 2D video, the user must estimate, paint or somehow acquire a sufficiently accurate depth map—a grayscale picture in which a pixel’s brightness specifies that pixel’s distance from the camera or viewer in the original picture. This correspondence map should be constructed for each frame of the input 2D video. The resulting stereo video is generated from the corresponding depth maps and the original 2D video by shifting each pixel of a given 2D image to the left or to the right depending on the corresponding depth map value, the type of stereo view (left or right) and the generation settings.

Such approaches have proven their practicality and versatility in a number of projects for 2D-to-3D conversion of theatrical movies. Explicit depth maps greatly simplify depth grading and dynamic changes in parallax for creating different releases (general theater, IMAX, Blu-ray Disc) and for providing a better psychovisual experience. Modifying the perceived depth of the scene is easy using depth maps. Conversion quality in this process is more predictable and repeatable than it is for approaches that use manual shifting and painting of objects. On the other hand, the process is simpler and usually more practical than are methods that perform 3D scene reconstruction.

The crucial factors affecting quality are depth map accuracy and stereo generator quality. For the latter, probably the most important step is filling unknown areas that appear after image objects are shifted according to their attributed depths.

We offer automatic depth map creation, editing and stereo generation from 2D+Depth video.

General Flowchart

The key steps of our 2D-to-2D+Depth conversion process are

• Automatically create plausible depth maps for a scene
• Select and edit the best depth maps for a few frames, making these frames key frames; the depth maps for other frames are mostly generated automatically
• Perform rotoscoping (manual selection and tracing) for critical foreground objects whenever necessary
• Use depth propagation to protract semiautomatically generated depth maps of selected frames to entire scene
• Refine depth using special brushes or other manual editing tools, and repeat propagation until the desired quality is achieved; results are evaluated by generating stereo for intermediate depth maps

We can balance the quality and cost of the process by rotoscoping more objects or using automatically estimated depth maps for a larger proportion of frames. Also, we can adjust the correction accuracy for generated depth maps.

Automatic Depth Estimation

We provide several methods for automatic depth map construction:

• Depth from Motion — real depth is obtained from the difference in motion of background and foreground areas in a scene
• Depth Effects — plausible depth is generated on the basis of geometry and the simple assumption that foreground objects are situated in the bottom part of a frame—an assumption that is true for many panoramic shots of nature and city landscapes
• Depth from Focus — plausible depth is generated from edge blur information; useful for obtaining the depth of picture elements with complex fuzzy edges like hairs, fur and foliage

Each method has various settings to adapt to a given situation and maximize the resulting depth map accuracy. By running the tools using different settings, the user is given several competitive results to choose from. Often, merging the different depth maps is beneficial for getting the best result.

Depth Propagation

This tool is very important for automation. It propagates a given depth map from one or two frames to the entire scene or the selected scene fragment. The algorithm tracks objects and background in the video and associates them with areas in given depth maps. If the depth map is specified for the first and last frame of a video fragment, the propagation is bidirectional and accounts for changes in the depth of objects. Thus, users need not manually create a depth map for each frame, as it is usually sufficient to have a few depth frames per shot; depth maps for all frames in the video can be generated automatically from these few frames.

Depth Propagation is very good at generating depth maps for backgrounds. Depending on the speed of motion, motion complexity and quality requirements, it can construct the entire depth map, including important foreground objects.

Learn more about Depth Propagation

Stereo Generation from 2D+Depth

Stereo Generator automatically produces stereo from 2D video by shifting image pixels to the left or right depending on the provided depth map. Unknown, uncovered areas appear afterwards and should be filled with visually appealing patches to minimize any reconstruction artifacts. Two sources provide images for unknown areas:

• Background Reconstruction — the tool uses actual picture details not covered in other video frames
• Interpolation and texture generation technique — the tool generates an artificial image by attempting to minimize the visibility of filled (previously unknown) regions

Besides reconstruction methods, users can regulate the following:

• Parallax — define how much the foreground objects will «protrude»
• Zero Level — place the entire scene «behind» or «in front of» the screen
• Automatic foreground depth map dilation and foreground object expansion — compensate for an inaccurate depth map and decrease the visibility of artifacts in uncovered areas
• Matting — helps to correctly handle transparency on the edges between foreground objects and high-textured background

 

See also:

• Semitransparent edges processing with Stereo Generator
• Depth map dilation – helps to eliminate halos and stripes of wrong color around foreground objects on the resulting stereo video.
• Background Reconstruction – various methods for different types of video.
• Depth Effects usage for scenes with foliage.
• Depth Propagation from a depth key frame created by Depth Effects.
• Depth from Motion plugin for Adobe After Efects.
• Depth from Focus usage for underwater scenes processing.

Contact Information

For information on licensing, contact YUVsoft’s sales staff.