Background Reconstruction Plugin for The Foundry Nuke

 

Technical requirements

• The Foundry Nuke version: 6.2 or above
• Memory requirements: 2 GB minimum, 8 GB recommended
• Works with: Matting and Stereo Generator
• Main use cases: restoration of background in occlusion areas for high-quality stereo generation

Background Reconstruction

Various methods of background reconstruction are useful in the stereo-generation process. Large parallax values require high-quality restoration of unknown image regions in occlusion zones. The YUVsoft Background Reconstruction plug-in for Nuke offers you the ability to perform quality temporal restoration of backgrounds for moving foreground objects using information from other frames in the sequence. In other words, the plug-in generates a “real” background as it appears in neighboring frames.

Key features of YUVsoft Background Reconstruction:

• Generation of “real” background areas using temporal interpolation methods
• Easy to use; does not require additional manual effort, but instead automatically generates the background and corresponding depth map using just the source 2D video and its depth map in the simplest case
• Adaptive analysis works with complex motion and is not limited to one plane of motion
• Generation of background for multiple objects simultaneously
• Ability to use an incomplete input depth map

For your convenience and to allow you to better control the reconstruction process, YUVsoft Background Reconstruction supports two different modes: one is based entirely on the depth map, and the other uses additional roto-mask input to enable more-precise background generation for a specific object or area.

YUVsoft Background Reconstruction with depth input in use:

 

 

 

In this mode, the plug-in uses the source video and its depth map for each frame and can reconstruct the edges of multiple objects in one pass.

YUVsoft Background Reconstruction with depth and mask input in use:

 

 

 

In the mode that uses the roto mask, the plug-in takes advantage of the additional information to improve the results. Also, it gives you better control of which areas to reconstruct.

 

The YUVsoft Background Reconstruction plug-in for Nuke takes the source video and depth video sequence as input, and it can also use an optional mask of foreground objects for each frame. The output includes the following:

• Restored background without foreground objects for roto-mask mode, or, for depth-map-only mode, restored areas that require background for stereo generation
• Background depth
• Remnant mask
• Initial mask (in case of depth-only mode)

The remnant mask contains areas that were not reconstructed during execution. This result can occur because of insufficient motion in a processed shot/frame range or because the algorithm is uncertain about this particular area for the selected settings. The initial mask shows areas that needed reconstruction as estimated from depth map and in accordance with the required parallax of the resulting stereo. The background and its depth map can be used by YUVsoft Stereo Generator immediately.

 

See also:

• Tutorial on Background Reconstruction – covers the basics of using Background Reconstruction for creating clean plates.
• A sample of the reconstructed background and how converted stereo can be improved with it.
• 2D to S3D conversion process – general paradigm.
• Depth Refinement technology – corrects edge mismatches between the source frames and depth map.
• Stereo Generator – the tool for high quality stereo generation from 2D+Depth video.
• Stereo Generator plugin for Nuke.
• Fuzzy edges processing – handles such complicate details as fur, hair and translucent areas.
• Depth map dilation – helps to eliminate halos and stripes of wrong color around foreground objects on the resulting stereo video.