| Customize Help

Generating an index map from a scene


From a set of two-dimensional images of the same scene taken at different focus distances, the MIL Registration module can generate an image that conveys depth information. Essentially, each pixel in the resulting image corresponds to the index of the image in the set where the corresponding pixel appears most in focus. The resulting image is referred to as an index map and the operation is referred to as a depth-from-focus registration operation. This operation uses multiple images of the same scene, taken at different focus distances, and calculates the index map, confidence map, and intensity map of the scene.

The depth-from-focus registration operation works by considering the sharp portions in the input images. Then, for each pixel, it computes in which image that pixel is found most in focus. It replaces the corresponding pixel entry with the index of the image. You can also apply a regularization operation which ensures that neighboring index map pixels are coherent in your resulting index map, if regularization is enabled. The above image shows a set of source images of the same scene, taken with different focus distances. The source images each contain a portion of the scene in focus; after the depth-from-focus registration operation, the index map for the object is generated based on the best focus images for the object.

In a depth-from-focus registration operation, the order of the source images is important; the source images must either be in an order in which objects appear out-of-focus and become in focus, or in which objects appear in focus and become out-of-focus.

In addition to generating the index map of a scene, you can also use a depth-from-focus registration operation to calculate a confidence map and/or intensity map of the scene. The confidence map indicates how reliable the index map is for a particular pixel; the higher the grayscale value of a pixel in the confidence map, the more confident you can be that the corresponding pixel in the index map is correct. The intensity map represents the scene in focus; each pixel in the intensity map is set to the intensity of the corresponding pixel in the image where that pixel is found to be most in focus. The following image demonstrates the confidence map and intensity map generated from a set of source images using a depth-from-focus registration operation.

Steps to performing a depth-from-focus registration

The following steps provide a basic methodology for performing a depth-from-focus operation on multiple images:

  1. Allocate a depth-from-focus registration context, using MregAlloc() with M_DEPTH_FROM_FOCUS.

  2. If necessary, allocate a depth-from-focus registration result buffer to store the results of the registration operation, using MregAllocResult() with M_DEPTH_FROM_FOCUS_RESULT. You don't need to allocate a result buffer if you intend on passing an image buffer to MregCalculate().

  3. If necessary, specify the control settings for your depth-from-focus registration operation using MregControl().

  4. Set up an array with the identifiers of image buffers that hold the input images. All the images should be of the same scene, all taken with the same camera settings except focus, and in buffers of the same size, type, and with a single band.

  5. If necessary, allocate an image buffer to hold the depth-from-focus index map image (alternatively, the index map image can be stored in the depth-from-focus result buffer). The image buffer should be of the same size, type, and number of bands as the input images.

  6. Perform the depth-from-focus operation, using MregCalculate(). You can pass and process the images in batches using MregCalculate() with M_ACCUMULATE_AND_COMPUTE, or all at once using M_COMPUTE.

  7. If you passed a result buffer to MregCalculate(), retrieve the required results from the result buffer, using MregGetResult().

  8. If you passed a result buffer to MregCalculate(), you can draw the index map image from the result buffer into an image buffer using MregDraw() with M_DRAW_DEPTH_INDEX_MAP. You can use MregGetResult() to establish the size and type of the image buffer required to draw the depth-from-focus index map image. In addition, you can use MregDraw() with M_DRAW_DEPTH_CONFIDENCE_MAP to draw the depth confidence map, or with M_DRAW_DEPTH_INTENSITY_MAP to draw the depth intensity map, if those settings were previously enabled using MregControl().

  9. If necessary, save your registration context, using MregSave() or MregStream().

  10. You can clear the result buffer to perform additional operations, using MregCalculate() with M_RESET.

  11. Free your allocated registration objects, using MregFree().

When using MregCalculate() with M_ACCUMULATE_AND_COMPUTE, you can repeat steps 6 through 8 as many times as are required.

Customizing your depth-from-focus context

You can customize a depth-from-focus registration operation using MregControl().

The coherency of neighboring pixels in an index map can be important, to ensure the accuracy of the generated index map. You can apply a regularization operation to ensure that neighboring index map pixels are coherent. To do so, you must enable regularization and select the mode in which it should be performed using MregControl() with M_REGULARIZATION_MODE.

The regularization mode controls how the registration operation treats the smoothing of pixels from the source images, when generating the index map, in terms of their neighbors. By default, the registration operation will establish pixels in the index map using the average dominance of the neighboring pixels.

You can also use an adaptive smoothing regularization mode. This mode uses the local geometry of the content of the images. In this case, you must use MregControl() with M_ADAPTIVE_INTENSITY_DELTA to identify the maximum variation in pixel values for objects in the image. You must also use MregControl() to specify an M_ADAPTIVE_SMOOTHING value, to control the threshold used to perform the smoothing. Alternatively, you can use MregControl() to set M_REGULARIZATION_MODE to M_DISABLE to disable regularization, and use the raw index map values. The following image illustrates the different regularization modes that can be used to generate index maps.

The specified regularization mode affects the processing time of the depth-from-focus registration operation. The M_ADAPTIVE and M_AVERAGE regularization modes use the values of neighboring pixels to compute the new value of a given pixel in the index map. The efficiency of these modes depends on the M_REGULARIZATION_SIZE control type, which controls the size of the neighborhood over which regularization takes place.

The efficiency of a depth-from-focus registration operation also depends on the M_FOCUS_DEPTH_SIZE control type, which specifies the range of images in which an object in the image is in focus; that is, the number of images in which an object in the images goes from out of focus, to in focus, to out of focus again. A larger focus depth size indicates a more precise operation, and a longer processing time.

You can reduce processing time by disabling regularization (M_REGULARIZATION_MODE set to M_DISABLE); however, this can potentially reduce the accuracy of the operation, since nothing will be done to ensure coherence between neighboring pixels in the index map.

Note that since the input images must be of the same scene, it is assumed that the images completely overlap. Consequently, you do not set rough locations with MregSetLocation().

Computing a depth-from-focus registration

To calculate the index map from images of a scene at different focus depths, use MregCalculate(). The calculation can be done in one operation, or in stages; regardless of the method of calculation, an index map is generated after each call to MregCalculate().

To calculate the index map in stages, call MregCalculate() with M_ACCUMULATE_AND_COMPUTE one or more times, passing a new array of images in each call. In this case, M_ACCUMULATE_AND_COMPUTE computes the depth-from-focus index map using this new information and all of the information previously processed and stored in the depth-from-focus registration result buffer. This kind of operation is useful if you are grabbing and don't have all the source images available at once.

To calculate the index map in one operation, call MregCalculate() with M_COMPUTE, and pass the array of images required to generate the index map. Subsequent calls to MregCalculate() with M_COMPUTE will clear the depth-from-focus registration result buffer before storing new results.

When generating the index map in stages, you can only store the results in a depth-from-focus registration result buffer; however, when generating the index map in one operation, you can store the index map directly in an image buffer or in a result buffer.

When results are stored in a depth-from-focus result buffer, the index map can be drawn into an image buffer using MregDraw() with M_DRAW_DEPTH_INDEX_MAP. Whenever you draw the index map into an image, care must be taken so that the image in which the index map is drawn, is big enough to hold all index values; for instances, an 8-bit image cannot be used to store the index map when the depth-from-focus operation uses 300 images, since each source image requires a unique index in the index map.

In addition to generating an index map for a set of images of a scene, MregCalculate() can also generate a confidence map and the intensity map for that set of images. Use MregControl() with M_CONFIDENCE_MAP or M_INTENSITY_MAP to control whether these maps will be computed. When you are computing the confidence map or intensity map for a scene, you must use a depth-from-focus registration result buffer to store the results. You can then use MregDraw() to retrieve the resulting confidence map or intensity map image. To do so, you can use MregDraw() with M_DRAW_DEPTH_CONFIDENCE_MAP or M_DRAW_DEPTH_INTENSITY_MAP, respectively, and the identifier of an image buffer.

The following diagram shows the ways to use MregCalculate() to perform a depth-from-focus registration operation.

Retrieving and using depth-from-focus results

If MregCalculate() used a result buffer as the destination, the index map image is stored in the result buffer along with some results and some information about the source images. You can use MregGetResult() with M_STATUS to check if the index map image is computed. If MregGetResult() returns M_COMPLETE, the preprocessing information and the index map image are available.

Many of the results that you can retrieve using MregGetResult() depend on their availability within the result buffer. For these result types, you should add the combination constant M_AVAILABLE when calling MregGetResult(). If you retrieve M_TRUE, the information is available and can be retrieved using another call to MregGetResult() without M_AVAILABLE. If the function returns M_FALSE, the requested information is not available, and calling MregGetResult() without M_AVAILABLE will cause an error.

Depth from focus example

The depth from focus example DepthFromFocus.cpp illustrates how the operation can be used to combine multiple images taken at different focus distances to obtain a resulting index map.

depthfromfocus.cpp