Using overlays to aggregate information

The list of overlays

Most tabs in ilastik show the list of currently configured overlays in the lower right part of the window.

The concept of overlays can be compared to stacking transparencies made for an overhead projector as you would do in GIMP or other professional image editing applications. Beginning with the last entry of the list, overlays are stacked on top of each other with the first item in the list finally being on top. ilastik then displays the stack of "transparencies" in its image view (2D) or slice views (3D to 5D) at the center of the tab.

In the above example, the user-drawn labels are displayed on top of the raw data and a new layer, Gaussian smoothing, was added and switched off (see below).

To export "screenshots" of the current image view (2D) or all slice views (3D) as they appear inside ilastik, use the Export View button. In the 3D case, entire image stacks in x, y and z-direction can be written.

Manipulating overlays

In this section, we discuss manipulating overlays as shown in the overlay list. How to add overlays (either from a pool of pre-configured overlays, from external sources or by applying a transformation to an existing overlay) is discussed in adding and removing overlays.
  • Enabling.
    The checkbox next to the overlay's name indicates whether the overlay is considered as part of the overlay stack when computing the image displayed in the image view(s) or not. A checked overlay has 100% opacity, an unchecked overlay is fully transparent.
  • Reordering.
    Overlays can be reordered by first selecting an overlay in the list and then using the up/down arrow buttons to the right of the list of overlays.
  • Changing opacity.
    The amount of transparency (opposite: opacity) can be configured for each overlay such that it is possible to have the information contained in lower parts of the overlay stack to "shimmer through". To change opacity double-click an a target overlay in the list; the dialog below appears where the opacity can be changed from fully transparent (left) to fully opaque (right). The same functionality can be accessed through the context menu that can be opened with right click on the overlay. ("Change opacity")

    Note: Some pre-configured overlays in ilastik already have an alpha channel which gives a transparency value for each pixel. An example is the "Labels" overlay which is transparent wherever there is no label. The opacity discussed here is a single value for the entire overlay and acts in addition to the overlay's alpha channel.

  • Changing color.
    When an overlay indicates a single scalar per pixel, such as the output of a smoothing filter, different tint colors can be chosen. By default, ilastik will create red tinted overlays for all computed image features. Using the context menu "Change color" and selecting black will result in a more conventional grayscale image representing that image feature. Form more details on how to display an image feature as overlay see below adding and removing overlays.

Creating New Overlays

In ilastik 0.5, overlays are a powerful tool to display features and results. Whenever you want to get insight into a certain feature response, smooth the borders of the current segmentation, compare ilastik segmentation with other external results or simply save the image which is currently displayed by the viewer (possibly for the entire volume): then you need overlays.

Adding and removing overlays

The three buttons: plus, minus and new allow addition and removal of overlays. These buttons are located on the right panel below the overlay list.

  • Add a pre-configured overlay
    The plus buttons brings up a dialog (below) to select from a number of pre-configured overlays, which are organized by ilastik in a hierarchical pool called the overlay pool. Each selected feature in the feature tab, for example, adds a number of overlays to the overlay pool. Overlays which are selected in this dialog are added to the overlay list; and can be displayed by the central viewer.
  • Removing an overlay
    The minus button removes the currently selected overlay from the overlay list. However, it is still present in the "overlay pool" and can be added again. This also holds for custom overlays (see below).
  • Add custom overlays
    The new button brings up a dialog to create your own overlays. This can be done either modifying an existing overlay (for example thresholding the probability map) or by importing external data. The new buttons open the add overlay dialog.

Add overlay dialog

The add overlay dialog is invoked by clicking the new overlay button beneath the overlay list .

Several available options are listed on the left of the dialog, with a short description of use case showed on the lower right. New overlays can be created in two ways:

  1. by applying a transformation to existing data
    (thresholding overlay, normalize overlay)
  2. by importing external data
    (add stack overlay (3D), add overlay from h5 file)

Normalize overlay

This operation applies the normalization to a range (min, max) to an existing overlay which is useful, for example, to display 16 bit raw data.
First, an existing overlay is selected, in the example below is the raw data.

Then, the normalization range can be configured in the new opening dialog:

Finally, the new overlay's name is configured in another dialog:

Here, the notation of names seperated by forward-slashes is used to indicate the overlay's place in the hierarchy of overlays of the overlay pool. The new overlay is added to the overlay list, and the normalization range can be changed by using the context menu and Configure.

Thresholding overlay

This option is useful to balance the amount of predicted pixels of a certain class in a multiclass prediction and/or smooth the probability produced by the classification module. Suppose we have made a prediction for four classes in the classification tab. A result may look like this:

To produce a segmentation by thresholding, we will add an overlay via the new overlay button and the Thresholding Overlay option, which brings up the following dialog:

First, we need to select a single overlay as background by clicking the Select Background button. Everything that is assigned to the background in the thresholding will be transparent in the final overlay. In this example, we select "Label 3" which corresponds to the blue colored prediction.
Now, we select the overlays corresponding to the prediction of the other classes as foreground using Select Foreground
If the smooth option is enabled, a Gaussian smoothing filter with the given sigma will be run over the input data before thresholding, giving a less-noisy result.
In this dialog, due to a bug, the sliders do not work. First add the overlay with OK, give it a name, and then access the dialog again through the context menu and Configure.
Now, use the sliders to adjust the thresholds. The result will be shown in real-time in the image view (2D) or the three slice views (3D).
The results of the above operations looks like below, where we can note that the yellow class was not selected, the blue class is now transparent and the probability for red and green class has been smoothed.

Add stack overlay (3D)

This option is similar for 2D and 3D, hence, we demonstrate it here in 3D only. This option is particularly useful to compare results of an external processing with the probability maps obtained in ilastik. To load external volume data from an image stack as an overlay, use the following dialog (similar to the dialog in loading a z-stack)

If the dimensions do not match, a message will be printed in the shell output.

Add overlay from h5 file

An overlay that has been exported previously as an hdf5 file can be added through this dialog. All options in the Color selection group are filled automatically from the file and need not be touched.

If you want to load your own data, write a file according to the overlay hdf5 file format. It is sufficient to have the correct dataset and shape, additional attributes are not necesary. In this case, the Color selection options can be used.

Exporting overlays

To export an overlay, use the context menu of any overlay in the overlay list and choose Export overlay
  • Exporting as a stack of images
    To export the overlay as a stack of images (in z slicing direction), choose either png or tif from the combobox.
    The exported file(s) will be written to "Path" and will have the name prefix_z%05d.extension.
    In our example above, the generated image stack will consist of the files /tmp/slice_z00000.png, /tmp/slice_z00001.png etc.
  • Exporting as a single hdf5 file
    Select h5 from the combobox. The file will be written to the directory "Path" and will have the name prefix.h5. To learn more about the data format, see overlay hdf5 file format below.

Overlay data format

When exporting an overlay as an hdf5 file with ilastik, it has the following format:
  • the overlay's data is stored in the hdf5 group volume/data in ilastik's canonical 5D coordinate system (the datasets's shape is 5D, the order of axes is time, x, y, z, channel).
  • the overlay's tint color, color table, min and max value and several other properties are stored, if applicable, as attributes of the volume/data dataset.