Introduction to the Gerbil SDK -- Building, image operations and modules

In this tutorial we cover a selection of key usage patterns of the Gerbil SDK. We develop a small program that outputs the Spectral Angle Mapper (SAM) similarity between each pixel in the image and a reference image region.

How-to use this tutorial

You can go through this document and follow the code samples in your favorite programming environment. The input files to test your program are located in the example folder in the ZIP file below. You should set the working directory of your project to that folder so the program will find the input files. In the code folder you find the complete source code of the tutorial module as a reference.

Tutorial ZIP file

1. Setting up a new module

For this tutorial, we will develop using Gerbil’s powerful build system called Vole. It allows us to develop a cross-platform application using our favorite IDE while concentrating solely on the code itself.

The build system is based on the popular CMake software. It manages dependencies of all modules and resolves used software libraries. Due to the nature of CMake, it can output files for the selected software developing environment, e.g. Makefiles on UNIX or a Visual Studio Solution file on Windows.

We create a new module within the SDK root, which will be automatically found and built. In the first step, our module creates a stand-alone binary.

This is how it fits into the directory tree of the SDK:

Gerbil SDK /
    build (1)
    cmake
    core
    csparse
    doc
    …
    tutorial (2)
    CMakeLists.txt
1 New folder for building
2 New folder for the tutorial module

In the CMake GUI (Windows/OS X), point the source directory to the SDK root, and the output directory to build. Or when developing under Linux:

cd build
cmake ../
ccmake . # for further tuning

Now we can bring our tutorial module to life by adding a source file and corresponding build description (CMakeLists.txt) to the tutorial folder:

mapper.cpp
#include <iostream>

int main()
{
	std::cout << "It works!" << std::endl;
}

As you can see, the program doesn’t do much yet.

CMakeLists.txt
vole_module_name("tutorial")
vole_module_description("Similarity Heatmap using Spectral Angle Mapper (SAM)")
vole_module_variable("Gerbil_Tutorial")

vole_compile_library(
	mapper
)

vole_add_executable("simmap" "mapper")
vole_add_module()

In lines 1-3 we define the module’s name and how it will appear in the CMake configuration. Then in lines 5-7 we denote all source files, whereas the extensions .cpp, .h etc. may be omitted.

In line 9 we create an executable named simmap. On Windows this will automatically be called simmap.exe. The second argument points to the source file that contains the main() method.

Finally the module is composed by calling vole_add_module() in the last line.

Now when re-configuring CMake, our module should be visible via the new config parameter Gerbil_Tutorial, which is enabled by default. When creating a new Solution/Makefile, it will include instructions to build the simmap executable next to the gerbil executable.

The gerbil executable is built by the shell module. To prevent it from building, disable Gerbil_Shell in the CMake configuration. We will use this module later in the tutorial.

Executables are found in build/bin/ or build/bin/Debug/ and build/bin/Release/, depending on the operating system.

2. Basic image operations

Now that our module is up and running, we can start by reading in a multispectral image. We can do so with the multi_img class which is part of the core module.

The multi_img class is our internal representation of a multispectral or hyperspectral image. It provides both band-wise and pixel-wise (interleaved) access to the image data. Band descriptions are stored in the member meta.

We load a multispectral image as well as a mask image. The mask image is a color or grayscale image and describes pixels of interest in the multispectral image. All pixels that are not black in the mask will be considered. The idea is that we want to extract samples (pixels) of a specific material. The mask file can be created by labeling in the Gerbil GUI application.

image.txt (rendered as true-color here) and mask.png

image.txt mask.png

mapper.cpp
#include <multi_img/multi_img.h>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>
#include <exception>

int main()
{
	multi_img image("image.txt");
	cv::Mat1b mask = cv::imread("mask.png", CV_LOAD_IMAGE_GRAYSCALE);
	if (image.empty() || mask.empty())
		throw std::runtime_error("Image file(s) could not be read!");
	if (image.width != mask.cols || image.height != mask.rows)
		throw std::runtime_error("Image and mask geometries do not match!");
}

While the multispectral image is loaded in the multi_img constructor, we use OpenCV directly to load the mask image. Gerbil is built around OpenCV, which means great interoperability between Gerbil methods and OpenCV data structures.

The multi_img constructor only reads files in the custom Gerbil descriptor file format or LAN format. All other file types are read by the powerful imginput module, which we will use later in the tutorial.

We will now extract a representative spectrum of a material by averaging samples which are marked via the mask. To obtain the spectra in question, we use the getSegment() accessor method.

image.rebuildPixels();
auto spectra = image.getSegment(mask);
The rebuildPixels() method pre-caches all pixels in the interleaved format. It is preferred to letting the pixels being built on-demand.
More commonly used accessors to image data are operator[] (single band) and operator() (single pixel). Likewise, there exist setBand(), setPixel() and setSegment() methods.

Now we calculate the average spectrum:

multi_img::Pixel mean(image.size());
for (auto p : spectra) {
	std::transform(mean.begin(), mean.end(), p->begin(), mean.begin(),
	               std::plus<multi_img::Value>());
}
for (auto v : mean) {
	v /= spectra.size();
}
A multi_img::Pixel is a std::vector<multi_img::Value>, whereas multi_img::Value is float by default.

However, as seen in this example, using standard C++ can get a bit cumbersome for performing these operations. An alternative is to use OpenCV. Most OpenCV operations can be performed on STL vectors directly. For others, we can construct an OpenCV matrix header around them. Instead of the code above, we write:

multi_img::Pixel mean(image.size());
for (auto p : spectra) {
	cv::add(mean, *p, mean);
}
cv::Mat1f meanMat(mean);
meanMat /= spectra.size();

Note that meanMat points to the same data as mean. We do not need to convert back but can continue on working with the vector, for example print it to the command line:

std::cout << "Reference spectrum:";
for (auto v : mean) {
	std::cout << "  " << v;
}
std::cout << std::endl;

In the case of the data provided within this tutorial, we observe the following output:

Reference spectrum:  14.2145  8.19017  7.29627  3.83346  3.53786  2.0817  1.51586  1.35737  1.32929  1.13829  0.962368  0.895458  0.966911  0.97876  0.931279  1.00898  1.75047  5.19298  15.3666  40.4076  80.8653  104.896  96.6044  84.8207  79.8035  69.0159  59.7343  51.8378  44.5212  40.7069  41.9881

When looking at the output, you will see values in the range 0—​255, despite 16 bit input data. When reading image data, Gerbil scales the values to this range, regardless of original bitdepth (accuracy is not lost due to the use of a floating-point datatype). This is for convenience as most image processing algorithms work comfortably in this data range. This behavior can be changed by setting minval and/or maxval on the object before loading.

3. Usage of other modules

We now have all needed data at hands to create our similarity map: The image pixels as well as a reference spectrum/pixel to compare against. However we do not implement the spectral angle mapper ourselves. Instead we employ an existing Gerbil module that comes with it: similarity_measures.

3.1. Similarity Measures module

Three simple steps provide us with the module functionality:

  1. Add it to our build configuration:

    CMakeLists.txt (excerpt)
    …
vole_module_variable("Gerbil_Tutorial")

vole_add_required_modules(similarity_measures)

vole_compile_library(
…

  2. Include header from the module:

    mapper.cpp (excerpt)
    #include <multi_img/multi_img.h>
#include <sm_factory.h>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>
#include <exception>

namespace similarity_measures = sm;
…
    By convention, all modules operate within their own namespace, which resembles the module name.

  3. Create object for similarity measurement:

    auto simfun = sm::SMFactory<multi_img::Value>::spawn(sm::MOD_SPEC_ANGLE);

It is now a piece of cake to calculate similarities across the image in a loop:

cv::Mat1f map(image.height, image.width);
for (int y = 0; y < map.rows; ++y) {
	for (int x = 0; x < map.cols; ++x) {
		map(y,x) = -simfun->getSimilarity(image(y,x), mean);
	}
}

Note that we negate the value due to a high similarity resulting in a low Spectral Angle value. However we like to display high similarity with a high intensity (white), while low similarities, which correspond to high values for the Spectral Angle, should be displayed as black. Finally, we normalize the map range and write the image:

cv::normalize(map, map, 0, 255, cv::NORM_MINMAX);
cv::imwrite("output.png", map);

Our program is now working! This is the output:

output.png

input.txt

3.2. Advanced image input

However, this will not run on image inputs of other formats like ENVI. So we bring in another very helpful module called imginput. This module does not only load images, it can also manipulate them, e.g. reduce bands, crop, transform them to other feature spaces. We again follow three simple steps to enhance the image loading in our program:

  1. Add it to our build configuration:

    CMakeLists.txt (excerpt)
    …
vole_add_required_modules(similarity_measures imginput)
…

  2. Include header from the module:

    mapper.cpp (excerpt)
    …
#include <sm_factory.h>
#include <imginput.h>
…

  3. Load image using the module:

    auto image = imginput::ImgInput::load("image.txt");
    imginput returns a shared pointer to the image. This changes how we access the image’s methods and members (image->… instead of image.…).

4. Parallelization

Next to OpenCV, the Gerbil SDK incorporates Intel Thread Building Blocks (TBB) to allow easy speedup of computation via dynamic parallelization. With only a few changes we can use all processor cores for our similarity calculation.

  1. Add requirement to our build configuration:

    CMakeLists.txt (excerpt)
    …
vole_add_required_(similarity_measures imginput)
vole_add_required_dependencies(TBB)
…

  2. Include header from the module:

    mapper.cpp (excerpt)
    …
#include <tbb/blocked_range2d.h>
#include <tbb/parallel_for.h>
…

  3. Rewrite loop using TBB:

    tbb::parallel_for(tbb::blocked_range2d<int>(0, map.rows, 0, map.cols),
	                  [&](tbb::blocked_range2d<int> r) {
		for (int y = r.rows().begin(); y != r.rows().end(); ++y) {
			for (int x = r.cols().begin(); x != r.cols().end(); ++x) {
				map(y,x) = -simfun->getSimilarity((*image)(y,x), mean);
			}
		}
	});

Now our calculation will automatically be distributed to all available CPU cores/threads in the system by TBB. We do not need to take care about thread creation or synchronization.

5. Shell/Config interface

At this stage, we have a very simple program that provides us with the desired output in a fast manner. However, we are stuck with one single similarity measure and fixed input/output file names. Also, the image manipulation capabilities of the imginput module are not exposed.

The Gerbil SDK comes with a convenient and flexible system for interfacing on the command line and setting of algorithmic and other parameters. Instead of building a custom executable, we will now employ the shell module to access this functionality.

5.1. Build configuration

We add a command instead of an executable in the CMake configuration:

CMakeLists.txt (excerpt)
…
vole_compile_library(
	mapper
	mapper_config
)

vole_add_command("mapper" "mapper.h" "mapper::Mapper")

vole_add_module()

The command is provided by a class Mapper in the namespace mapper (called after our module), which we will add to our source code. Note that we also we introduce a MapperConfig class in mapper_config.h, mapper_config.cpp.

5.2. MapperConfig class

We create a class (struct for easy access of configuration variables) derived from Config that also has two members derived from Config: The configurations of the similarity measures and imginput modules.

mapper_config.h
#ifndef MAPPER_CONFIG_H
#define MAPPER_CONFIG_H

#include <vole_config.h>
#include <imginput_config.h>
#include <sm_config.h>

namespace mapper {

struct MapperConfig : public Config
{
	MapperConfig(const std::string& prefix = std::string());

	std::string mask;
	std::string output;

	imginput::ImgInputConfig input;
	similarity_measures::SMConfig similarity;
};

}

#endif // MAPPER_CONFIG_H
The prefix argument to the constructor is used to chain configuration objects — exactly what we are also doing here.

In the source file we need to set defaults and initialize command line argument parsing (done by shell module):

mapper_config.cpp
#include "mapper_config.h"

namespace mapper {

// descriptions of configuration options
namespace desc { (1)
DESC_OPT(mask, "Mask file (black means excluded, else included)")
DESC_OPT(output, "Output image file")
}

MapperConfig::MapperConfig(const std::string &p)
    : Config(p),
      mask("mask.png"),
      output("output.png"),
      input(prefix + "input"), (2)
      similarity(prefix + "similarity") (2)
{
	options.add_options() (3)
	        BOOST_OPT(mask)
	        BOOST_OPT(output)
	        ;
	options.add(input.options);
	options.add(similarity.options);
}

}
1 Descriptions used for command line help
2 Prefixes for command line arguments
3 Add member variables to command line argument parsing

5.3. Mapper command class

As we are now creating a class, we need a header file. Our Mapper header is pretty basic though:

mapper.h
#ifndef MAPPER_H
#define MAPPER_H

#include "mapper_config.h"
#include <command.h>

namespace mapper {

class Mapper : public shell::Command {
public:
	Mapper() : Command("mapper", config) {}

	int execute();

	void printShortHelp() const;
	void printHelp() const;

public:
	MapperConfig config;
};

}

#endif // MAPPER_H

We implement a derivation from Command, which is provided by the shell module. The main() method is replaced by execute(). Through the Command class our config member is populated with the parameters set by the user.

And this is our final implementation:

mapper.cpp
#include "mapper.h"
#include <multi_img/multi_img.h>
#include <sm_factory.h>
#include <imginput.h>
#include <opencv2/highgui/highgui.hpp>
#include <tbb/blocked_range2d.h>
#include <tbb/parallel_for.h>
#include <iostream>
#include <exception>

namespace sm = similarity_measures;
namespace mapper {

int Mapper::execute()
{
	auto image = imginput::ImgInput(config.input).execute(); (1)
	cv::Mat1b mask = cv::imread(config.mask, CV_LOAD_IMAGE_GRAYSCALE); (2)
	if (image->empty() || mask.empty())
		throw std::runtime_error("Image file(s) could not be read!");
	if (image->width != mask.cols || image->height != mask.rows)
		throw std::runtime_error("Image and mask geometries do not match!");

	image->rebuildPixels();
	auto spectra = image->getSegment(mask);
	multi_img::Pixel mean(image->size());
	for (auto p : spectra) {
		cv::add(mean, *p, mean);
	}
	cv::Mat1f meanMat(mean);
	meanMat /= spectra.size();

	if (config.verbosity > 1) { (3)
		std::cout << "Reference spectrum: " << std::endl;
		for (auto v : mean) {
			std::cout << v << "  ";
		}
		std::cout << std::endl;
	}

	auto simfun = sm::SMFactory<multi_img::Value>::spawn(config.similarity); (4)
	cv::Mat1f map(image->height, image->width);
	tbb::parallel_for(tbb::blocked_range2d<int>(0, map.rows, 0, map.cols),
		                  [&](tbb::blocked_range2d<int> r) {
			for (int y = r.rows().begin(); y != r.rows().end(); ++y) {
				for (int x = r.cols().begin(); x != r.cols().end(); ++x) {
					map(y,x) = -simfun->getSimilarity((*image)(y,x), mean);
				}
			}
		});

	cv::normalize(map, map, 0, 255, cv::NORM_MINMAX);
	cv::imwrite(config.output, map); (2)

	return 0; // success
}

void Mapper::printShortHelp() const { (5)
	std::cout << "Pixel-wise similarity mapper based on reference mask." << std::endl;
}

void Mapper::printHelp() const { (5)
	std::cout << "Calculates the similarity of each pixel with the average\n"
	             "of the reference region in the same image.";
	std::cout << std::endl;
}

}
1 We pass the ImgInputConfig when loading the image
2 We use the standard configuration parameter verbosity (defaults to 1)
3 We pass the SMConfig when creating the similarity measure
4 We use the filename parameters for reading and writing
5 These methods provide helpful output when the shell is called with an -H or --help argument

5.4. Command line interface

Our command is now part of the gerbil executable. This is the output when calling gerbil in a command line shell:

# ./gerbil
            .s_,  ._ssaoXXXZqaa,.
           _mmZmaoZXSnooooXXZmWWma.
         _mQWmZZoo2ooonnnnnXZ#mWWBma.
        <QWQB#ZXnnoo2onvnnnXX#mBmWWWm,.
       =WWmW#ZenvnoonI|+|+{nX##WBWBWBWWga,
       ???Y?"---<vIl|i=====I3X#mWmBm###:?Wc
             )nnii"----   ---*YX##!~-   .mk
              -           :iiv?!~-   . .j#~
                                     _saZ!`
  G E R B I L        -{I;_asssas%IY!!^


Usage: bin/gerbil [--help] command [configfile] [options ...]

All options can also be given in the specified config file.
Options given in the command line will overwrite options from the file.

Available commands:
  edge_detect:  Edge detection using a Self-Organizing Map (SOM)
  classifier:   Classification via kNN or SOM
  felzenszwalb: Superpixel Segmentation by Felzenszwalb, Huttenlocher
  graphseg:     Graph Cut / Power Watershed segmentation by Grady
  mapper:       Pixel-wise similarity mapper based on reference mask. (1)
  meanshift:    Fast adaptive mean shift segmentation by Georgescu
  meanshiftsom: Fast adaptive mean shift segmentation on SOM
  meanshiftsp:  Fast adaptive mean shift segmentation on superpixels
  rgb:  RGB image creation (true-color or false-color)
1 Our module is here!

To run our method, we call

# ./gerbil mapper

A help screen provides us with all parameters:

# ./gerbil mapper -H
-------------------------------------------------------------------------------
Calculates the similarity of each pixel with the average
of the reference region in the same image.
-------------------------------------------------------------------------------
Options for mapper:
  -V [ --verbose ] arg (=1)             Verbosity Level: 0 = silent, 1 =
                                        normal, 2 = much output, 3 = insane
  --mask arg (=mask.png)                Mask file (black means excluded, else
                                        included)
  --output arg (=output.png)            Output image file
--input.verbose arg (=1)              Verbosity Level: 0 = silent, 1 =
                                      normal, 2 = much output, 3 = insane
--input.file arg                      Input image filename
--input.roi arg                       Region of Interest, specify in the form
                                      x:y:w:h to apply
--input.normalize                     Normalize vector magnitudes
--input.gradient                      Transform to spectral gradient
--input.bands arg (=0)                Reduce number of bands by linear
                                      interpolation (if >0)
--input.bandlow arg (=0)              Select bands and use band index as
                                      lower bound (if >0)
--input.bandhigh arg (=0)             Select bands and use band index as
                                      upper bound (if >0)
--input.removeIllum arg (=0)          Remove black body illuminant specified
                                      in Kelvin (if >0)
--input.addIllum arg (=0)             Add black body illuminant specified in
                                      Kelvin (if >0)
--similarity.verbose arg (=1)         Verbosity Level: 0 = silent, 1 =
                                      normal, 2 = much output, 3 = insane
--similarity.measure arg (=EUCLIDEAN) Similarity measurement function.
                                      Available are: MANHATTAN, EUCLIDEAN,
                                      CHEBYSHEV, SPECTRAL_ANGLE,
                                      SPEC_INF_DIV, SIDSAM1, SIDSAM2, NORM_L2

With this command we obtain the same results as with our executable before:

# ./gerbil mapper --input.file image.txt --similarity.measure SPECTRAL_ANGLE --mask mask.png --output output.png

In the help screen we see that we now can try various different similarity measures, band ranges in the input image or different image representations (spectral gradient and L2-normalized feature space).

When a static module configuration is desired, the corresponding Config object can also be created and populated in the program code without a user interface.

6. Wrap up

We now have a new, fully-fledged module in the Gerbil SDK at hands. Trying out several different similarity measures is a piece of cake:

# ./gerbil mapper --input.file image.txt --similarity.measure CHEBYSHEV --output linf.png
# ./gerbil mapper --input.file image.txt --input.normalize --similarity.measure CHEBYSHEV --output linf-norm.png
# ./gerbil mapper --input.file image.txt --similarity.measure SPECTRAL_ANGLE --output sam.png
# ./gerbil mapper --input.file image.txt --similarity.measure SPEC_INF_DIV --output sid.png

These are our output files:

l2.png, l2-grad.png, sam.png, and sid.png

image.txt image.txt image.txt image.txt

As we can see, we can play with the parameters to find the best discrimination for our application. We could then use a simple thresholding or more complex means to separate a material of interest from the scene.

Conclusion

In this tutorial, we got an insight in the basic usage of the Gerbil SDK. We now understand how to use the SDK as a basis for writing new programs either by using the shell interface or by creating a standalone executable. We learned how to incorporate the functionality of existing SDK modules and libraries bundled with the Gerbil SDK, e.g. for easy parallelization of our code. We should now be able to also work with other modules from the SDK, e.g. clustering or classification.