1. Supervolume Clustering: A Segmentation Method from Image

A supervoxel is a set whose elements are "bodies". Similar to the volume in voxel filters, they are essentially small squares. Unlike all the previous segmentation methods, the purpose of hypervolume clustering is not to segment a specific object. It implements over segmentation of point clouds, transforms scene point clouds into many small pieces, and studies the relationship between each small piece. The idea of merging smaller units has been around for some years. In image segmentation, pixel clustering is used to form super-pixels, and understanding images by super-pixel relationship has been widely studied. Essentially, this method is a summary of parts. The parts with similar texture, material and color will be automatically segmented into one block, which is conducive to the subsequent recognition work. For example, human recognition, if the hair, face, limbs, trunk can be separated, it will be better to identify the various postures, gender of people.

Point clouds are different from images in that they do not have pixel adjacency. Therefore, before Super-volume clustering, octree must be used to partition point clouds to obtain the adjacency relationship between different point clusters. There are many adjacency relationships with image similar point clouds, such as face adjacency, line adjacency and point adjacency. The specific explanations are as follows:

Point cloud segmentation based on hypervolume clustering uses point adjacency (blue) as adjacency criterion.

2. Implementation steps of hypervolume clustering

A simple example is given to illustrate the process and crystallization of superbody clustering. But it is not the crystallization of water into ice, but the polycrystalline nucleus crystallization in the supersaturated state of salt solution. All seeds begin to grow at the same time, eventually filling the whole space, so that the material has a crystal structure. Superbody clustering is actually a special region growing algorithm. Unlike unlimited growth, superbody clustering first requires regular distribution of regions to grow "nuclei". Nuclei are actually uniformly distributed in space and Rseed is specified. Then specify the particle distance (Rvoxel). Minimum grain size (MOV) is designated. Excessively small grains need to be incorporated into the nearest large grains. The relationship is shown as follows:

With the grain size and crystallization range, we can divide the whole space only by controlling the crystallization process. The essence of the crystallization process is to constantly absorb similar particles (octave space). Similarity is a relatively vague concept. There are the following formulas for similar definitions:

The Dc in the formula represents the difference in color, the Dn represents the difference on the normal line, and the Ds represents the difference in the distance between points. w * denotes a series of weights. Used to control crystalline shape. Looking for a circle around the nucleus, the D-smallest voxel is considered the next "Party member to be developed". It should be noted that the crystallization process does not consist of one nucleus growing to the next. Secondly, all nuclei begin to grow at the same time. The order of growth is as follows:

Next, all nuclei continue to compete fairly and develop a second "Party member" so as to recycle and eventually all crystals should grow almost simultaneously. The entire point cloud is also separated by the lattice. It also ensures that the particles in a crystal package are similar.

#include <pcl/console/parse.h>
#include <pcl/point_cloud.h>
#include <pcl/point_types.h>
#include <pcl/io/pcd_io.h>
#include <pcl/visualization/pcl_visualizer.h>
#include <pcl/segmentation/supervoxel_clustering.h>

//VTK include needed for drawing graph lines
#include <vtkPolyLine.h>

// Types
typedef pcl::PointXYZRGBA PointT;
typedef pcl::PointCloud<PointT> PointCloudT;
typedef pcl::PointNormal PointNT;
typedef pcl::PointCloud<PointNT> PointNCloudT;
typedef pcl::PointXYZL PointLT;
typedef pcl::PointCloud<PointLT> PointLCloudT;

void addSupervoxelConnectionsToViewer(PointT &supervoxel_center,
	PointCloudT &adjacent_supervoxel_centers,
	std::string supervoxel_name,
	boost::shared_ptr<pcl::visualization::PCLVisualizer> & viewer);


int
main(int argc, char ** argv)
{
	if (argc < 2)
	{
		pcl::console::print_error("Syntax is: %s <pcd-file> \n "
			"--NT Dsables the single cloud transform \n"
			"-v <voxel resolution>\n-s <seed resolution>\n"
			"-c <color weight> \n-z <spatial weight> \n"
			"-n <normal_weight>\n", argv[0]);
		return (1);
	}


	PointCloudT::Ptr cloud = boost::shared_ptr <PointCloudT>(new PointCloudT());
	pcl::console::print_highlight("Loading point cloud...\n");
	if (pcl::io::loadPCDFile<PointT>(argv[1], *cloud))
	{
		pcl::console::print_error("Error loading cloud file!\n");
		return (1);
	}

	//Setting crystallization parameters
	bool disable_transform = pcl::console::find_switch(argc, argv, "--NT");

	float voxel_resolution = 0.008f;
	bool voxel_res_specified = pcl::console::find_switch(argc, argv, "-v");
	if (voxel_res_specified)
		pcl::console::parse(argc, argv, "-v", voxel_resolution);

	float seed_resolution = 0.1f;
	bool seed_res_specified = pcl::console::find_switch(argc, argv, "-s");
	if (seed_res_specified)
		pcl::console::parse(argc, argv, "-s", seed_resolution);

	float color_importance = 0.2f;
	if (pcl::console::find_switch(argc, argv, "-c"))
		pcl::console::parse(argc, argv, "-c", color_importance);

	float spatial_importance = 0.4f;
	if (pcl::console::find_switch(argc, argv, "-z"))
		pcl::console::parse(argc, argv, "-z", spatial_importance);

	float normal_importance = 1.0f;
	if (pcl::console::find_switch(argc, argv, "-n"))
		pcl::console::parse(argc, argv, "-n", normal_importance);

	//////////////////////////////  //////////////////////////////
	////// This is how to use supervoxels
	//////////////////////////////  //////////////////////////////

	//Input point cloud and crystallization parameters
	pcl::SupervoxelClustering<PointT> super(voxel_resolution, seed_resolution);
	if (disable_transform)
		super.setUseSingleCameraTransform(false);
	super.setInputCloud(cloud);
	super.setColorImportance(color_importance);
	super.setSpatialImportance(spatial_importance);
	super.setNormalImportance(normal_importance);
	//Output Crystal Segmentation Results: The result is a mapping table
	std::map <uint32_t, pcl::Supervoxel<PointT>::Ptr > supervoxel_clusters;

	pcl::console::print_highlight("Extracting supervoxels!\n");
	super.extract(supervoxel_clusters);
	pcl::console::print_info("Found %d supervoxels\n", supervoxel_clusters.size());

	boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(new pcl::visualization::PCLVisualizer("3D Viewer"));
	viewer->setBackgroundColor(0, 0, 0);
	//Achieving Crystal Center
	PointCloudT::Ptr voxel_centroid_cloud = super.getVoxelCentroidCloud();
	viewer->addPointCloud(voxel_centroid_cloud, "voxel centroids");
	viewer->setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 2.0, "voxel centroids");
	viewer->setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_OPACITY, 0.95, "voxel centroids");
	//Obtaining crystals
	PointLCloudT::Ptr labeled_voxel_cloud = super.getLabeledVoxelCloud();
	viewer->addPointCloud(labeled_voxel_cloud, "labeled voxels");
	viewer->setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_OPACITY, 1, "labeled voxels");

	PointNCloudT::Ptr sv_normal_cloud = super.makeSupervoxelNormalCloud(supervoxel_clusters);
	//We have this disabled so graph is easy to see, uncomment to see supervoxel normals
	//viewer->addPointCloudNormals<PointNT> (sv_normal_cloud,1,0.05f, "supervoxel_normals");
	//Connect the connected crystal centers and display them. 
	pcl::console::print_highlight("Getting supervoxel adjacency\n");
	std::multimap<uint32_t, uint32_t> supervoxel_adjacency;
	super.getSupervoxelAdjacency(supervoxel_adjacency);
	//To make a graph of the supervoxel adjacency, we need to iterate through the supervoxel adjacency multimap
	std::multimap<uint32_t, uint32_t>::iterator label_itr = supervoxel_adjacency.begin();
	for (; label_itr != supervoxel_adjacency.end(); )
	{
		//First get the label
		uint32_t supervoxel_label = label_itr->first;
		//Now get the supervoxel corresponding to the label
		pcl::Supervoxel<PointT>::Ptr supervoxel = supervoxel_clusters.at(supervoxel_label);

		//Now we need to iterate through the adjacent supervoxels and make a point cloud of them
		PointCloudT adjacent_supervoxel_centers;
		std::multimap<uint32_t, uint32_t>::iterator adjacent_itr = supervoxel_adjacency.equal_range(supervoxel_label).first;
		for (; adjacent_itr != supervoxel_adjacency.equal_range(supervoxel_label).second; ++adjacent_itr)
		{
			pcl::Supervoxel<PointT>::Ptr neighbor_supervoxel = supervoxel_clusters.at(adjacent_itr->second);
			adjacent_supervoxel_centers.push_back(neighbor_supervoxel->centroid_);
		}
		//Now we make a name for this polygon
		std::stringstream ss;
		ss << "supervoxel_" << supervoxel_label;
		//This function is shown below, but is beyond the scope of this tutorial - basically it just generates a "star" polygon mesh from the points given
		addSupervoxelConnectionsToViewer(supervoxel->centroid_, adjacent_supervoxel_centers, ss.str(), viewer);
		//Move iterator forward to next label
		label_itr = supervoxel_adjacency.upper_bound(supervoxel_label);
	}

	while (!viewer->wasStopped())
	{
		viewer->spinOnce(100);
	}
	return (0);
}

void
addSupervoxelConnectionsToViewer(PointT &supervoxel_center,
	PointCloudT &adjacent_supervoxel_centers,
	std::string supervoxel_name,
	boost::shared_ptr<pcl::visualization::PCLVisualizer> & viewer)
{
	vtkSmartPointer<vtkPoints> points = vtkSmartPointer<vtkPoints>::New();
	vtkSmartPointer<vtkCellArray> cells = vtkSmartPointer<vtkCellArray>::New();
	vtkSmartPointer<vtkPolyLine> polyLine = vtkSmartPointer<vtkPolyLine>::New();

	//Iterate through all adjacent points, and add a center point to adjacent point pair
	PointCloudT::iterator adjacent_itr = adjacent_supervoxel_centers.begin();
	for (; adjacent_itr != adjacent_supervoxel_centers.end(); ++adjacent_itr)
	{
		points->InsertNextPoint(supervoxel_center.data);
		points->InsertNextPoint(adjacent_itr->data);
	}
	// Create a polydata to store everything in
	vtkSmartPointer<vtkPolyData> polyData = vtkSmartPointer<vtkPolyData>::New();
	// Add the points to the dataset
	polyData->SetPoints(points);
	polyLine->GetPointIds()->SetNumberOfIds(points->GetNumberOfPoints());
	for (unsigned int i = 0; i < points->GetNumberOfPoints(); i++)
		polyLine->GetPointIds()->SetId(i, i);
	cells->InsertNextCell(polyLine);
	// Add the lines to the dataset
	polyData->SetLines(cells);
	viewer->addModelFromPolyData(polyData, supervoxel_name);
}