AP 186 - Activity 7 Image Segmentation

Image segmentation is segregating the image using a particular region in your image as a reference. An example is this picture below where you want to emphasize the rust-colored object below. You cannot use  grayscale to segregate the area, since it has the same color as the area you don't need.  

Figure 1. (Left) Original image, (right) grayscale of original image.[1]

In grayscale this is easy to do. For example, given this image below:

Figure 2. Check image in grayscale.[1]

and when you use thresholding you get the segmentation of the image easily in Scilab. 

Figure 3. Segemented image of grayscale check.[1] 

Figure 4. Histogram of check image.

So what will we do to segment the image in color? We know that 3D objects even in one color have a variety of shading. This means that the object is subject to shadows so there will be different brightness levels of the same color. We will use the normalized chromaticity coordinates. What is that? 

For example, a red ball such as in Figure 5 will appear to have various shades of red from top to bottom. 

Figure 5. Simulated 3D red ball [1] .

For this reason, it is better to represent color space not by the RGB but by one that can separate brightness and chromaticity (pure color) information. One such color space is the normalized chromaticity coordinates or NCC. Per pixel, let I = R+G+B. Then the normalized chromaticity coordinates are

r = R / (R+G+B) = R / I

g = G / I

b = B / I

We note that r+g+b = 1 which implies r, g and b can only have values between 1 and 0 and b is dependent on r and g. Therefore, it is enough to represent chromaticity by just two coordinates, r and

g. [1]

We have thus reduced color information from 3 dimensions to 2. Thus, when segmenting 3D objects it is better to first transform RGB into rgI.

Figure 6. Normalized chromaticity coordinates. [1]

Figure 6 shows the r-g color space. When r and g are both zero, b = 1. Therefore, the origin corresponds to blue while points corresponding to r=1 and g=1 are points of pure red and green, respectively. 

Segmentation based on color can be performed by determining the probability that a pixel belongs to a color distribution of interest. We have two methods: parametric probability distribution estimation and non-parametric probability distribution estimation. 

First, we use the method of parametric estimation. So we take a picture from the internet which are colorful objects. Then we take the part of the picture we are interested in. This region of interest that I picked is the red pompom, and I cropped it out. 

Figure 7. Colorful yarn pompoms. [3]

Figure 8. Region of interest which is the red pompom. [3]

        We need this ROI so that we will compute its histogram. We get the histogram by normalizing by the number of pixels. This result is already the probability distribution function (PDF) of the color. To tag a pixel as belonging to a region of interest or not is to find its probability of belonging to the color of the ROI. Since our space is r and g we can have a joint probability p(r) p(g) function to test the likelihood of pixel membership to the ROI. We can assume a Gaussian distribution independently

along r and g, that is, from the r-g values of the cropped pixel we compute the mean r and mean g and standard deviation of r and g from the pixel samples. The probability that a pixel with chromaticity r belongs to the ROI is then 

This equation was also used to get the probability of pixel with chromaticity g. The joint probability is taken by the product of the probability of chromaticity r and probability of chromaticity g. [1]

So first, I had to compute the mean of r and g of the cropped picture and then the standard deviation of r and g of the cropped picture. I will then use the mean r and g, standard deviation of r and g into the above equation. I will get two probabilities, one a function of r and the other a function of g. I get the product of the two and I will get the segmented image which is based on red. 

Figure 9. Segmented image based on red ROI.

Figure 10. Side by side, (left) segmented and (right) original image.

Now we will use the non-parametric estimation.

In non-parametric estimation, the histogram itself is used to tag the membership of pixels. Histogram backprojection is one such technique where based on the color histogram, a pixel location is given a value equal to its histogram value in chromaticity space. This has the advantage of faster processing because no more computations are needed, just a look-up of histogram values. [1]

 This was done by first looking for the 2D histogram of the ROI which is again Figure 8. So its 2D histogram looks like this:

Figure 11. Rotated 2D histogram of ROI (left) and normalized chromaticity coordinates (NCC) (right).

The 2D histogram was rotated because its position with respect to the NCC was off by -90 degrees. You can see that the white parts correspond to the red parts in the NCC. Now we have to do histogram backprojection to the image. 

Figure 12. Segmentation using histogram backprojection. 

Figure 13. Histogram backprojection of original image using red ROI (left) and original image (right).

We can see that the lightest ball is the red yarn pompom and our region of interest. You can see that some parts of the orange yarn pompom matches the histogram of the red pompom. 

Comparing the two methods, I prefer the histogram backprojection since it focuses on the color of your region of interest compared to the parametric estimation where more colors are segmented that wasn't in the region of interest. In histogram backprojection, you can see that there is only a little bit of segmentation compared to that of the parametric method. But if you want the segmentation of your images to have a sharper edge, I recommend using parametric estimation. It really depends on your needs, if you want specific color image segmentation chose histogram backprojection. If you want a cleaner edge and clear area of segmentation go for parametric estimation. I also want to note that the parametric estimation is faster in processing the image outputs while in histogram backprojection it took 45 seconds longer for the image outputs to process. I still prefer the histogram backprojection.

Figure 14. (Left) Parametric

Using another image and using the parametric and non-parametric estimation/histogram backprojection: 

Figure 14. Colorful yarn balls. [2]

Figure 15. Green region of interest. [2]

Figure 16. Segmented image using parametric estimation.

Figure 17. Side by side, (left) segmented using parametric estimation and (right) original image.

Figure 18. Segmentation by histogram backprojection.

Figure 19. Side by side, (left) segmented using histogram backprojection and (right) original image.

As you can see, the histogram backprojection did a better job of segmenting the ROI which is in this case the green yarn ball in the middle row. In parametric segmentation, the bright light blue ball, light green yarn ball below the ROI were also included in the segmentation of the image.  This shows that the non-parametric estimation is better than the parametric estimation if you want the region of interest to be segmented only.

If I will score myself 10/10 for doing this activity and segmenting another image. This activity was very fun since we now have to deal with colors. I had a lot of fun figuring out how to do the activity and looking for the colorful pictures.

References:

[1] M. Soriano, AP 186 manual, A7 - Image Segmentation, 2014.

[2] "The Colorful White: Colorful Yarn Balls", Retrived from: http://thecolorfulwhite.blogspot.com/2012/11/colorful-yarn-balls.html
[3] "The Colorful White: Colorful Yarn PomPoms", Retrived from:  http://thecolorfulwhite.blogspot.com/2012/12/colorful-yarn-pompoms.html