CS129 / Project 6 / Automated Panorama Stitching

Example of a automated panorama image.





1. Algorithm

      1.1 part one: define correspondence

      To achieve the goal of define correspondence, we have to select the appropriate points that can be stable in images and also distinguishable. The points source we choose is corner points, using Harris's algorithm we can easily get all the corner points in a image. Since the Harris corners has implemented for us, I plotted all the corner points to see the distribution of all the points in the image:

      left

      right



      As is shown above, the original Harris corner points are distributed randomly, for some image patches that have more high level energy, the points around that area will be intensive, and for the ares that has most low energy, than the corner points there are rare. To get the more evenly distributed points, I use ANMS: Adaptive non-maximal suppression. The idea here is to calculate the radius for each points and select the points according to the radius in descending order. The results of applying ANMS is showed below:

      Original

      ANMS



      We can see from above that the points is selected so that all the areas can be considered. After the ANMS, we got all the points that may be a correspondence for both two images. How can we know that a point is a correspondence, we can use SSD metric, but what is to be compared between the points, that is, how to define features? The easiest way to come up with is to get an image patch around a specific point and compare the image patches between two images. To simplify our calculation, sample the patches down to 8*8 patch and normalize it( minus the average value and divided by standard variation). Below is the descriptor of a corner point:

      Descriptor (8*8)

      Using the distance between the descriptors is a good method to define correspondence, but using the ratio between the first nearest distance and the second nearest distance is better, because it is hard to find some global metric in distance value that can apply to all the points, but the ratio between the nearest 2 is uniform, from the paper: Brown, et al. I choose the ratio threshold to be 0.5. And finally, I get correspondence points.

      The correspondence

      Total 20 points in correspondence

      1.2 part two: RANSAC + Recover homography

      After define the correspondence of the two images, it is not a good idea to calculate the transformation matrix using all the points, because some of the points maybe outliers. We can use RANSAC to get the inline points and than we will calculate the matrix using all the inline points. The success of RANSAC here is not that there are more inline points than outline points, but that the outline points are outlined differently. The parameters for RANSAC has given to us: iterate 1000 times and get the most inline points. Below is total 11 inline points of image1.

      Left

      Right



      Total 11 points that is used to construct the transformation matrix. Using lmdivide function we can achieve the least squares result from all the corresponding points. The result of this image is showed at the top of the webpage, which is aligned from the right image to the left.



2. Some discussions

Aligning more than 2 images:

When aligning more than 2 images, the result can be not so good because the images are still aligned every 2 times, which may have artifact that will have effects on the images to be aligned later. The tricky part here is that sometimes the correspondence lies on the edge of a composed image that the model may be far from good taking the a lot of edge points into account.




3. Results

My Images:

Image of Brown from CIT 5th floor:

Original_1

Original_2

Result






Images of my office( Room 508 in CIT)

Original_1

Original_2

Original_3

Result







Sample Images:

Left

Right

Panorama












Original_1

Original_2

Original_3

Result














Original_1

Original_2

Original_3

Original_4

Result







Original_1

Original_2

Original_3

Original_4

Original_5

Result