Stitching Parrot Sequoia individual spectral band images into a single false color RGB composite image.

by | June 8, 2017

First, I have to do two things I hate.

Let me start of by saying, sorry for the non-existent updates on this site. I’ve been busier than I ever thought was possible so I’ve not been able to keep updating this site like I had hoped.

Also, if you find this helpful and decide you want to use it in a commercial workflow I would appreciate whatever small donation you see fit to provide using either of the two options below:




or by Ethereum ETH to the following address:

0xc9F43eF1f913ceBd1BA4f736F05341D45f1fc026

etheraddress

 

Now that that unpleasantness is over, let’s get to the goods. This script will allow you to batch process all of the images taken by your Parrot Sequoia multi-spectral camera and turn them into false color composites (JPEG format by default)

Turn this:

Turn this

Into this:

IMG_170520_174501_0015_NIR_merged

For this to work you will need the following installed on your PC:

  • Python 3.x (I recommend Anaconda as a distribution as it comes with almost all the most important libraries pre-installed)
  • numpy
  • openCV3 with SIFT (This is no longer included with the standard openCV implementation, you can find a .whl file for you to install on a Windows Python installation here)

If you are like me, and disappointed in the fact that the camera does not stitch these images together into a composite image. Then this is for you. If you try to directly overlay the bands and merge them into a single photo you get an image that is horribly out of focus with spectral bands completely misaligned. It is also impossible to simply shift the images by a certain number of pixels because the distortion involves both translation along the X and Y axes as well as rotation around X, Y and Z.

Warning – I get wordy in here. If you want to just download the script and README file click HERE

Now, coming from a remote sensing background. Particularly in Structure from Motion (SfM) photogrammetry I knew the solution to this problem resided in using SIFT to identify the exact translation needed to align the images. This is pretty easy to do in MATLAB, but MATLAB is not a software package I enjoy using, nor does it allow for tidy integration with other aspects of my workflow. So, out comes my trusty OpenCV handbook and a beer (OK, two beers and a bottle of my ever improving homemade Riesling).

I want to mention now that I found this tutorial on the OpenCV-Python-Tutorials site very helpful.

First we need to create a list of all the files in the input folder. I used the os.listdir() and os.path.isfile() in a list comprehension to accomplish this.

image_list = [f for f in os.listdir(input_folder) if os.path.isfile(os.path.join(input_folder,f))]

Then we need to break this into an iterable set where the images taken at the same time are grouped together. I decided to create a list of tuples that accomplishes precisely this. Because there are always 4 images per set I used another list comprehension to zip every 4 images in the image_list variable into its own tuple within another list called image_tups

image_tups = zip(*[image_list[i::4] for i in range(4)])

Next we need to pick an image to use as the fixed image, or the image to which we are trying to force the others to conform to. After looking at the camera’s sensor layout I decided that the sensor that split the difference in horizontal and vertical orientation change between the other two spectral bands was ideal.

In the case that I want an image with the Green, Red Edge and NIR bands, I chose the Red Edge band as the fixed image as this sensor splits the difference. If I had chosen either Green or NIR as the fixed image there would be a lot more translation necessary for the spectral band at the far end.

I accomplish this by using a simple if-elif block:

if 1 in channel_order and 2 in channel_order and 3 in channel_order:
 fixed_image = 1
 moving_im1 = 0
 moving_im2 = 2
elif 2 in channel_order and 3 in channel_order and 4 in channel_order:
 fixed_image = 2
 moving_im1 = 1
 moving_im2 = 3
elif 1 in channel_order and 3 in channel_order and 4 in channel_order:
 fixed_image = 2
 moving_im1 = 0
 moving_im2 = 3
elif 1 in channel_order and 2 in channel_order and 4 in channel_order:
 fixed_image = 1
 moving_im1 = 0
 moving_im2 = 3

Now we use a FOR loop to iterate through each of the tuples in image_tups. We run the align_images() function defined at the top of the code to place the appropriate image object into the variables band1 and band2. band3 reads the fixed image directly off the disk and converts it into grayscale using the 0 integer flag in cv2.imread().

Then we merge the 3 bands into a single BGR (OpenCV is weird that way) image using the cv2.merge() function.

for tup in image_tups:
    band1 = align_images(input_folder, output_folder, os.path.join(input_folder, tup[moving_im1]),
                         os.path.join(input_folder, tup[fixed_image]))
    band2 = align_images(input_folder, output_folder, os.path.join(input_folder, tup[moving_im2]),
                         os.path.join(input_folder, tup[fixed_image]))
    band3 = cv2.imread(os.path.join(input_folder, tup[fixed_image]), 0)

    merged = cv2.merge((band1, band2, band3))

    cv2.imwrite(os.path.join(output_folder, tup[fixed_image][-30:-4]) + '_merged.jpg', merged)

The align_images() function is pretty well commented. However it is worth pointing out that I used the RANSAC method to fit the shifts identified using SIFT to the outer corners of the moving images. I then reshaped the image based on these point shifts using cv2.warpPerspective()

Hope this all helps out. Again you can download the source code from my git repository here:

https://github.com/tpubben/SequoiaStacking/

 

Leave a Reply

Your email address will not be published. Required fields are marked *