System and method for panoramic image processing
The present disclosure provides a computer implemented method of image processing comprising, upon receiving of first and second images from an imaging unit, the first and second images being respectively associated with first and second rotational changes between a reference orientation and the orientations of the first and second images: processing data representative of the first image and of the second image to compensate the first and second rotational changes between the reference orientation and the respective orientations of the first and second images, thereby obtaining first and second corrected images; processing the first corrected image to detect distinctive keypoints within a fronto-parallel strip of the first corrected image; searching keypoints in the second corrected image corresponding to the detected keypoints, and estimating a geometric transformation between the first and second images based on matching the keypoints in the first and the second corrected images.
1. A non-transitory computer readable medium including instructions that when executed by a processor cause the processor to perform a method for stitching a sequence of images captured by a handheld device, the method comprising:
receiving a sequence of images of a shelving unit acquired using an imaging unit along a scanning direction in a retail store environment, wherein a plurality of images in the sequence are rotated relative to a reference orientation;
determining a first orientation of the imaging unit relative to the reference orientation while a first image is acquired based on an inclination level of shelves of the shelving unit detected in the first image;
determining a fronto-parallel strip for the first image based on the first orientation, wherein the fronto-parallel strip is substantially perpendicular to the scanning direction and positioned substantially in a center of the first image;
detecting distinctive features within the fronto-parallel strip of the first image;
matching the distinctive features detected in the fronto-parallel strip with distinctive features found in a second image associated with a second orientation of the imaging unit relative to the reference orientation; and
based on the matching, estimating a geometric transformation to enable stitching of the first image with the second image.
2. The non-transitory computer readable medium of claim 1 , wherein a width of the fronto-parallel strip is variable and includes a sufficient amount of distinctive features for enabling estimation of the geometric transformation.
3. The non-transitory computer readable medium of claim 2 , wherein the width of the fronto-parallel strip is in a range between of 1% and 10% of a field of view of an imaging sensor of the handheld device.
4. The non-transitory computer readable medium of claim 1 , wherein additional distinctive features located in the first image and outside of the fronto-parallel strip are discarded from further processing.
5. The non-transitory computer readable medium of claim 1 , wherein the reference orientation is an orientation of an initial image that differs from the first image.
6. The non-transitory computer readable medium of claim 1 , wherein determining the fronto-parallel strip of the first image includes determining an orientation of the first image relative to the reference orientation using measurements obtained from a positional sensor within the handheld device.
7. The non-transitory computer readable medium of claim 6 , wherein determining the fronto-parallel strip of the first image includes correcting the orientation of the first image with respect to the reference orientation based on a rotational change of first image.
8. The non-transitory computer readable medium of claim 7 , wherein the fronto-parallel strip is determined to be in a center of the corrected first image.
9. The non-transitory computer readable medium of claim 1 , wherein determining the fronto-parallel strip of the first image includes determining a theoretical central strip and a rotational threshold, and when the rotational change of the first image relative to the reference orientation is higher than the threshold rotational, the fronto-parallel strip is determined as the band in closest proximity to the theoretical central strip that contains distinctive features.
10. The non-transitory computer readable medium of claim 9 , wherein the rotational threshold is determined based on parameters associated with an imaging sensor within the handheld device.
11. The non-transitory computer readable medium of claim 1 , wherein the fronto-parallel strip is a vertical strip when the sequence of images results from a horizontal scanning.
12. The non-transitory computer readable medium of claim 1 , wherein the fronto-parallel strip is a horizontal strip when the sequence of images results from a vertical scanning.
13. The non-transitory computer readable medium of claim 1 , wherein matching the detected distinctive features includes:
defining a search area in the second image based on a position of a detected feature in the first image and on a rotational change of the first and second images; and
searching for the detected feature in the defined search area.
14. The non-transitory computer readable medium of claim 1 , wherein the geometric transformation includes a scale deformation based on distinctive features found in the fronto-parallel strip.
15. The non-transitory computer readable medium of claim 1 , further comprising:
estimating multiple geometric transformations between a plurality of successive pairs of images in the sequence of images to enable stitching a plurality of the images in the sequence of images.
16. The non-transitory computer readable medium of claim 1 , wherein the sequence of images is acquired during a rectilinear movement.
17. A handheld device, comprising:
memory;
an imaging unit including at least one imaging sensor configured to capture a sequence of images of a shelving unit acquired along a scanning direction in a retail store environment, wherein a plurality of images in the sequence are rotated relative to a reference orientation;
a processor configured to:
determine a first orientation of the imaging unit relative to the reference orientation while a first image is acquired based on an inclination level of shelves of the shelving unit detected in the first image;
determine a fronto-parallel strip for the first image based on an amount orientation, wherein the fronto-parallel strip is substantially perpendicular to the scanning direction and positioned substantially in a center of the first image;
detect distinctive features within the fronto-parallel strip of the first image;
match the distinctive features detected in the fronto-parallel strip with distinctive features found in a second image associated with a second orientation of the imaging unit relative to the reference orientation; and
based on the match, estimate a geometric transformation to enable stitching of the first image with the second image.
18. The handheld device of claim 17 , wherein the width of the fronto-parallel strip is in a range between of 1% and 5% of a field of view of the imaging sensor.
19. The handheld device of claim 17 , further comprising a positional sensor, and the processor is further configured to determine the fronto-parallel strip of the first image using measurements obtained from the positional sensor.