IP Library Granted Patent US 12,198,425
Granted Patent B2
US 12,198,425 · App. 18/324,603 · Granted Jan 14, 2025

Method, system, and medium having stored thereon instructions that cause a processor to execute a method for obtaining image information of an organism comprising a set of optical data

Inventors: Morio Ogura (Kanagawa, JP); Hirofumi Sumi (Kanagawa, JP)
Assignee: Sony Group Corporation
G06V20/188A01G7/00G01J3/36G01J3/513G01N21/251G01N21/255G01N21/314G06Q30/00G06Q50/02G06T7/0004H04N13/204H04N23/11G01J2003/2826G01N2021/0118G01N2021/1797G01N2021/3155G01N21/359G01N2021/8466G01N33/0098G06T2207/30128G06T2207/30188
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,425
App. No.
18/324,603
Granted
Jan 14, 2025
Kind
B2
Abstract

The present disclosure relates to methods and systems for obtaining image information of an organism including a set of optical data; calculating a growth index based on the set of optical data; and calculating an anticipated harvest time based on the growth index, where the image information includes at least one of: (a) visible image data obtained from an image sensor and non-visible image data obtained from the image sensor, and (b) a set of image data from at least two image capture devices, where the at least two image capture devices capture the set of image data from at least two positions.

Claims (16)

1. A method comprising:

obtaining image information of an organism comprising a set of image data;

calculating a growth index based on the set of image data; and

calculating an anticipated harvest time based on the growth index,

wherein the image information is obtained by at least two image capture devices, wherein the at least two image capture devices capture the set of image data from at least two different positions, wherein the image information comprises RGB image data obtained using a sensor detecting RGB light, and NIR (near infrared) image data obtained using a sensor detecting NIR light, wherein the image capture devices are installed at an angle to capture, in the obtained image information, R light intensity and NIR light intensity of incident light that is not reflected from the organism and light intensity of light that is reflected from the organism, and wherein the growth index is calculated based on measured R light intensity and NIR light intensity of the incident light that is not reflected from the organism and the light intensity of light that is reflected from the organism.

2. The method of claim 1 , wherein the image information comprises visible image data and non-visible image data and wherein the visible image data is generated by demosaicing of a RGB pixel signal to provide R, G and B component image signals and wherein the non-visible image data is generated based on a demosaiced R and IR signal.

3. The method of claim 1 , wherein the image information comprises visible image data and non-visible image data and wherein the visible image data is generated by demosaicing of a RGB pixel signal to provide R, G and B component image signals and wherein the non-visible image data is generated based on a R and IR signal without demosaicing.

4. The method of claim 1 , wherein the image capture devices used to obtain the set of image data each comprise a stack type image sensor, wherein the stack type image sensor has a blue light sensor layer stacked on a green light sensor layer, wherein the green light sensor layer is stacked on a red light sensor layer, and wherein the red light sensor layer is stacked on a near-infrared ray (NIR) sensor layer.

5. The method of claim 1 , wherein the image capture devices used to obtain the set of image data each comprise a set of RGB color filters provided over a laminated film, wherein the laminated film comprises at least one of SiO and SiN, and wherein the set of RGB color filters comprises a FR color filter, a FG color filter, and a FB color filter.

6. The method of claim 1 , further comprising calculating parallax image data based on at least two of the image data from the set of image data obtained from the at least two image capture devices; and calculating the growth index based on the parallax image data.

7. The method of claim 1 , wherein the image capture devices each comprise a set of RGB color filters provided over a set of infrared (IR) cut filters, wherein the set of IR cut filters is provided over a set of RGB sensors, wherein the image sensor further comprises a black filter provided over a NIR (near infrared) sensor with no IR cut filter provided over the NIR sensor and therefore with no blocking of NIR from the NIR sensor.

8. A method comprising:

obtaining image information of an organism comprising a set of image data;

calculating a growth index based on the set of image data; and

calculating an anticipated harvest time based on the growth index,

wherein the image information comprises RGB image data obtained using a sensor detecting RGB light, and NIR (near infrared) image data obtained using a sensor detecting NIR light, wherein the sensors are installed to capture, in the obtained image information, R light intensity and NIR light intensity of incident light that is not reflected from the organism and light intensity of light that is reflected from the organism, and wherein the growth index is calculated based on measured R light intensity and NIR light intensity of the incident light that is not reflected from the organism and the light intensity of light that is reflected from the organism.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 15, 2023
From: OGURA, MORIO; SUMI, HIROFUMI
To: SONY CORPORATION
Reel/Frame 064917/0435 →
CHANGE OF NAME Recorded Sep 15, 2023
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 064917/0693 →
Priority Claims (1)
JP 2013-062017 · Mar 25, 2013 · national
Continuity (4)
Continuation 17713639 · Apr 5, 2022
Continuation 16812678 · Mar 9, 2020
Continuation 14777549
Related Publication 20230306736A1 · Sep 28, 2023
References Cited (76)
US 4519193A · Yoshida et al. · 1985 [cited by applicant]
US 6160902A · Dickson et al. · 2000 [cited by applicant]
US 7928352B2 · Toda · 2011 [cited by applicant]
US 9113590B2 · Johnson · 2015 [cited by applicant]
US 10607078B2 · Ogura et al. · 2020 [cited by applicant]
US 11443509B2 · Ogura et al. · 2022 [cited by applicant]
US 11699286B2 · Ogura et al. · 2023 [cited by applicant]
US 20020101895A1 · Augusto · 2002 [cited by applicant]
US 20040264761A1 · Mas et al. · 2004 [cited by applicant]
US 20060167926A1 · Verhey et al. · 2006 [cited by applicant]
US 20060213167A1 · Koselka et al. · 2006 [cited by applicant]
US 20070003107A1 · Wei et al. · 2007 [cited by applicant]
US 20070005208A1 · Han et al. · 2007 [cited by applicant]
US 20080087800A1 · Toda · 2008 [cited by applicant]
US 20080123097A1 · Muhammed et al. · 2008 [cited by applicant]
US 20090168181A1 · Su et al. · 2009 [cited by applicant]
US 20090212946A1 · Pikaz · 2009 [cited by applicant]
US 20100098342A1 · Davis et al. · 2010 [cited by applicant]
US 20100115830A1 · Dube · 2010 [cited by applicant]
US 20100140461A1 · Sprigle et al. · 2010 [cited by applicant]
US 20100223276A1 · Al-Shameri et al. · 2010 [cited by applicant]
US 20100231755A1 · Sekine · 2010 [cited by applicant]
US 20110221895A1 · Sharma · 2011 [cited by applicant]
US 20110235017A1 · Iwasaki · 2011 [cited by applicant]
US 20110276336A1 · Sweely · 2011 [cited by applicant]
US 20120086095A1 · Nishiyama et al. · 2012 [cited by applicant]
US 20120109614A1 · Lindores · 2012 [cited by applicant]
US 20120155714A1 · Douglass et al. · 2012 [cited by applicant]
US 20120237083A1 · Lange et al. · 2012 [cited by applicant]
US 20120250962A1 · Scharf et al. · 2012 [cited by applicant]
US 20130141589A1 · Lin et al. · 2013 [cited by applicant]
US 20140093138A1 · Naganuma et al. · 2014 [cited by applicant]
US 20160283791A1 · Ogura et al. · 2016 [cited by applicant]
US 20200210699A1 · Ogura et al. · 2020 [cited by applicant]
US 20200356775A1 · Ogura et al. · 2020 [cited by applicant]
US 20220230431A1 · Ogura et al. · 2022 [cited by applicant]
US 20230245446A1 · Ogura et al. · 2023 [cited by applicant]
CN 101839979A · 2010 [cited by applicant]
CN 102013021A · 2011 [cited by applicant]
CN 102521564A · 2012 [cited by applicant]
CN 102565061A · 2012 [cited by applicant]
JP 2006250827A · 2006 [cited by applicant]
JP 2007310463A · 2007 [cited by applicant]
JP 2010166851A · 2010 [cited by applicant]
JP 2011027600A · 2011 [cited by applicant]
JP 2011203874A · 2011 [cited by applicant]
KR 20050043368A · 2005 [cited by applicant]
WO WO9919824A1 · 1999 [cited by applicant]
WO WO2003069315A · 2003 [cited by applicant]
WO WO2009116613A1 · 2009 [cited by applicant]
WO WO2009153983A1 · 2009 [cited by applicant]
WO WO2010063075A1 · 2010 [cited by applicant]
International Search Report and Written Opinion mailed Jun. 26, 2014 in connection with International Application No. PCT/JP2014/001497. [cited by applicant]
International Preliminary Report on Patentability mailed Oct. 8, 2015 in connection with International Application No. PCT/JP2014/001497. [cited by applicant]
Article 94(3) Communication issued May 31, 2017 in connection with European Application No. 14719877.4. [cited by applicant]
Chinese Office Action issued May 31, 2017 in connection with Chinese Application No. 201480016948.4 and English translation thereof. [cited by applicant]
Korean Office Action mailed Jun. 1, 2020 in connection with Korean Application No. 10-2015-7021071, and English translation thereof. [cited by applicant]
European Communication pursuant to Article 94(3) EPC dated Dec. 7, 2020 in connection with European Application No. 14719877.4. [cited by applicant]
Dworak et al., Strategy for the development of a smart N DVI camera system for outdoor plant detection and agricultural embedded systems. Sensors 13.2 (Jan. 2013): 1523-1538. [cited by applicant]
Garrity et al., A simple filtered photodiode instrument for continuous measurement of narrowband NDVI and PRI over vegetated canopies. Agricultural and Forest Meteorology. Mar. 15, 2010;150(3):489-96. [cited by applicant]
Hilker et al., Remote sensing of photosynthetic light-use efficiency across two forested biomes: Spatial scaling. Remote Sensing of Environment. Dec. 15, 2010;114(12):2863-74. [cited by applicant]
Hui et al., Image Segmentation Algorithm for Overlapping Fruits Based on Disparity Map, Agricultural Machinery Journal, Jun. 30, 2012, vol. 43, Issue 6, pp. 167-173. [cited by applicant]
Leemans et al., A method for plant leaf area measurement by using stereo vision. Proceedings of CIGR-AgEng 2012 International Conference on Agricultural Engineering. [cited by applicant]
Leuning et al., A multi-angle spectrometer for automatic measurement of plant canopy reflectance spectra. Remote Sensing of Environment. Aug. 15, 2006;103(3):236-45. [cited by applicant]
Nguy-Robertson et al., Green leaf area index estimation in maize and soybean: Combining vegetation indices to achieve maximal sensitivity. Agronomy Journal. Sep. 2012;104(5):1336-47. [cited by applicant]
Rundquist et al., Remote estimation of vegetation fraction in corn canopies. Proceedings of the 3rd European Conference on Precision Agriculture. 2001:301-6. [cited by applicant]
Sakamoto et al., Assessment of digital camera-derived vegetation indices in quantitative monitoring of seasonal rice growth. ISPRS Journal of Photogrammetry and Remote Sensing 66.6 (2011): 872-882. [cited by applicant]
Soudani et al., Ground-based Network of NDVI measurements for tracking temporal dynamics of canopy structure and vegetation phenology in different biomes. Remote sensing of environment. Aug. 1, 2012;123:234-45. [cited by applicant]
Tanaka et al., An Image Change Detection Application for Field Server. Proc. of IAALD-AFITA-WCCA. 2008:49-54. [cited by applicant]
Weber et al., Crop Field Reflectance Measurements. AIP Conference Proceedings. American Institute of Physics. Apr. 15, 2008(992)1: 671-76. [cited by applicant]
Yao et al., Comparison and intercalibration of vegetation indices from different sensors for monitoring above-ground plant nitrogen uptake in winter wheat. Sensors. Mar. 2013;13(3):3109-30. [cited by applicant]
Andersen et al., Geometric plant properties by relaxed stereo vision using simulated annealing. Computers and electronics in agriculture. Nov. 1, 2005;49(2):219-32. [cited by applicant]
Rankin et al., Daytime mud detection for unmanned ground vehicle autonomous navigation. Proceedings of the 26th Army Science Conference. Dec. 1, 2008. 10 pages. [cited by applicant]
Rovira-Más et al., Creation of three-dimensional crop maps based on aerial stereoimages. Biosystems engineering. Mar. 1, 2005;90(3):251-9. [cited by applicant]
Sun et al., Evaluation of maize growth by ground based multi-spectral image. 2011 IEEE/SICE International Symposium on System Integration (SII) Dec. 20, 2011:207-11. [cited by applicant]
Extended European Search Report dated Jun. 29, 2023 in connection with European Application No. 23169780.6. [cited by applicant]