IP Library Granted Patent US 10,339,380
Granted Patent B2
US 10,339,380 · App. 15/271,569 · Granted Jul 2, 2019

Hi-fidelity computer object recognition based horticultural feedback loop

Inventors: Adam Phillip Takla Greenberg (San Francisco, CA); Matthew Charles King (Seattle, WA)
Assignee: IUNU, Inc.
G06K9/00657A01G7/00G06K2209/17G06T7/00G06T2207/30188
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,339,380
App. No.
15/271,569
Granted
Jul 2, 2019
Kind
B2
Abstract

Infrastructure and methods to implement a feedback loop for a horticultural operation are disclosed. Image capture devices are deployed on a per plant or near per plant basis, and upload images to image analysis server. The image analysis server applies image preprocessing to the uploaded images, and then applies image analysis to identify plants and plant artifacts. Where artifacts indicate an issue with a plant, a course of action to remediate is identified. Static analysis techniques, sequential analysis techniques, and adaptive techniques are some of the techniques enabled and disclosed by the application of object recognition techniques to horticultural applications. Images may be associated with a plant state vector to enable complex comparison operations.

Claims (74)

1. A computing device to detect anomalies on a plant, the computing device comprising:

a processor;

a memory communicatively coupled to the processor, the memory storing a set of software components, comprising:

an image retriever software component, which when executed by the processor, directs the computing device to receive a plurality of images from one or more grow operations;

a classifier software component, which when executed by the processor, directs the computing device to identify a plurality of discrete objects in a first image of the plurality of images as portions of the first image that represent an individual plant present in one of the one or more grow operations;

an identifier software component, which when executed by the processor, directs the computing device to:

identify the individual plant as a same plant previously identified by the computing device based on at least one of the discrete objects; and

identify at least one artifact of the individual plant based on at least one of the discrete objects; and

an analyzer software component, which when executed by the processor, directs the computing device to:

select an analysis method from among a plurality of analysis methods based on the identification of the at least one artifact;

analyze the at least one artifact, via the selected analysis method, to identify at least one issue of the individual plant; and

store the at least one issue of the individual plant, wherein storing identified issues includes storing the identified issues on a per plant basis.

2. The computing device of claim 1 , wherein the set of software components further comprises:

an image preprocessor software component, which when executed by the processor, directs the computing device to perform at least one of the following:

a histogram generation of red-green-blue (RGB) measurements of the first image;

a histogram generation of hue-luminance-saturation (HLS) measurements of the first image;

an iteration count of morphological opens and/or closes of the first image;

a Gaussian blur transformation of the first image; and

an application of Otsu segmentation to the first image.

3. The computing device of claim 2 , wherein the image preprocessor software component is further configured to determine whether the first image has sufficient content for further processing.

4. The computing device of claim 1 , wherein the classifier software component is further configured to direct the computing device to:

separate the identified discrete objects representing plants from identified discrete objects not representing plants based on size, color, inertia ratio, and/or contour area.

5. The computing device of claim 1 , wherein the identifier software component is further configured to direct the computing device to:

identify the individual plant based on a plant state vector comprised of variables that are translation and transformation invariant; and

identify the individual plant as the same plant previously identified regardless of whether the individual plant has been moved or rotated.

6. The computing device of claim 5 , wherein the plant state vector variables include at least some of the following:

contour area;

inertia area;

color histogram;

Hu moments;

leaf count; and/or

leaf configuration.

7. The computing device of claim 1 , wherein the analyzer software component is further configured to direct the computing device to:

identify at least one course of action to mitigate the at least one issue; and

provide the at least one course of action to the one grow operation.

8. The computing device of claim 7 , wherein providing the at least one course of action to the one grow operation comprises communicating with the one grow operation to adjust a light spectrum or intensity of light provided to the individual plant by a lumière feedback device included in the one grow operation.

9. The computing device of claim 1 , wherein selecting the analysis method from among the plurality of analysis methods comprises:

selecting a leaf articulation analysis in response to identifying the at least one artifact as a leaf of the individual plant, wherein analyzing the at least one artifact via the leaf articulation analysis comprises determining an orientation of the leaf of the individual plant with respect to a light source;

selecting a plant branch analysis in response to identifying the at least one artifact as a branch of the individual plant, wherein analyzing the at least one artifact via the plant branch analysis comprises generating a representation of a size and orientation of the branch of the individual plant; and

selecting a root analysis in response to identifying the at least one artifact as a root of the individual plant, wherein analyzing the at least one artifact via the root analysis comprises determining a structure of the root of the individual plant.

10. A method to identify potential issues in a plant, comprising:

receiving, at a computing device, a plurality of images from one or more grow operations;

identifying, by the computing device, a plurality of discrete objects in a first image of the plurality of images as portions of the first image that represent an individual plant present in one of the one or more grow operations;

identifying, by the computing device, the individual plant as a same plant previously identified in a second image by the computing device based on at least one of the discrete objects;

identifying, by the computing device, a first plurality of artifacts of the individual plant based on at least one of the discrete objects identified in the first image;

comparing the first plurality of artifacts corresponding to the first image with a second plurality of artifacts corresponding to the second image on a per artifact basis to detect at least one difference in the individual plant;

identifying at least one issue of the individual plant based on the detection of the at least one difference in the individual plant; and

storing the at least one issue of the individual plant, wherein storing identified issues includes storing the identified issues on a per plant basis.

11. The method of claim 10 , wherein identifying the individual plant as the same plant is based on a plant state vector, and wherein identifying at least one artifact of the first plurality of artifacts is based on an edge cleaner software component operation.

12. The method of claim 10 , wherein the first image is associated with a first time stamp and the second image is associated with a second time stamp.

13. The method of claim 10 , wherein the detected at least one difference in the individual plant comprises a Euclidean distance measure of the individual plant.

14. The method of claim 13 , further comprising:

determining whether the Euclidean distance measure is greater than a predetermined threshold; and

determining whether the individual plant was moved or rotated in response to the Euclidean distance measure being greater than the predetermined threshold to confirm that the individual plant in the first image is the same plant as the individual plant in the second image.

15. The method of claim 10 , further comprising:

identifying at least one course of action to mitigate the at least one issue; and

providing the at least one course of action to the one grow operation.

16. The method of claim 15 , wherein providing the at least one course of action to the one grow operation comprises communicating with the one grow operation to adjust a light spectrum or intensity of light provided to the individual plant by a lumière feedback device included in the one grow operation.

17. A non-transitory computer-readable media storing instructions, which when executed by at least one processor of a computing device, direct the computing device to:

receive a plurality of images from one or more grow operations;

collect histogram data over the plurality of images for images that are of a same category;

generate an average histogram based on the collected histogram data;

receive a subsequent image from the one or more grow operations;

identify a plurality of discrete objects in the subsequent image as portions of the subsequent image that represent an individual plant present in one of the one or more grow operations;

identify the individual plant as a same plant previously identified by the computing device based on at least one of the discrete objects;

identify at least one artifact of the individual plant based on at least one of the discrete objects and based on the average histogram;

recalculate the average histogram using a histogram of the subsequent image;

analyze the at least one artifact to identify at least one issue of the individual plant; and

store the at least one identified issue of the individual plant, wherein storage of the at least one identified issue includes storing a plurality of identified issues on a per plant basis.

18. The non-transitory computer-readable media of claim 17 , further comprising instructions to direct the computing device to:

identify the individual plant as the same plant previously identified regardless of whether the individual plant has been moved or rotated.

19. The non-transitory computer-readable media of claim 17 , further comprising instructions to direct the computing device to:

identify at least one course of action to mitigate the at least one identified issue; and

communicate the at least one course of action to the one grow operation to adjust a light spectrum or intensity of light provided to the individual plant by a lumière feedback device included in the one grow operation.

Assignments (5)
SECURITY INTEREST Recorded Nov 15, 2024
From: IUNU, INC.
To: FIRST-CITIZENS BANK & TRUST COMPANY
Reel/Frame 069282/0466 →
CHANGE OF NAME Recorded May 18, 2017
From: IUNU, LLC
To: IUNU, INC.
Reel/Frame 042500/0892 →
RELEASE OF SECURITY INTEREST Recorded May 4, 2017
From: LINDQUIST, THOMAS M.
To: IUNU, LLC.
Reel/Frame 042247/0113 →
SECURITY INTEREST Recorded Nov 16, 2016
From: IUNU, LLC.
To: LINDQUIST, THOMAS M.
Reel/Frame 040347/0797 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2016
From: GREENBERG, ADAM PHILLIP TAKLA; KING, MATTHEW CHARLES
To: IUNU, LLC
Reel/Frame 040314/0074 →
Continuity (1)
Related Publication 20180082412A1 · Mar 22, 2018
Cited By (3)
US 12,197,219 US 12,423,762 US 12,716,879