Fruit and vegetable picking method and device based on machine vision and storage medium

1. A fruit and vegetable picking method based on machine vision, comprising:

collecting a data set of different types of fruits and vegetables and building a Mask r-cnn network model by using a deep learning framework on the dataset, the data set of different types of fruits and vegetables comprises a plurality of pulp images or plant images of different varieties of fruits and vegetables, a plurality of pulp images in different maturity levels of the fruits and vegetables, a plurality of environment images of a garden or a vegetable greenhouse, and a plurality of images of various fruits and vegetables in a to-be-picked state in the gardens or the vegetable greenhouses;

acquiring a fruit image of a fruit or a vegetable to be picked, calling the Mask r-cnn network model to identify the fruit image from the dataset, and determining a type of the fruit or the vegetable to be picked;

acquiring the type of the fruit or the vegetable to be picked, and determining a cuttable area on a fruit stalk of the fruit or the vegetable to be picked according to the type of the fruit or the vegetable, before locating a cutting point; and

controlling an end picking apparatus to cut off the stalk of the fruit or the vegetable to be picked, according to the cutting point determined.

2. The fruit and vegetable picking method based on machine vision according to claim 1 , further comprising, after the step of collecting a data set of different types of fruits and vegetables, and building a Mask r-cnn network model by using a deep learning framework, and before the step of acquiring a fruit image of a fruit or a vegetable to be picked, calling a pre-trained neural network model to identify the fruit image, and determining a type of the fruit or the vegetable to be picked:

performing a model learning and training on the data set by using a supervised learning method, performing a feature extraction on the data set of different types of fruits and vegetables by using the model trained, and constructing a neural network model for automatically identifying the pulp of a fruit or a vegetable.

3. The fruit and vegetable picking method based on machine vision according to claim 1 , wherein the step of acquiring a fruit image of a fruit or a vegetable to be picked, calling a pre-trained neural network model to identify the fruit image, and determining a type of the fruit or the vegetable to be picked further comprises:

using a camera apparatus preset on the picking device to obtain a plurality of fruit images of the fruit or the vegetable to be picked, and preprocessing the fruit images;

extracting a plurality of features from the fruit images preprocessed, and calling the neural network model to identify the features extracted; and

determining the type of the fruit or the vegetable to be picked, according to a recognition result.

4. The fruit and vegetable picking method based on machine vision according to claim 3 , wherein the step of preprocessing the fruit images further comprises: performing a noise reduction, a light supplement, and a white balance processing on the fruit images, to improve a quality of the fruit images.

5. The fruit and vegetable picking method based on machine vision according to claim 3 , wherein the step of extracting a plurality of features from the fruit images preprocessed, and calling a neural network model to identify the features extracted further comprises:

calling a Mask r-cnn backbone network to extract a feature map of the fruit images pre-processed, and using a regional recommendation network to generate a region proposal for a target; and

selecting a region of interest in the region proposal, before detecting the region of interest and outputting a recognition result.

6. The fruit and vegetable picking method based on machine vision according to claim 1 , wherein the step of acquiring the type of the fruit or the vegetable to be picked, and determining a cuttable area on a fruit stalk of the fruit or the vegetable to be picked according to the type of the fruit or the vegetable, before locating a cutting point further comprises:

acquiring the detected type of the fruit or the vegetable to be picked, predicting and selecting the stalks in the fruit images according to the type of the fruit or the vegetable to select a plurality of stalk areas; and

according to the stalk area selected, a cuttable area on the stalk is determined, and a best cutting point is located.

7. The fruit and vegetable picking method based on machine vision according to claim 1 , wherein the step of controlling an end picking apparatus to cut off the fruit stalk of the fruit or the vegetable to be picked, according to the cutting point selected further comprises:

controlling the end picking apparatus to clamp the stalk of the fruit or the vegetable to be picked, according to the cutting point determined; and

cutting the stalk off after clamping the stalk, and controlling the end picking apparatus to restore an initial state after the stalk is cut.

8. A picking device, comprising: a processor and a storage medium communicating with the processor, the storage medium stores a plurality of instructions; the processor follows the instructions stored in the storage medium to implement a plurality of steps of a fruit and vegetable picking method based on machine vision, the steps of the fruit and vegetable picking method based on machine vision comprising:

collecting a data set of different types of fruits and vegetables, and building a Mask r-cnn network model by using a deep learning framework on the dataset, wherein the data set of different types of fruits and vegetables comprises a plurality of pulp images or plant images of different varieties of fruits and vegetables, a plurality of pulp images in different maturity levels of the fruits and vegetables, a plurality of environment images of a garden or a vegetable greenhouse, and a plurality of images of various fruits and vegetables in a to-be-picked state in the gardens or the vegetable greenhouses;

acquiring a fruit image of a fruit or a vegetable current to be picked, calling the Mask r-cnn network model to identify the fruit image from the dataset, and determining a type of the fruit or the vegetable to be picked;

acquiring the type of the fruit or the vegetable to be picked, and determining a cuttable area on a fruit stalk of the fruit or the vegetable to be picked, according to the type of the fruit or the vegetable, before locating a cutting point; and

controlling an end picking apparatus to cut off the stalk of the fruit or the vegetable to be picked, according to the cutting point determined.

9. A storage medium, having a plurality of instructions stored, wherein the instructions are able to be loaded and executed by a processor, to implement a plurality of steps of a fruit and vegetable picking method based on machine vision, the steps of the fruit and vegetable picking method based on machine vision comprising:

collecting a data set of different types of fruits and vegetables, and building a Mask r-cnn network model by using a deep learning framework on the dataset;

wherein the data set of different types of fruits and vegetables comprises a plurality of pulp images or plant images of different varieties of fruits and vegetables, a plurality of pulp images in different maturity levels of the fruits and vegetables, a plurality of environment images of a garden or a vegetable greenhouse, and a plurality of images of various fruits and vegetables in a to-be-picked state in the gardens or the vegetable greenhouses;

acquiring a fruit image of a fruit or a vegetable current to be picked, calling the Mask r-cnn network model to identify the fruit image from the dataset, and determining a type of the fruit or the vegetable to be picked;

acquiring the type of the fruit or the vegetable to be picked, and determining a cuttable area on a fruit stalk of the fruit or the vegetable to be picked, according to the type of the fruit or the vegetable, before locating a cutting point; and

controlling an end picking apparatus to cut off the stalk of the fruit or the vegetable to be picked, according to the cutting point determined.