Robotic control of object placement

1 . A robot control system comprising circuitry configured to:

acquire, from an imaging device that images a designated area in which at least one object is to be placed, a designated area image showing the designated area;

execute image analysis on the designated area image to detect a current state of the designated area as a current area state;

generate, by simulation based on the current area state, a virtual state of the designated area which simulates one or more objects including an additional object as having been placed in the designated area, as a predicted area state;

generate object information on the additional object which is simulated in the designated area, based on the predicted area state, wherein the object information indicates a position or a posture of the additional object in the predicted area state; and

control a robot so as to physically place the additional object in the designated area in accordance with the position or the posture indicated by the object information,

wherein the circuitry is further configured to:

generate, for each of a plurality of predicted area states, a virtual image of the predicted area state by the simulation based on the current area state;

input, for each of the plurality of predicted area states, the virtual image of the predicted area state into an inference engine to obtain an evaluation value of the predicted area state, wherein the inference engine is configured to receive an input image and output the evaluation value for a subject depicted in the input image;

select one of the plurality of predicted area states based on the evaluation value of each of the plurality of the predicted area states; and

generate the object information based on the selected one predicted area state.

2 . The robot control system according to claim 1 ,

wherein the inference engine is learned in advance using a plurality of placement images indicating the plurality of post-placement area states and a plurality of evaluation values of placement, wherein each of the plurality of evaluation values of placement corresponds to each of the plurality of placement images, and

wherein the circuitry is configured to input each of a plurality of images indicating the plurality of predicted area states to the inference engine to calculate the evaluation value for each of the plurality of predicted area states.

3 . The robot control system according to claim 2 , wherein the plurality of evaluation values of placement used to learn the inference engine are set based on a color balance between objects in the designated area.

4 . The robot control system according to claim 3 , wherein the color balance is a balance of colors represented by a plurality of grades.

5 . The robot control system according to claim 1 , wherein the inference engine is a machine learning model constructed by a neural network.

6 . The robot control system according to claim 1 , wherein the circuitry is configured to simulate, as the predicted area state, a state of the designated area in which a predetermined amount of objects including the additional object has been placed.

7 . The robot control system according to claim 6 , wherein the circuitry is configured to predict a mass of the additional object and simulate the predicted area state based on the mass.

8 . The robot control system according to claim 2 , wherein the circuitry is configured to:

predict a total mass of the one or more objects in the designated area, for each of the plurality of predicted area states;

calculate, for each of the plurality of predicted area states, a total value of a mass score that increases as a difference between the total mass and a predetermined amount decreases and an evaluation score that increases as the evaluation value increases; and

select the one predicted area state from the plurality of predicted area states based on a plurality of the total values.

9 . The robot control system according to claim 1 , wherein the circuitry is configured to:

determine the position and the posture of the additional object simulated in the designated area based on the current area state;

generate the object information indicating the position and the posture of the additional object; and

control the robot so as to place the additional object in the designated area in accordance with the position and the posture indicated by the object information.

10 . The robot control system according to claim 1 , wherein the circuitry is configured to:

detect a plurality of candidate objects for the additional object;

determine which of the plurality of candidate objects has a quality that meets a predetermined quality standard;

select the additional object from the plurality of candidate objects having the quality that meets the quality standard; and

generate the object information indicating the selected additional object.

11 . The robot control system according to claim 1 , wherein the circuitry is configured to:

further evaluate a final state that is a state of the designated area in which a predetermined amount of the objects is placed;

generate, based on the final state, adjustment information for changing, in the designated area, a position or a posture of at least part of the predetermined amount of objects, such that the result of the evaluation is increased; and

control the robot based on the adjustment information.

12 . The robot control system according to claim 11 , wherein the circuitry is configured to:

analyze a calculation process of an inference engine used in the evaluation to extract an area of interest indicating a basis of the evaluation of the final state by the inference engine;

identify a partial area corresponding to the area of interest from the designated area; and

generate the adjustment information based at least on a position of the partial area.

13 . A method for controlling a robot, the method comprising:

acquiring, from an imaging device that images a designated area in which at least one object is to be placed, a designated area image showing the designated area;

executing image analysis on the designated area image to detect a current state of the designated area as a current area state;

generating, by simulation based on the current area state, a virtual state of the designated area which simulates one or more objects including an additional object as having been placed in the designated area, as a predicted area state;

generating object information on the additional object which is simulated in the designated area, based on the predicted area state, wherein the object information indicates a position or a posture of the additional object in the predicted area state; and

controlling a robot so as to physically place the additional object in the designated area in accordance with the position or the posture indicated by the object information,

wherein the method further comprises:

generating, for each of a plurality of predicted area states, a virtual image of the predicted area state by the simulation based on the current area state;

inputting, for each of the plurality of predicted area states, the virtual image of the predicted area state into an inference engine to obtain an evaluation value of the predicted area state, wherein the inference engine is configured to receive an input image and output the evaluation value for a subject depicted in the input image;

selecting one of the plurality of predicted area states based on the evaluation value of each of the plurality of the predicted area states; and

generating the object information based on the selected one predicted area state.

14 . A non-transitory computer-readable storage medium storing processor-executable instructions to:

acquire, from an imaging device that images a designated area in which at least one object is to be placed, a designated area image showing the designated area;

execute image analysis on the designated area image to detect a current state of the designated area as a current area state;

generate, by simulation based on the current area state, a virtual state of the designated area which simulates one or more objects including an additional object as having been placed in the designated area, as a predicted area state;

generate object information on the additional object which is simulated in the designated area, based on the predicted area state, wherein the object information indicates a position or a posture of the additional object in the predicted area state; and

control a robot so as to physically place the additional object in the designated area in accordance with the position or the posture indicated by the object information,

wherein the processor-executable instructions further cause at least one processor to:

generate, for each of a plurality of predicted area states, a virtual image of the predicted area state by the simulation based on the current area state;

input, for each of the plurality of predicted area states, the virtual image of the predicted area state into an inference engine to obtain an evaluation value of the predicted area state, wherein the inference engine is configured to receive an input image and output the evaluation value for a subject depicted in the input image;

select one of the plurality of predicted area states based on the evaluation value of each of the plurality of the predicted area states; and

generate the object information based on the selected one predicted area state.