Object enrollment in a robotic cart coordination system
One or more simulated capture paths through a physical environment may be determined for a robot based on an environment navigation model of the physical environment. A plurality of simulated object parameter values may be determined for an object type. Simulated sensor data for a plurality of simulated instances of the object type may be determined based on the one or more simulated capture paths, the environment navigation model, and the simulated object parameter values. An object recognition model to recognize an object corresponding with the object type based on the simulated sensor data.
1 . A method comprising:
determining one or more simulated capture paths for a virtual robot through a physical environment based on an environment navigation model of the physical environment determined based on sensor data collected via a handheld computing device, the environment navigation model including a plurality of semantic labels corresponding to elements of the physical environment;
receiving as input a three-dimensional model of an object type;
determining a plurality of simulated object parameter values for the object type, the simulated object parameter values identifying a plurality of possible variations of real-world conditions in which an object of the object type may appear, the simulated object parameter values including a parameter value selected from the group consisting of: object grouping, object rotation, and object stacking;
determining simulated sensor data for a plurality of simulated instances of the object type based on the one or more simulated capture paths, the environment navigation model, the three-dimensional model, and the simulated object parameter values, the simulated sensor data including image data collected from a simulated camera at the virtual robot;
training an environment-specific object recognition model based on the simulated sensor data, the environment-specific object recognition model being trained to recognize an object corresponding with the object type based on variations in appearance of objects corresponding to the object type that are specific to the physical environment;
transmitting the environment-specific object recognition model to a plurality of physical robots; and
instructing the plurality of physical robots to autonomously navigate the physical environment and recognize objects based on the environment-specific object recognition model and sensor data collected via the plurality of physical robots.
2 . The method recited in claim 1 , wherein the simulated object parameter values include one or more object intrinsic characteristic parameters that characterize one or more physical characteristics of one or more of the plurality of simulated instances of the object type.
3 . The method recited in claim 2 , wherein the one or more object intrinsic characteristic parameters are selected from the group consisting of: size, shape, texture, shading, stacking, and blur.
4 . The method recited in claim 1 , wherein the simulated object parameter values include one or more object placement characteristic parameters that characterize one or more physical characteristics of one or more of the plurality of simulated instances of the object type.
5 . The method recited in claim 1 , wherein the simulated object parameter values include one or more object extrinsic characteristic parameters that characterize the physical environment.
6 . The method recited in claim 5 , wherein the one or more object extrinsic characteristic parameters include a parameter related to a lighting or shading condition of the physical environment.
7 . The method recited in claim 1 , wherein instructing the plurality of physical robots to autonomously navigate the physical environment comprises transmitting a navigation instruction from a fleet management system configured to manage operation of the plurality of physical robots at the physical environment.
8 . The method recited in claim 1 , wherein the handheld computing device is a mobile phone.
9 . The method recited in claim 1 , wherein the environment navigation model for the physical environment is determined based on simultaneous localization and mapping (SLAM).
10 . The method recited in claim 1 , wherein the simulated sensor data includes depth sensor data collected from a simulated depth sensor at the virtual robot.
11 . The method recited in claim 1 , wherein the simulated sensor data includes LIDAR data collected from a simulated LiDAR sensor at a robot.
12 . One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
determining one or more simulated capture paths for a virtual robot through a physical environment based on an environment navigation model of the physical environment determined based on sensor data collected via a handheld computing device, the environment navigation model including a plurality of semantic labels corresponding to elements of the physical environment;
receiving as input a three-dimensional model of an object type;
determining a plurality of simulated object parameter values for the object type, the simulated object parameter values identifying a plurality of possible variations of real-world conditions in which an object of the object type may appear, the simulated object parameter values including a parameter value selected from the group consisting of: object grouping, object rotation, and object stacking;
determining simulated sensor data for a plurality of simulated instances of the object type based on the one or more simulated capture paths, the environment navigation model, the three-dimensional model, and the simulated object parameter values, the simulated sensor data including image data collected from a simulated camera at the virtual robot;
training an environment-specific object recognition model based on the simulated sensor data, the environment-specific object recognition model being trained to recognize an object corresponding with the object type based on variations in appearance of objects corresponding to the object type that are specific to the physical environment;
transmitting the environment-specific object recognition model to a plurality of physical robots; and
instructing the plurality of physical robots to autonomously navigate the physical environment and recognize objects based on the environment-specific object recognition model and sensor data collected via the plurality of physical robots.
13 . A system comprising a processor, a storage device, and memory, the memory including instructions to perform a method comprising:
determining one or more simulated capture paths for a virtual robot through a physical environment based on an environment navigation model of the physical environment determined based on sensor data collected via a handheld computing device, the environment navigation model including a plurality of semantic labels corresponding to elements of the physical environment;
receiving as input a three-dimensional model of an object type;
determining a plurality of simulated object parameter values for the object type, the simulated object parameter values identifying a plurality of possible variations of real-world conditions in which an object of the object type may appear, the simulated object parameter values including a parameter value selected from the group consisting of: object grouping, object rotation, and object stacking;
determining simulated sensor data for a plurality of simulated instances of the object type based on the one or more simulated capture paths, the environment navigation model, the three-dimensional model, and the simulated object parameter values, the simulated sensor data including image data collected from a simulated camera at the virtual robot;
training an environment-specific object recognition model based on the simulated sensor data, the environment-specific object recognition model being trained to recognize an object corresponding with the object type based on variations in appearance of objects corresponding to the object type that are specific to the physical environment;
transmitting the environment-specific object recognition model to a plurality of physical robots; and
instructing the plurality of physical robots to autonomously navigate the physical environment and recognize objects based on the environment-specific object recognition model and sensor data collected via the plurality of physical robots.