Operating room objects and workflow tracking using depth cameras
Embodiments described herein provide systems and techniques for tracking workflow inside an operating room (OR) by identifying and tracking target objects, such as a patient bed or a surgical table in the OR. In one aspect, a process for identifying and tracking a target object in an OR begins by receiving a depth image among a sequence of depth images captured by a depth camera installed in the OR. The process then generates a three-dimensional (3D) point cloud based on the depth image. Next, the process identifies a set of potential target points in the 3D point cloud that potentially belongs to the target object based on one or more target object criteria. The process next extracts one or more object clusters of from the set of potential target points using a data-point clustering technique. The process subsequently identifies the target object from the extracted one or more object clusters.
1. A computer-implemented method for identifying and tracking a target object being a person in an operating room (OR) and de-identifying a color video captured by a 2D camera, the method comprising:
receiving a 2D color image captured by a 2D camera and a depth image among a sequence of depth images captured by a depth camera both cameras operating simultaneously and installed in the OR;
identifying a set of potential target points in the received depth image that potentially belongs to the target object based on one or more target object criteria;
extracting one or more object clusters from the set of potential target points using a data-point clustering technique;
identifying the target object from the extracted one or more object clusters as a body of a person and extracting a 3D body contour from the extracted one or more clusters;
projecting the 3D body contour into a 2D body outline in the received 2D color image;
overlaying the 2D body outline onto the received 2D color image; and
de-identifying the received 2D color image based on the overlaid 2D body outline.
2. The computer-implemented method of claim 1 , wherein prior to identifying the set of potential target points, the method further comprises generating a three-dimensional (3D) point cloud based on the received depth image by projecting each 2D pixel (u, v) and the corresponding distance value d (u, v) in the depth image into a 3D point in a 3D-coordinate system aligned with the depth camera.
3. The computer-implemented method of claim 2 , wherein the method further comprises, for each 3D point in the 3D point cloud:
computing a surface orientation at the 3D point based on one or more vector cross-products between two or more edges formed by the 3D point and two or more neighboring 3D points of the 3D point in the 3D point cloud; and
associating the computed surface orientation with the 3D coordinates of the 3D point.
4. The computer-implemented method of claim 3 , wherein identifying the set of potential target points in the 3D point cloud that potentially belongs to the target object based on one or more target object criteria include:
transforming each 3D point and the associated surface orientation in the 3D point cloud into a 3D-coordinate system aligned with the ground surface of the OR; and
extracting the set of 3D potential target points from the transformed 3D point cloud by filtering the transformed 3D point cloud base on one or more geometrical criteria of the target object.
5. The computer-implemented method of claim 1 further comprising:
generating a set of 3D body joints for the extracted 3D body contour;
projecting the set of 3D body joints into a 2D skeleton figure in the received 2D color image; and
overlaying the 2D skeleton figure onto the received 2D color image, wherein de-identifying the received 2D color image is further based on the overlaid 2D skeleton figure.
6. The computer-implemented method of claim 1 , wherein the data-point clustering technique is a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering technique which is configured to identify a cluster of target points in the set of potential target points having a higher density than the remainder of the set of potential target points.
7. The computer-implemented method of claim 1 , wherein identifying the target object from the extracted one or more object clusters includes:
for each object cluster in the extracted one or more object clusters,
generating a minimum bounding box for the object cluster; and
determining whether the object cluster is the target object by comparing the dimensions of the generated minimum bounding box against the dimensions of the target object.
8. The computer-implemented method of claim 7 , wherein determining whether the object cluster is the target object further includes applying one or more additional detection requirements listed below:
determining whether the number of data points inside the generated minimum bounding box satisfies a number-of-points requirement of the target object;
determining whether the orientation of the generated minimum bounding box satisfies a surface orientation requirement of the target object; and
determining whether the position of the generated minimum bounding box satisfies a position requirement of the target object.
9. The computer-implemented method of claim 1 , wherein after installing the depth camera, the method further comprises calibrating the depth camera to obtain the position and the orientation of the depth camera with respect to a ground surface in the OR.
10. The computer-implemented method of claim 1 , wherein after identifying the target object from the extracted one or more object clusters, the method further comprises tracking the identified target object by:
determining if the identified target object has been previously-identified in one or more preceding depth images in the sequence of depth images; and
if so, estimating a movement of the identified target object;
otherwise, determining that the identified target object is a newly-identified target object not previously identified.
11. An apparatus for identifying and tracking a target object being a person in an operating room (OR) and de-identifying a color video captured by a 2D camera, the apparatus comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
receive a 2D color image captured by a 2D camera and a depth image among a sequence of depth images captured by a depth camera both cameras operating simultaneously while installed in the OR;
identify a set of potential target points in the received depth image that potentially belongs to the target object based on one or more target object criteria;
extract one or more object clusters from the set of potential target points using a data-point clustering technique;
identify the target object from the extracted one or more object clusters as a body of a person and extract a 3D body contour from the extracted one or more clusters;
project the 3D body contour into a 2D body outline in the received 2D color image;
overlay the 2D body outline onto the received 2D color image; and
de-identify the received 2D color image based on the overlaid 2D body outline.
12. The apparatus of claim 11 , wherein prior to identifying the set of potential target points, the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to generate a three-dimensional (3D) point cloud based on the received depth image by projecting each 2D pixel (u, v) and the corresponding distance value d(u, v) in the depth image into a 3D point in a 3D-coordinate system aligned with the depth camera.
13. The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
compute a surface orientation, at each 3D point in the 3D point cloud, based on one or more vector cross-products between two or more edges formed by the 3D point and two or more neighboring 3D points of the 3D point in the 3D point cloud; and
associate the computed surface orientation at each 3D point in the 3D point cloud with the 3D coordinates of the 3D point.
14. The apparatus of claim 13 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to identify the set of potential target points in the 3D point cloud based on one or more target object criteria by:
transforming each 3D point and the associated surface orientation in the 3D point cloud into a 3D-coordinate system aligned with the ground surface of the OR; and
extracting the set of 3D potential target points from the transformed 3D point cloud by filtering the transformed 3D point cloud base on one or more geometrical criteria of the target object.
15. The apparatus of claim 14 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
generate a set of 3D body joints for the extracted 3D body contour;
project the set of 3D body joints into a 2D skeleton figure in the received 2D color image; and
overlay the 2D skeleton figure onto the received 2D color image, wherein de-identifying the received 2D color image is further based on the overlaid 2D skeleton figure.
16. The apparatus of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to identify the target object from the extracted one or more object clusters by:
for each object cluster in the extracted one or more object clusters,
generating a minimum bounding box for the object cluster; and
determining whether the object cluster is the target object by comparing the dimensions of the generated minimum bounding box against the dimensions of the target object.
17. A system for identifying and tracking a target object being a person in an operating room (OR) and de-identifying a color video captured by a 2D camera, the system comprising:
a depth camera installed in the OR;
a 2D camera installed in the OR;
one or more processors coupled to the depth camera and to the 2D camera;
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a 2D color image captured by the 2D camera and a depth image among a sequence of depth images captured by the depth camera both cameras operating simultaneously;
identify a set of potential target points in the received depth image that potentially belongs to the target object based on one or more target object criteria;
extract one or more object clusters from the set of potential target points using a data-point clustering technique;
identify the target object from the extracted one or more object clusters as a body of a person and extract a 3D body contour from the extracted one or more clusters;
project the 3D body contour into a 2D body outline in the received 2D color image;
overlay the 2D body outline onto the received 2D color image; and
de-identify the received 2D color image based on the overlaid 2D body outline.
18. The system of claim 17 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:
generate a set of 3D body joints for the extracted 3D body contour;
project the set of 3D body joints into a 2D skeleton figure in the received 2D color image; and
overlay the 2D skeleton figure onto the received 2D color image, wherein de-identifying the received 2D color image is further based on the overlaid 2D skeleton figure.