Automatic de-identification of operating room (OR) videos based on depth images
Embodiments described herein provide systems and techniques for tracking and de-identifying person in a captured operating room (OR) video. In one aspect, a process for de-identifying OR personnel in an OR video begins by simultaneously receiving a color image captured by an RGB camera and a depth image captured by a depth camera installed in the vicinity of the RGB camera. The process then generates a 3D point cloud based on the depth image. Next, the process applies a human-body detector to the 3D point cloud to detect a set of 3D bodies in the 3D point cloud, wherein each detected 3D body corresponds to a detected person in the OR. The process next projects each detected 3D body into a 2D body outline in the color image to represent the same detected person in the color image. The process subsequently de-identifies the detected people in the color image based on the projected 2D body outlines.
1. A computer-implemented method for de-identifying personnel in an operating room (OR) video, the method comprising:
simultaneously receiving a color image captured by an RGB camera installed in the OR and a depth image captured by a depth camera installed in the vicinity of the RGB camera, wherein the color image and the depth image are captured at the same or substantially the same time;
generating a three-dimensional (3D) point cloud based on the depth image;
applying a machine-learning human-body detector to the 3D point cloud to detect a set of 3D bodies in the 3D point cloud, wherein each 3D body in the set of 3D bodies corresponds to a detected person in the OR, and wherein a set of human-body keypoints for the detected person is identified based on a set of extracted shapes and a set of computed orientations of the set of extracted shapes identified in a 3D-point cluster of the 3D point cloud;
projecting each detected 3D body in the set of detected 3D bodies, utilizing the set of human-body keypoints of each 3D body, into a two-dimensional (2D) body outline in the color image including personal identifiable information (PII) to represent the same detected person in the color image; and
de-identifying the set of detected people by removing the PII in the color image based at least on the corresponding set of projected 2D body outlines.
2. The computer-implemented method of claim 1 , wherein generating the 3D point cloud based on the received depth image includes projecting each 2D pixel (u, v) and the corresponding distance value d(u, v) in the depth image into a 3D point in a 3D-coordinate system aligned with the depth camera based at least on a known lens model of the depth camera.
3. The computer-implemented method of claim 1 , wherein detecting the set of 3D bodies in the 3D point cloud using the machine-learning human-body detector includes detecting a set of 3D body contours by:
applying a data-point clustering technique to the 3D point cloud to identify a plurality of 3D-point clusters that potentially represent objects and people in the OR;
identifying a subset of the plurality of 3D-point clusters that have high probabilities to represent human bodies; and
for each identified 3D-point cluster in the subset of the 3D-point clusters, extracting a corresponding 3D body contour by identifying a boundary surface of the identified 3D-point cluster.
4. The computer-implemented method of claim 3 , wherein projecting each detected 3D body in the set of detected 3D bodies into the corresponding 2D body outline in the color image includes transforming each 3D point in the extracted 3D body contour from the coordinate system of the depth camera to a pair of 2D-coordinates in the coordinate system of the RGB camera.
5. The computer-implemented method of claim 1 , wherein identifying the set of human-body keypoints for the detected person includes:
extracting the set of shapes from the identified 3D-point cluster; and
computing the set of orientations associated with the set of extracted shapes.
6. The computer-implemented method of claim 5 , wherein projecting each detected 3D body in the set of detected 3D bodies into the corresponding 2D body outline in the color image further includes:
generating a skeleton figure of the detected person based on the set of human-body keypoints of the detected person; and
overlaying the skeleton figure onto the corresponding 2D body outline of the detected person in the color image.
7. The computer-implemented method of claim 6 , wherein processing the color image to de-identify the set of detected people in the color image further includes:
for a given detected person in the set of detected people,
identifying one or more parts of the full body of the detected person that are known or likely to contain PII based on the set of human-body keypoints in the skeleton figure and the 2D body outline; and
blurring out or otherwise obfuscating portions of the color image corresponding to the one or more identified parts of the full body to de-identify the detected person.
8. The computer-implemented method of claim 7 , wherein the one or more parts of the full body include a face of the detected person and a portion of the torso of the detected person which normally contains a name tag.
9. The computer-implemented method of claim 1 , wherein processing the color image to de-identify the set of detected people in the color image further includes blurring out or otherwise obfuscating portions of the color image inside the set of projected 2D body outlines of the set of detected people.
10. The computer-implemented method of claim 1 , wherein prior to transforming each detected 3D body from the depth image into the 2D body outline in the color image, the method further comprises independently calibrating each of the depth camera and the RGB camera to obtain a first calibrated lens model for the depth camera and a second calibrated lens model for the RGB camera.
11. The computer-implemented method of claim 1 , wherein the received color image is among a sequence of color images captured by the RGB camera during a time period of a surgical procedure, and the received depth image is among a sequence of depth images captured by the depth camera during the same time period, wherein the method further comprises:
processing the sequence of depth images to generate a sequence of 2D body outlines of a detected person;
tracking the detected person through the sequence of color images based on the locations of the sequence of 2D body outlines of the detected person in the sequence of color images; and
generating an OR workflow notification when the detected person is determined to have exited the OR.
12. The computer-implemented method of claim 1 , wherein projecting each detected 3D body in the set of detected 3D bodies into the corresponding 2D body outline in the color image further includes:
projecting a set of body joints identified in the 3D-point cluster into the 2D body outline in the color image.
13. A system for de-identifying personnel in an operating room (OR) video, the system comprising:
one or more processors;
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to;
simultaneously receive a color image captured by an RGB camera installed in the OR and a depth image captured by a depth camera installed in the vicinity of the RGB camera, wherein the color image and the depth image are captured at the same or substantially the same time;
generate a three-dimensional (3D) point cloud based on the depth image;
apply a machine-learning human-body detector to the 3D point cloud to detect a set of 3D bodies in the 3D point cloud, wherein each 3D body in the set of 3D bodies corresponds to a detected person in the OR, and wherein a set of human-body keypoints for the detected person is identified based on a set of extracted shapes and a set of computed orientations of the set of extracted shapes identified in a 3D-point cluster of the 3D point cloud;
project each detected 3D body in the set of detected 3D bodies, utilizing the set of human-body keypoints of each 3D body, into a two-dimensional (2D) body outline in the color image including personal identifiable information (PII) to represent the same detected person in the color image; and
de-identify the set of detected people in the color image based at least on the corresponding set of projected 2D body outlines.
14. The system of claim 13 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to generate the 3D point cloud by projecting each 2D pixel (u, v) and the corresponding distance value d(u, v) in the depth image into a 3D point in a 3D-coordinate system aligned with the depth camera based at least on a known lens model of the depth camera.
15. The system of claim 13 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to detect a set of 3D body contours from the 3D point cloud by:
applying a data-point clustering technique to the 3D point cloud to identify a plurality of 3D-point clusters that potentially represent objects and people in the OR;
identifying a subset of the plurality of 3D-point clusters that have high probabilities to represent human bodies; and
for each identified 3D-point cluster in the subset of the 3D-point clusters, extracting a corresponding 3D body contour by identifying a boundary surface of the identified 3D-point cluster.
16. The system of claim 15 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to project each detected 3D body in the set of detected 3D bodies into the corresponding 2D body outline in the color image by transforming each 3D point in the extracted 3D body contour from the coordinate system of the depth camera to a pair of 2D-coordinates in the coordinate system of the RGB camera.
17. The system of claim 15 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the steps of:
extracting the set of shapes from the identified 3D-point cluster; and
computing the set of orientations associated with the set of extracted shapes.
18. The system of claim 17 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to:
generate a skeleton figure of the detected person based on the set of body keypoints of the detected person; and
overlay the skeleton figure onto the corresponding 2D body outline of the detected person in the color image.
19. The system of claim 18 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to de-identify the set of detected people in the color image by:
for a given detected person in the set of detected people,
identifying one or more parts of the full body of the detected person that are known or likely to contain PII based on the set of body keypoints in the skeleton figure and the 2D body outline; and
blurring out or otherwise obfuscating portions of the color image corresponding to the one or more identified parts of the full body to de-identify the detected person.
20. A computer-implemented method for tracking personnel in an operating room (OR), the method comprising:
simultaneously receiving a color image in a sequence of color images captured by an RGB camera installed in the OR and a depth image in a sequence of depth images captured by a depth camera installed in the vicinity of the RGB camera, wherein the color image and the depth image are captured at the same or substantially the same time;
generating a three-dimensional (3D) point cloud based on the received depth image;
applying a machine-learning human-body detector to the 3D point cloud to detect a set of 3D bodies in the 3D point cloud, wherein the set of detected 3D bodies includes a given 3D body corresponds to a given detected person in the OR, and wherein a set of human-body keypoints for the detected person is identified based on a set of extracted shapes and a set of computed orientations of the set of extracted shapes identified in a 3D-point cluster of the 3D point cloud;
projecting the given 3D body in the set of detected 3D bodies, utilizing the set of human-body keypoints of each 3D body, into a 2D body outline in the received color image including personal identifiable information (PII) to represent the location and the body geometry of the given detected person in the received color image; and
tracking the given detected person in the OR based on a sequence of locations associated with a sequence of projected 2D body outlines of the given detected person in the sequence of color images.
21. The computer-implemented method of claim 20 , wherein the method further comprises de-identifying the given detected person by blurring out or otherwise obfuscating portions of the color images inside the sequence of projected 2D body outlines of the given detected person in the sequence of color images.