JOINT ANGLE DETERMINATION UNDER LIMITED VISIBILITY
Provided herein are methods and systems for determining joint information, such as joint angle and range of movement, quickly and accurately without using cumbersome, specialized equipment even when visibility to the joint may be limited. Such methods and systems may be achieved by using computationally efficient approaches that are accurate and robust enough to handle noises and occlusions. The methods and systems provided herein may provide quick, low-cost, on-demand approaches to obtain information about the joint health of a patient. The joint angle information may facilitate diagnosis, treatment, prognosis, and/or rehabilitation of the patient from a joint injury, pain, or discomfort by providing quick, actionable information to the healthcare provider.
1 . A computer-implemented method for determining an angle in an object of interest, the method comprising:
(a) obtaining an image or a video of the object of interest;
(b) generating a plurality of key point heatmaps and a plurality of segment heatmaps from the image or the video;
(c) blending at least one of the plurality of key point heatmaps with at least one of the plurality of segment heatmaps to generate at least one blended heatmap;
(d) extracting features from the at least one blended heatmap; and
(e) determining the angle in the object of interest by calculating an angle formed by the extracted features.
2 . The method of claim 1 , wherein the extracted features comprise key points extracted from the at least one blended heatmap, wherein the angle formed by the extracted key points is defined by at least two extracted segments formed by connecting the extracted key points.
3 . The method of claim 1 , wherein the extracted features comprise segments extracted from the plurality of segment heatmaps, wherein the angle formed by the extracted segments.
4 . The method of claim 1 , wherein the extracted features comprise segments extracted from the at least one blended heatmap, wherein the angle formed by the extracted segments.
5 . The method of any one of claims 3 - 4 , wherein the extracted segment extracted using a line detection method.
6 . The method of any one of claims 3 - 5 , wherein the extracted segment extracted using Hough transform.
7 . The method of any one of claims 1 - 6 , wherein the object of interest comprises a joint of a subject.
8 . The method of claim 7 , wherein the joint comprises a knee joint, a hip joint, an ankle joint, an elbow joint, or a shoulder joint.
9 . The method of claim 8 , wherein the knee joint comprises lateral epicondyle.
10 . The method of claim 8 , wherein the hip joint comprises greater trochanter.
11 . The method of claim 8 , wherein the ankle joint comprises lateral malleolus.
12 . The method of any one of claims 1 - 11 , further comprising generating an output comprising the angle in the object of interest.
13 . The method of any one of claims 1 - 12 , wherein generating the plurality of key point heatmaps and the plurality of segment heatmaps in step (b) uses a deep neural network.
14 . The method of any one of claims 1 - 13 , wherein the deep neural network comprises convolutional networks.
15 . The method of any one of claims 1 - 14 , wherein the deep neural network comprises convolutional pose machine.
16 . The method of any one of claims 1 - 15 , wherein the deep neural network comprises a rectified linear unit (ReLU) activation function.
17 . The method of any one of claims 1 - 16 , wherein the plurality of key point heatmaps represents landmarks on the image or the video of the object of interest.
18 . The method of any one of claims 1 - 17 , wherein the landmarks comprise a joint and at least one body part adjacent to the joint.
19 . The method of any one of claims 1 - 18 , wherein the plurality of segment heatmaps represents segments along a body part adjacent to a joint.
20 . The method of any one of claims 1 - 19 , wherein one of the segments connects at least two of the landmarks along a body part adjacent to a joint.
21 . The method of any one of claims 1 - 20 , wherein step (b) further comprises generating a combined negative heatmap from the image or the video for training the deep neural network.
22 . The method of any one of claims 1 - 21 , wherein step (c) blends the at least one of the plurality of key point heatmaps that represents a key point spatially adjacent to a segment represented by the at least one of the plurality of segment heatmaps.
23 . The method of any one of claims 1 - 22 , wherein blending comprises taking an average of pixel intensity at each corresponding coordinate of at least one of the plurality of key point heatmaps and at least one of the plurality of segment heatmaps.
24 . The method of any one of claims 1 - 23 , wherein blending provides improved handling of a noisy heatmap or a missing heatmap.
25 . The method of any one of claims 1 - 24 , wherein extracting the key points in step (d) uses at least one of non-maximum suppression, blob detection, or heatmap sampling.
26 . The method of any one of claims 1 - 25 , wherein extracting the key points comprises selecting coordinates with highest pixel intensity in the at least one blended heatmap.
27 . The method of any one of claims 1 - 26 , wherein at least three key points are extracted.
28 . The method of any one of claims 1 - 27 , wherein the plurality of key point heatmaps or the plurality of segment heatmaps comprises at least two heatmaps.
29 . A computer-based system for determining an angle in an object of interest, the system comprising:
(a) a processor;
(b) a non-transitory medium comprising a computer program configured to cause the processor to:
(i) obtain an image or a video of the object of interest and input the image or the video into a computer program;
(ii) generate, using the computer program a plurality of key point heatmaps and a plurality of segment heatmaps from the image or the video;
(iii) blend, using the computer program, at least one of the plurality of key point heatmaps with at least one of the plurality of segment heatmaps to generate at least one blended heatmap;
(iv) extract, using the computer program, features from the at least one blended heatmap; and
(v) determine, using the computer program, the angle in the object of interest by calculating an angle formed by the extracted features.
30 . The system of claim 29 , wherein the extracted features comprise key points extracted from the at least one blended heatmap, wherein the angle formed by the extracted key points is defined by at least two extracted segments formed by connecting the extracted key points.
31 . The system of claim 29 , wherein the extracted features comprise segments extracted from the plurality of segment heatmaps, wherein the angle formed by the extracted segments.
32 . The system of claim 29 , wherein the extracted features comprise segments extracted from the at least one blended heatmap, wherein the angle formed by the extracted segments.
33 . The system of any one of claims 31 - 32 , wherein the extracted segment extracted using a line detection method.
34 . The system of any one of claims 31 - 33 , wherein the extracted segment extracted using Hough transform.
35 . The system any one of claims 29 - 34 , wherein the object of interest comprises a joint of a subject.
36 . The system of any one of claims 29 - 35 , wherein the joint comprises a knee joint, a hip joint, an ankle joint, an elbow joint, or a shoulder joint.
37 . The system of claim 36 , wherein the knee joint comprises lateral epicondyle.
38 . The system of claim 36 , wherein the hip joint comprises greater trochanter.
39 . The system of claim 36 , wherein the ankle joint comprises lateral malleolus.
40 . The system of any one of claims 29 - 39 , wherein the computer program is configured to cause the processor to generate an output comprising the angle.
41 . The system of any one of claims 29 - 40 , wherein the computer program comprises a deep neural network.
42 . The system of any one of claims 29 - 41 , wherein the deep neural network comprises convolutional networks.
43 . The system of any one of claims 29 - 42 , wherein the deep neural network comprises convolutional pose machines.
44 . The system of any one of claims 29 - 43 , wherein the deep neural network comprises a rectified linear unit (ReLU) activation function.
45 . The system of any one of claims 29 - 44 , wherein the plurality of key point heatmaps represents landmarks on the image or the video of the object of interest.
46 . The system of any one of claims 29 - 45 , wherein the plurality of landmarks comprises a joint and a body part adjacent to the joint.
47 . The system of any one of claims 29 - 46 , wherein the plurality of segment heatmaps represents segments along a body part adjacent to a joint.
48 . The system of any one of claims 29 - 47 , wherein one of the segments connects at least two of the landmarks along a body part adjacent to a joint.
49 . The system of any one of claims 29 - 48 , wherein step (b)(ii) further comprises generating a combined negative heatmap from the image or the video for training the deep neural network.
50 . The system of any one of claims 29 - 49 , wherein step (b)(iii) blends the at least one of the plurality of key point heatmaps that represents a key point spatially adjacent to a segment represented by the at least one of the plurality of segment heatmaps.
51 . The system of any one of claims 29 - 50 , wherein blending comprises taking an average intensity of pixels at each corresponding coordinate of at least one of the plurality of key point heatmaps and at least one of the plurality of segment heatmaps.
52 . The system of any one of claims 29 - 51 , wherein blending provides improved handling of a noisy heatmap or a missing heatmap.
53 . The system of any one of claims 29 - 52 , wherein extracting the key points uses at least one of non-maximum suppression, blob detection, or heatmap sampling.
54 . The system of any one of claims 29 - 53 , wherein extracting the key points comprises selecting coordinates with highest intensity in the at least one blended heatmap.
55 . The system of any one of claims 29 - 54 , wherein at least three key points are extracted.
56 . The system of any one of claims 29 - 55 , wherein the plurality of key point heatmaps or the plurality of segment heatmaps comprises at least two heatmaps.
57 . The system of any one of claims 29 - 56 , wherein the system comprises a mobile phone, a tablet, or a web application.