Granted Patent B2

US 12,343,218 · App. 17/998,835 · Granted Jul 1, 2025

Inventors: Brian William Quist (Salem, NH); Nathan Anil Netravali (Littleton, MA); Paul Alexander Torrie (Marblehead, MA)

Assignees: SMITH & NEPHEW, INC.; SMITH & NEPHEW ORTHOPAEDICS AG; SMITH & NEPHEW ASIA PACIFIC PTE. LIMITED

A61B90/36A61B34/10A61B34/20A61B34/25A61B2034/105A61B2034/2065A61B2034/252A61B2090/365A61B2090/366A61B2090/368G06T2207/30008

Abstract

Claims (58)

1. A method for registration of arthroscopic video images to preoperative models, the method implemented by one or more surgical computing devices and comprising:

applying a machine learning model to diagnostic video data captured via an arthroscope to identify an anatomical structure represented in the diagnostic video data;

registering one of a plurality of anatomical structures in a three-dimensional (3D) anatomical model to the anatomical structure represented in the diagnostic video data, wherein the 3D anatomical model is generated from preoperative image data;

tracking the anatomical structure intraoperatively based on the registration;

generating a simulated projected view of the registered one of the plurality of anatomical structures from the 3D anatomical model based on a determined orientation of the arthroscope during capture by the arthroscope of intraoperative video data; and

outputting the simulated projected view scaled and oriented based on one or more landmark features of the anatomical structure extracted from the intraoperative video data.

2. The method of claim 1 , further comprising:

generating an overlay comprising the scaled and oriented simulated projected view;

merging the generated overlay with the intraoperative video data based on the registration to generate merged video data; and

outputting to a display device the merged video data.

3. The method of claim 2 , further comprising determining a stage of a surgical procedure based on an obtained surgical plan for the surgical procedure and identification of the anatomical structure in the intraoperative video data, wherein the generated overlay further comprises guidance extracted from the obtained surgical plan.

4. The method of claim 3 , wherein the guidance comprises one or more of:

textual directions associated with a current task in the surgical procedure; or

a visual indication of another one of the plurality of anatomical structures corresponding to a subsequent task in the surgical procedure.

5. The method of claim 2 , further comprising obtaining an annotated version of the 3D anatomical model or the preoperative image data that identifies an anatomical point corresponding to a portion of patient anatomy or at least one of the one or more landmark features, wherein the generated overlay further comprises an indication of the anatomical point.

6. The method of claim 2 , wherein the display device comprises a mixed reality headset and the method further comprises:

tracking one or more of a position or an orientation of the mixed reality headset; and

generating the overlay using a field of view of the arthroscope to determine a local reference frame and based on a known spatial and scale relationship between the arthroscope field of view and another reference frame of the mixed reality headset determined based on the tracking.

7. The method of claim 1 , further comprising training the machine learning model based on additional video data comprising a plurality of image frames each comprising at least one annotated representation of one or more of the plurality of anatomical structures.

8. The method of claim 1 , further comprising:

determining an eye position of a user; and

outputting to a projector the scaled and oriented simulated projected view for projection by the projector onto patient skin based on the determined eye position.

9. The method of claim 1 , wherein the anatomical structure represented in the intraoperative video data comprises soft tissue and the method further comprises:

determining a size and position of a first portion of the soft tissue from the intraoperative video data; and

applying another machine learning model to the 3D anatomical model and the determined size and position to generate a representation of a second portion of the soft tissue in a morphed state, wherein the simulated projected view comprises the representation of the second portion of the soft tissue in the morphed state.

10. The method of claim 1 , further comprising:

generating a weighting value for each of a plurality of portions of the 3D anatomical model; and

generating the simulated projected view to include a subset of the plurality of portions based on the weighting values.

11. A surgical computing device, comprising:

a non-transitory computer-readable medium comprising programmed instructions stored thereon for registration of arthroscopic video images to preoperative models; and

one or more processors coupled to the non-transitory computer-readable medium and configured to execute the stored programmed instructions which causes the one or more processors to:

apply a machine learning model to diagnostic video data captured via an arthroscope to identify an anatomical structure represented in the diagnostic video data;

register one of a plurality of anatomical structures in a three-dimensional (3D) anatomical model to the anatomical structure represented in the diagnostic video data, wherein the 3D anatomical model is generated from preoperative image data;

track the anatomical structure intraoperatively based on the registration;

generate a simulated projected view of the registered one of the plurality of anatomical structures from the 3D anatomical model based on a determined orientation of the arthroscope during capture by the arthroscope of intraoperative video data; and

output the simulated projected view scaled and oriented based on one or more landmark features of the anatomical structure extracted from the intraoperative video data.

12. The surgical computing device of claim 11 , wherein the stored programmed instructions further cause the one or more processors to:

generate an overlay comprising the scaled and oriented simulated projected view;

merge the generated overlay with the intraoperative video data based on the registration to generate merged video data; and

output to a display device the merged video data.

13. The surgical computing device of claim 12 , wherein the stored programmed instructions further cause the one or more processors to determine a stage of a surgical procedure based on an obtained surgical plan for the surgical procedure and identification of the anatomical structure in the intraoperative video data, wherein the generated overlay further comprises guidance extracted from the obtained surgical plan.

14. The surgical computing device of claim 13 , wherein the guidance comprises one or more of:

textual directions associated with a current task in the surgical procedure; or

a visual indication of another one of the plurality of anatomical structures corresponding to a subsequent task in the surgical procedure.

15. The surgical computing device of claim 12 , wherein the stored programmed instructions further cause the one or more processors to obtain an annotated version of the 3D anatomical model or the preoperative image data that identifies an anatomical point corresponding to a portion of patient anatomy or at least one of the one or more landmark features, wherein the generated overlay further comprises an indication of the anatomical point.

16. The surgical computing device of claim 12 , wherein the display device comprises a mixed reality headset and the stored program instructions further cause the one or more processors to:

track one or more of a position or an orientation of the mixed reality headset; and

generate the overlay using a field of view of the arthroscope to determine a local reference frame and based on a known spatial and scale relationship between the arthroscope field of view and another reference frame of the mixed reality headset determined based on the tracking.

17. The surgical computing device of claim 11 , wherein the stored programmed instructions further cause the one or more processors to train the machine learning model based on additional video data comprising a plurality of image frames each comprising at least one annotated representation of one or more of the plurality of anatomical structures.

18. The surgical computing device of claim 11 , wherein the stored programmed instructions further cause the one or more processors to:

determine an eye position of a user; and

output to a projector the scaled and oriented simulated projected view for projection by the projector onto patient skin based on the determined eye position.

19. The surgical computing device of claim 11 , wherein the anatomical structure represented in the intraoperative video data comprises soft tissue and the stored programmed instructions further cause the one or more processors to:

determine a size and position of a first portion of the soft tissue from the intraoperative video data; and

apply another machine learning model to the 3D anatomical model and the determined size and position to generate a representation of a second portion of the soft tissue in a morphed state, wherein the simulated projected view comprises the representation of the second portion of the soft tissue in the morphed state.

20. The surgical computing device of claim 11 , wherein the stored programmed instructions further cause the one or more processors to:

generate a weighting value for each of a plurality of portions of the 3D anatomical model; and

generate the simulated projected view to include a subset of the plurality of portions based on the weighting values.

Assignments (2)

Continuity (3)

References Cited (11)