Cooperative composite video streams layered onto the surgical site and instruments

1. A computing system comprising:

a processor configured to:

obtain a surgical video stream associated with a surgical procedure;

obtain overlay content associated with the surgical procedure;

identify a marker in a first frame of the surgical video stream;

determine a first overlay region location in the first frame of the surgical video stream for overlaying the overlay content based on the marker in the first frame of the surgical video stream;

identify the marker in a second frame of the surgical video stream;

determine a second overlay region location in the second frame of the surgical video stream for overlaying the overlay content based on the marker in the second frame of the surgical video stream; and

generate a composite video stream based on the first overlay region location and the second overlay region location.

2. The computing system of claim 1 , wherein the surgical video stream comprises a video feed of a surgical site from a laparoscope, and the processor is configured to adjust a location and orientation of the overlay content on the surgical site as the laparoscope moves.

3. The computing system of claim 1 ,

wherein the overlay content comprises a secondary surgical video stream associated with the surgical procedure, and

where the processor is further configured to:

extract a portion of the secondary surgical video stream that comprises a region of interest;

overlay the portion of the secondary surgical video stream onto the first frame of the surgical video stream at the first overlay region location; and

overlay the portion of the secondary surgical video stream onto the second frame of the surgical video stream at the second overlay region location.

4. The computing system of claim 1 , wherein the overlay content comprises at least one of:

a surgical instrument operation status;

a surgical instrument blade temperature;

a surgical instrument loading condition; or

steps-for-use instructions associated with a surgical instrument.

5. The computing system of claim 1 , wherein the overlay content comprises a supplementary image of an organ associated with the surgical procedure.

6. The computing system of claim 1 , wherein the marker in the first frame of the surgical video stream corresponds to a fiducial marker on a surgical instrument.

7. The computing system of claim 1 , wherein the processor is further configured to:

determine a first overlay region size for overlaying the overlay content onto the first frame based on content of the first frame of the surgical video stream;

determine a second overlay region size for overlaying the overlay content onto the second frame based on content of the second frame of the surgical video stream; and

scale the overlay content based on the first overlay region size and the second overlay region size.

8. The computing system of claim 7 , wherein the processor is further configured to:

identify that the marker in the first frame of the surgical video stream is a fiducial marker;

determine a size of the fiducial marker in the first frame;

calculate the first overlay region size based on the size of the fiducial marker in the first frame;

identify that the marker in the second frame of the surgical video stream is the fiducial marker;

determine a size of the fiducial marker in the second frame; and

calculate the second overlay region size based on the size of the fiducial marker in the second frame.

9. The computing system of claim 1 , wherein the processor is further configured to:

identify that the marker in the first frame of the surgical video stream is a fiducial marker;

determine an orientation of the fiducial marker in the first frame;

determine a first overlay orientation for overlaying the overlay content onto the first frame based on the orientation of the fiducial marker in the first frame;

identify that the marker in the second frame of the surgical video stream is the fiducial marker;

determine an orientation of the fiducial marker in the second frame; and

determine a second overlay orientation for overlaying the overlay content onto the second frame based on the orientation of the fiducial marker in the second frame.

10. A method comprising:

obtaining a surgical video stream associated with a surgical procedure;

obtaining overlay content associated with the surgical procedure;

identifying a marker in a first frame of the surgical video stream;

determining a first overlay region location in the first frame of the surgical video stream for overlaying the overlay content based on the marker in the first frame of the surgical video stream;

identify the marker in a second frame of the surgical video stream;

determining a second overlay region location in the second frame of the surgical video stream for overlaying the overlay content based on the marker in the second frame of the surgical video stream; and

generating a composite video stream based on the first overlay region location and the second overlay region location.

11. The method of claim 10 , wherein the surgical video stream comprises a sequence of images of a surgical site, the sequence of images being captured via a laparoscope, and method further comprises:

adjusting a location and orientation of the overlay content on the surgical site in the sequence of images as the laparoscope moves.

12. The method of claim 10 ,

wherein the overlay content comprises a secondary surgical video stream associated with the surgical procedure, and

wherein the method further comprises:

extracting a portion of the secondary surgical video stream that comprises a region of interest;

overlaying the portion of the secondary surgical video stream onto the first frame of the surgical video stream at the first overlay region location; and

overlaying the portion of the secondary surgical video stream onto the second frame of the surgical video stream at the second overlay region location.

13. The method of claim 10 , wherein the overlay content comprises at least one of:

a surgical instrument operation status;

a surgical instrument blade temperature;

a surgical instrument loading condition; or

steps-for-use instructions associated with a surgical instrument.

14. The method of claim 10 , wherein the overlay content comprises a supplementary image of an organ associated with the surgical procedure.

15. The method of claim 10 , wherein the marker in the first frame of the surgical video stream corresponds to a fiducial marker on a surgical instrument.

16. The method of claim 10 , further comprising:

determining a first overlay region size for overlaying the overlay content onto the first frame based on content of the first frame of the surgical video stream;

determining a second overlay region size for overlaying the overlay content onto the second frame based on content of the second frame of the surgical video stream; and

scaling the overlay content based on the first overlay region size and the second overlay region size.

17. The method of claim 16 , further comprising:

identifying that the marker in the first frame of the surgical video stream is a fiducial marker;

determining a size of the fiducial marker in the first frame;

calculating the first overlay region size based on the size of the fiducial marker in the first frame;

identifying that the marker in the second frame of the surgical video stream is the fiducial marker;

determining a size of the fiducial marker in the second frame; and

calculating the second overlay region size based on the size of the fiducial marker in the second frame.

18. The method of claim 10 , further comprising:

identifying that the marker in the first frame of the surgical video stream is a fiducial marker;

determining an orientation of the fiducial marker in the first frame;

determining a first overlay orientation for overlaying the overlay content onto the first frame based on the orientation of the fiducial marker in the first frame;

identifying that the marker in the second frame of the surgical video stream is the fiducial marker;

determining an orientation of the fiducial marker in the second frame; and

determining a second overlay orientation for overlaying the overlay content onto the second frame based on the orientation of the fiducial marker in the second frame.