Surgical system with augmented reality display

1. A surgical system comprising:

a detector, comprising an array of pixels configured to detect light reflected by a surgical environment that includes a surgical stapler comprising a jaw and a shaft, and generate a first signal comprising a first dataset representative of a visible image of the surgical environment and the surgical stapler; and

a processor configured to:

receive the first signal and a second signal representative of one or more operating parameters of the surgical stapler,

determine an orientation of the surgical stapler,

identify the surgical stapler from the first signal based on a predetermined orientation between the detector and the surgical stapler;

identify the shaft of the surgical stapler from the first signal by comparing the identified surgical stapler with an image of the surgical stapler in a database,

determine a loading status associated with the surgical stapler;

generate a modified image in real-time that includes at least a portion of the surgical environment and the surgical stapler, and information related to the one or more operating parameters of the surgical stapler, wherein a location of the information related to the one or more operating parameters of the surgical stapler in the modified image is superimposed to be on or adjacent to an image of surgical stapler and is oriented based on the orientation of the surgical stapler,

wherein an attribute of the identified shaft is determined based on the loading status associated with the surgical stapler; and

update the location of the information related to the one or more operating parameters in the modified image based on a motion of the surgical stapler in the surgical environment in real-time.

2. The system of claim 1 , further comprising a display device configured to display the modified image, wherein the display device includes at least one of a monitor and a display integrated into an accessory worn by a surgeon.

3. The system of claim 1 , wherein the modified image replaces one or more of at least a portion of the surgical stapler and at least a portion of the surgical field.

4. The surgical system of claim 1 , wherein the first dataset comprises values representing color of the light detected by one or more pixels of the first array of pixels.

5. The system of claim 1 , wherein the processor is configured to identify data representative of the surgical instrument from the first dataset to determine orientation of the surgical instrument based on position of the detector with respect to the surgical instrument.

6. The system of claim 1 , wherein the surgical stapler comprises one or more markers, each of the one or more markers configured to reflect a predetermined frequency of light.

7. The system of claim 6 , wherein the processor filters the first signal to determine orientation of the surgical stapler through an image recognition algorithm based on location of the one or more markers.

8. The system of claim 1 , wherein the surgical system is a robotic surgical system and the robotic surgical system comprises at least one robotic arm configured to hold and manipulate the surgical stapler.

9. The system of claim 1 , wherein the surgical stapler has a functionality including at least one of cutting, stapling, and energy delivery.

10. The system of claim 1 , wherein the one or more operating parameters includes at least one of articulation angle, shaft rotation angle, position of a cutting blade, motion of the cutting blade, tissue location, and reload information.

11. A surgical system comprising:

a first detector, comprising a first array of pixels configured to detect light reflected by a surgical device within a surgical environment having one or more regions with a predetermined color, the first array of pixels further being configured to generate a first signal comprising a first dataset representative of a first visible image;

a second detector, comprising a second array of pixels configured to detect light reflected by the surgical environment and generate a second signal comprising a second dataset representative of a second visible image; and

a processor configured to:

receive the first and second signals,

identify, from the first dataset, a first sub-dataset representative of the one or more regions of the surgical device having the predetermined color, and

identify, from the second dataset, a second sub-dataset representative of the one or more regions of the surgical device having the predetermined color,

generate a first modified image of the surgical environment, wherein a portion of the first visible image associated with the identified first sub-dataset is replaced with an image of underlying tissue in the surgical environment;

generate a second modified image of the surgical environment, wherein a portion of the second visible image associated with the identified second sub-dataset is replaced with an image of underlying tissue in the surgical environment; and

provide the first modified image to a first portion of a head mounted display associated with a first eye of a user, and provide the second modified image to a second portion of the head mounted display associated with a second eye of the user.

12. The surgical system of claim 11 , wherein the predetermined color is green.

13. The surgical system of claim 11 , wherein the first dataset comprises values representing color of the light detected by one or more pixels of the first array of pixels and the second dataset comprises values representing color of the light detected by one or more pixels of the second array of pixels.

14. The system of claim 11 , further comprising a display device configured to display the modified image.

15. A surgical method comprising:

detecting light reflected by a surgical environment that includes a surgical stapler comprising a jaw and a shaft;

generating a first signal comprising a first dataset representative of an image of the surgical stapler and the surgical environment;

receiving the first signal and a second signal representative of one or more operating parameters of the surgical stapler;

determining an orientation of the surgical stapler;

identifying the surgical stapler from the first signal based on a predetermined orientation between the detector and the surgical stapler;

identifying the shaft of the surgical stapler from the first signal by comparing the identified surgical stapler with an image of the surgical stapler in a database;

determining a loading status associated with the surgical stapler;

generating a modified image in real-time that includes information related to one or more operating parameters of the surgical stapler and at least a portion of the surgical environment and the surgical stapler, wherein a location of the information related to the one or more operating parameters of the surgical stapler in the modified image is superimposed to be on or adjacent to an image of at least one of the jaw and the shaft and is oriented based on the orientation of the surgical stapler,

wherein an attribute of the identified shaft is determined based on the loading status associated with the surgical stapler; and

updating the location of the information related to the one or more operating parameters in the modified image based on a motion of the surgical stapler in the surgical environment in real-time.

16. The surgical system of claim 1 , wherein the attribute is the color of the identified shaft in the modified image.

17. The surgical system of claim 1 , wherein the detector and the surgical stapler are coupled in the predetermined orientation.