Granted Patent B2

US 12,638,963 · App. 18/077,438 · Granted May 26, 2026

Inventors: Korry D. Kobel (Oshkosh, WI); Doug Shoup (Oshkosh, WI)

Assignee: OSHKOSH CORPORATION

G06F3/04845G01N21/8851G06F3/0482G06K7/1408G06Q30/0633G06T19/006

Abstract

Claims (59)

1 . A method for inspecting a machine using an augmented reality (AR) application, the method comprising:

providing, by the AR application, a user interface to a user device;

receiving, by the AR application, a video feed of the machine from a camera;

displaying, by the AR application, the video feed on the user interface;

aligning, by the AR application, a digital twin of the machine with the video feed of the machine on the user interface, the digital twin comprising a sensor reading from each of one or more sensors embedded in the machine; and

detecting, by the AR application after the digital twin is aligned with the video feed of the machine, that a first component of the machine is out of place relative to the other components of the machine by detecting that a corresponding component of the digital twin does not align with the first component in the video feed.

2 . The method of claim 1 , further comprising:

detecting, by the AR application, an identity of the machine via the video feed; and

matching, by the AR application, the identity of the machine with a first set of stored machine information, the first set of stored machine information comprising the digital twin.

3 . The method of claim 2 , wherein detecting the identity of the machine comprises using image processing to detect (a) a decal on the machine (b) a QR code affixed to the machine (c) a barcode affixed to the machine or (d) the shape of the machine.

4 . The method of claim 1 , further comprising displaying, by the AR application, one or more of the sensor readings on the video feed of the machine on the user interface.

5 . The method of claim 1 , further comprising:

receiving, by the AR application via the user interface, a selection of a machine function; and

displaying and emphasizing, by the AR application on the user interface in response to the selection, components of the digital twin associated with the selected machine function.

6 . The method of claim 1 , further comprising:

receiving, by the AR application via the user interface, a selection of a component of the digital twin; and

displaying, by the AR application on the user interface in response to the selection of the component, component data associated with a real-world counterpart of the selected component, the component data comprising one or more sensor readings associated with the real-world counterpart.

7 . The method of claim 6 , wherein the component data further comprises one or more of: an expected sensor range associated with each of the one or more sensor readings, an expected service life of the selected component, or a service age of the selected component.

8 . The method of claim 1 , further comprising:

detecting, by the AR application, a potential defect; and

displaying, by the AR application on the user interface in response to detecting the potential defect, an alert;

wherein detecting the potential defect comprises detecting one or more of: a service age of a component of the machine exceeding an expected service life of the component, or detecting that first sensor measurement is outside of an expected sensor range for the first sensor measurement.

9 . The method of claim 8 , further comprising at least one of (a) displaying and emphasizing, by the AR application, components associated with the potential defect on the user interface or (b) displaying, by the AR application, instructions for repairing the potential defect.

10 . The method of claim 1 , further comprising providing, by the AR application, a notification to a user in response to detecting that the first component of the machine is out of place.

11 . The method of claim 1 , further comprising:

receiving, by the AR application via the user interface, a selection of an inspect-able feature;

displaying, by the AR application on the user interface in response to the selection, instructions for inspecting the inspect-able feature; and

displaying and emphasizing, by the AR application on the user interface in response to the selection, components of the digital twin associated with inspecting the inspect-able feature on the user interface.

12 . A system for inspecting a machine using augmented reality (AR), the system comprising:

a user device configured to display a user interface; and

a server comprising a processing circuit comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:

receive a video feed of the machine from a camera;

display the video feed on the user interface;

overlay a digital twin of the machine onto the video feed of the machine on the user interface, the digital twin comprising information associated with a real-world component of the machine;

receive, via the user interface, a selection of an inspect-able component of the machine;

identify a component of the digital twin corresponding to the selected inspect-able component of the machine;

display, on the user interface in response to the selection, instructions for inspecting the inspect-able component, the instructions for inspecting the inspect-able component identifying an inspection tool and a sub-component of the inspect-able component to be engaged by the inspection tool; and

display and emphasize, on the user interface in response to the selection, the identified sub-component of the inspect-able component on the user interface.

13 . The system of claim 12 , wherein the information comprises at least one of an age of the component, a sensor reading associated with the component, a manufacture date of the component, an installation date of the component, a number of hours of active use of the component, an expiration date of the component, or a repair date of the component.

14 . The system of claim 12 , wherein the one or more processors are further configured to:

determine, based on the information, that one of an age of the component or a number of hours of active use of the component exceeds a threshold value; and

display an alert on the user interface in response to determining that the age or the number of hours exceeds the threshold value.

15 . The system of claim 14 , wherein the alert comprises a selectable link to purchase a replacement component or add the replacement component to a digital shopping cart.

16 . A building equipment inspection tool for inspecting equipment via augmented reality (AR), the tool comprising a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to implement operations comprising:

providing an AR application to a user interface;

receiving a video feed of a machine from a camera;

displaying the video feed on the user interface;

overlaying a digital twin of the machine onto the video feed of the machine on the user interface;

receiving, via the user interface and from a menu remote from the digital twin, a selection of a function performable by a hydraulic system of the machine;

displaying and emphasizing, on the user interface in response to the selection, components of the digital twin corresponding to hydraulic components of the machine configured to execute the selected function; and

detecting that a first component of the machine is out of place by detecting that a corresponding component of the digital twin does not align with the first component.

17 . The building equipment inspection tool of claim 16 , wherein the operations further comprise:

receiving, via the user interface, a selection of a component of the digital twin; and

displaying, on the user interface in response to the selection of the component, component data associated with a real-world counterpart of the selected component, the component data comprising one or more sensor readings associated with the real-world counterpart.

18 . The building equipment inspection tool of claim 16 , wherein the operations further comprise:

receiving, via the user interface, a selection of an inspect-able feature;

displaying, on the user interface in response to the selection, instructions for inspecting the inspect-able feature; and

displaying and emphasizing, on the user interface in response to the selection, components of the digital twin associated with inspecting the inspect-able feature on the user interface.

19 . The building equipment inspection tool of claim 16 , wherein the selected function is performable by a hydraulic system of the machine, and the displayed and emphasized components of the digital twin correspond to hydraulic components of the hydraulic system configured to execute the selected function.

Assignments (1)

Continuity (3)

References Cited (36)