System and method for implementing a multimodal assistant using large language models
An embodiment of the present invention is directed to a multimodal assistant for mechanics and technicians in military warehouses using large language models. An exemplary system provides a conversational assistant to mechanics and technicians working in challenging environments, such as shop or warehouse floors with heavy industrial equipment and componentry. The innovative system may utilize multimodal large language models as well as image segmentation techniques to efficiently retrieve relevant information from asset reference materials, documentation and other sources of instructional information. The system may also use multimodal large language models to extract information from the retrieved documents and/or other data sources to provide guidance on performing discrete tasks related to the asset, such as routine maintenance, part replacement, services and/or other related actions.
1 . A computer-implemented system for providing a conversational assistant using multimodal large language models and image segmentation, the system comprising:
an interactive user interface that is configured to receive one or more inputs;
a database interface that communicates with a database that stores and manages asset data; and
a processor executing on a mobile device and coupled to the interface and the database interface, the processor further configured to perform the steps of:
receiving, via the interactive user interface, a user query and a scan of an image associated with an asset;
detecting, via a computer vision detection model, a text from the scan wherein the text is imprinted on the asset and the asset is a customized asset without a conventional serial number;
performing, via a prediction model, text recognition of the text and identifying one or more predicted texts with corresponding confidence levels;
displaying, via the interactive user interface executing on the mobile device, the one or more predicted texts comprising an asset identifier;
identifying, via an asset lookup agent applying a segmentation model, a plurality of objects associated with the asset identifier wherein each object of the plurality of objects is fed to a lookup model to identify each object against an internal asset database;
for a selected object, identifying, via the interactive user interface, a corresponding service action;
retrieving, via a reference lookup agent, a set of reference data associated with the selected object;
based on the identified service action and the selected object, generating, via an advice agent, a set of instructions and corresponding set of tools; and
providing, via the interactive user interface, a response to the user query wherein the response comprises the set of instructions and the corresponding set of tools.
2 . The system of claim 1 , wherein the asset lookup agent relies on pre-processing of the user query for consistency in one or more agent prompts.
3 . The system of claim 1 , wherein the corresponding service action comprises: repair, replace and maintenance.
4 . The system of claim 1 , wherein the set of reference data comprises one or more documents from a document database that communicates with a client inventory management system via an integration API.
5 . The system of claim 1 , wherein the set of reference data comprises instructional information that relate directly and indirectly to the asset.
6 . The system of claim 1 , wherein the response comprises an estimated time required for the identified service action.
7 . The system of claim 1 , wherein the set of instructions comprises multimedia content comprising a combination of: audio, video, images and text.
8 . The system of claim 1 , wherein the response comprises an inventory status for at least one tool of the corresponding set of tools.
9 . The system of claim 1 , wherein at least one of the asset lookup agent, the reference lookup agent and the advice agent applies a multimodal large language model.
10 . The system of claim 1 , wherein the asset comprises military machinery.
11 . A computer-implemented method for providing a conversational assistant using multimodal large language models and image segmentation, the method comprising the steps of:
receiving, via an interactive user interface, a user query and a scan of an image associated with an asset;
detecting, via a computer vision detection model, a text from the scan wherein the text is imprinted on the asset and the asset is a customized asset without a conventional serial number;
performing, via a prediction model, text recognition of the text and identifying one or more predicted texts with corresponding confidence levels;
displaying, via the interactive user interface executing on the mobile device, the one or more predicted texts comprising an asset identifier;
identifying, via an asset lookup agent applying a segmentation model, a plurality of objects associated with the asset identifier wherein each object of the plurality of objects is fed to a lookup model to identify each object against an internal asset database;
for a selected object, identifying, via the interactive user interface, a corresponding service action;
retrieving, via a reference lookup agent, a set of reference data associated with the selected object;
based on the identified service action and the selected object, generating, via an advice agent, a set of instructions and corresponding set of tools; and
providing, via the interactive user interface, a response to the user query wherein the response comprises the set of instructions and the corresponding set of tools.
12 . The method of claim 11 , wherein the asset lookup agent relies on pre-processing of the user query for consistency in one or more agent prompts.
13 . The method of claim 11 , wherein the corresponding service action comprises: repair, replace and maintenance.
14 . The method of claim 11 , wherein the set of reference data comprises one or more documents from a document database that communicates with a client inventory management system via an integration API.
15 . The method of claim 11 , wherein the set of reference data comprises instructional information that relate directly and indirectly to the asset.
16 . The method of claim 11 , wherein the response comprises an estimated time required for the identified service action.
17 . The method of claim 11 , wherein the set of instructions comprises multimedia content comprising a combination of: audio, video, images and text.
18 . The method of claim 11 , wherein the response comprises an inventory status for at least one tool of the corresponding set of tools.
19 . The method of claim 11 , wherein at least one of the asset lookup agent, the reference lookup agent and the advice agent applies a multimodal large language model.
20 . The method of claim 11 , wherein the asset comprises military machinery.