SYSTEMS AND METHODS FOR ARTIFICIAL-INTELLIGENCE ASSISTANCE IN VIDEO COMMUNICATIONS
A communication assist service may extract one or more video frames during a communication session between a user device and a terminal device. The video frames may include a representation of an object associated with an issue for which the communication session was established. The communication assist service may generate a feature vector from the video frames and execute a trained neural network configured to generate predictions associated with a resolution to the issue. The neural network may output predicted actions that if executed may resolve the issue or provide additional information that will improve a likelihood of resolving the issue. The communication assist service may then transmit the predicted actions to the terminal device in real time.
1 . A computer-implemented method comprising:
extracting one or more video frames from video streams of a set of communication sessions, wherein the video frames include a representation of an object, and wherein the object is associated with an issue for which the communication session is established;
defining a training dataset from the one or more video frames and features extracted from the set of communication sessions;
training a neural network using the training dataset, the neural network being configured to generate predictions of actions associated with the object;
extracting a video frame from a new video stream of a new communication session, wherein the video frame includes a representation of a particular object, and wherein the object is associated with a particular issue;
executing the neural network using the video frame from the new video stream, wherein the neural network generates a predicted action associated with the particular object; and
facilitating a transmission of a communication to a device of the new communication session, the communication including a representation of the predicted action.
2 . The computer-implemented method of claim 1 , wherein the new communication session is between a user device and a terminal device.
3 . The computer-implemented method of claim 1 , wherein the particular issue is associated with a hardware or software fault in a device operated by a user.
4 . The computer-implemented method of claim 1 , wherein the neural network is an ensemble network comprising two or more neural networks configured to generate outputs of different types.
5 . The computer-implemented method of claim 1 , wherein the neural network is configured to generate a boundary box over the object.
6 . The computer-implemented method of claim 1 , wherein the neural network is configured to generate a predicted identification of the object.
7 . The computer-implemented method of claim 1 , wherein the predicted action associated with the particular object comprises a maintenance action or a repair action configured to restore operability in the particular action.
8 . A system comprising:
one or more processors; and
a non-transitory machine-readable storage medium storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations including:
extracting one or more video frames from video streams of a set of communication sessions, wherein the video frames include a representation of an object, and wherein the object is associated with an issue for which the communication session is established;
defining a training dataset from the one or more video frames and features extracted from the set of communication sessions;
training a neural network using the training dataset, the neural network being configured to generate predictions of actions associated with the object;
extracting a video frame from a new video stream of a new communication session, wherein the video frame includes a representation of a particular object, and wherein the object is associated with a particular issue;
executing the neural network using the video frame from the new video stream, wherein the neural network generates a predicted action associated with the particular object; and
facilitating a transmission of a communication to a device of the new communication session, the communication including a representation of the predicted action.
9 . The system of claim 8 , wherein the new communication session is between a user device and a terminal device.
10 . The system of claim 8 , wherein the particular issue is associated with a hardware or software fault in a device operated by a user.
11 . The system of claim 8 , wherein the neural network is an ensemble network comprising two or more neural networks configured to generate outputs of different types.
12 . The system of claim 8 , wherein the neural network is configured to generate a boundary box over the object.
13 . The system of claim 8 , wherein the neural network is configured to generate a predicted identification of the object.
14 . The system of claim 8 , wherein the predicted action associated with the particular object comprises a maintenance action or a repair action configured to restore operability in the particular action.
15 . A non-transitory machine-readable storage medium storing instructions that when executed by one or more processors, cause the one or more processors to perform operations including:
extracting one or more video frames from video streams of a set of communication sessions, wherein the video frames include a representation of an object, and wherein the object is associated with an issue for which the communication session is established;
defining a training dataset from the one or more video frames and features extracted from the set of communication sessions;
training a neural network using the training dataset, the neural network being configured to generate predictions of actions associated with the object;
extracting a video frame from a new video stream of a new communication session, wherein the video frame includes a representation of a particular object, and wherein the object is associated with a particular issue;
executing the neural network using the video frame from the new video stream, wherein the neural network generates a predicted action associated with the particular object; and
facilitating a transmission of a communication to a device of the new communication session, the communication including a representation of the predicted action.
16 . The non-transitory machine-readable storage medium of claim 15 , wherein the new communication session is between a user device and a terminal device.
17 . The non-transitory machine-readable storage medium of claim 15 , wherein the particular issue is associated with a hardware or software fault in a device operated by a user.
18 . The non-transitory machine-readable storage medium of claim 15 , wherein the neural network is an ensemble network comprising two or more neural networks configured to generate outputs of different types.
19 . The non-transitory machine-readable storage medium of claim 15 , wherein the neural network is configured to generate a boundary box over the object.
20 . The non-transitory machine-readable storage medium of claim 15 , wherein the neural network is configured to generate a predicted identification of the object.