IP Library Granted Patent US 11,537,871
Granted Patent B2
US 11,537,871 · App. 15/963,011 · Granted Dec 27, 2022

Deep neural network training for application program generation

Inventor: James Montantes (San Jose, CA)
Assignee: FUJITSU LIMITED
G06N3/08G06F40/40G06V30/194H04L63/08G06F3/013G06F3/041G06F40/169
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,537,871
App. No.
15/963,011
Granted
Dec 27, 2022
Kind
B2
Abstract

A computer architecture may comprise a processor, a memory, and a differential memory subsystem (DMS). A learning engine is stored on the memory and configured to present data to an expert user, to receive user sensory input measuring reactions related to the presented data, and to create an attention map based thereon. The attention map is indicative of portions of the presented data on which the expert user focused. The learning engine is configured to annotate the attention map with the natural language input labels and to train a neural network based on the user sensory input. The learning engine is configured to create a model based on the trained neural network, to provide an application program for an output target; and to instruct the output target via the application program to detect and remedy anomalous activity. The DMS is physically separate and configured for experimental data processing functions.

Claims (79)

1. A computer system architecture configured for deep neural network training, the computer system architecture comprising:

a processor;

a memory that includes random access memory (RAM) on which a learning engine is stored, wherein the learning engine is configured to:

present data about a particular subject to a user having knowledge about the particular subject,

as the data is presented to the user, receive user sensory input via one or more sensors measuring feedback and reactions of the user related to the presented data;

create an attention map based on the user sensory input, the attention map being indicative of portions of the presented data on which the user focused and including one or more data classification regions;

annotate the attention map with natural language input labels based on a natural language input of the user;

train a neural network based on the annotated attention map;

create a model based on the trained neural network that is optimized for an anomalous activity of an end use application;

provide an application program for an output target of the end use application; and

instruct the output target via the application program to detect and remedy the anomalous activity; and

a differential memory subsystem that is physically separate from and networked to the memory and the processor, the differential memory subsystem being configured to receive neural network information and processed user sensory input from the learning engine for experimental data processing functions.

2. The computer system architecture of claim 1 , wherein the differential memory subsystem employs operations to mimic a neural Turing machine for experimental data processing functions.

3. The computer system architecture of claim 1 , wherein the learning engine is further configured to:

receive from the user a session token identifier;

authenticate the user based on the session token; and

store information obtained from the user on a blockchain-style ledger.

4. The computer system architecture of claim 1 , wherein the learning engine is further configured to enable a quality check of the application program.

5. The computer system architecture of claim 4 , wherein the quality check includes:

receiving from a second user a second session token identifier;

authenticating the second user based on the second session token;

timestamping a quality check session of the second user; and

storing information obtained from the second user on a blockchain-style ledger.

6. The computer system architecture of claim 1 , wherein:

the user includes an agriculture user with knowledge about agriculture;

the attention map includes a heat map, an image with a grid overlay, or an object segmentation selection;

the user sensory input includes user gaze data representative of a gaze of the agriculture user as captured by a gaze sensor and physical touch data representative of a physical touch received on a touch screen; and

the natural language input of the agriculture user includes a voice input or a touch function.

7. The computer system architecture of claim 1 , wherein:

the user includes a security user with knowledge about security;

the attention map includes historical footage;

the user sensory input includes user gaze data representative of a gaze of the security user as captured by a gaze sensor and an electroencephalogram and a camera representative of an emotional response; and

the natural language input of the security user includes a voice input.

8. The computer system architecture of claim 1 , wherein the one or more sensors includes one or more of a combination of an RBGd camera, a gesture capture sensor, a biometric capture system, wearable sensor system, digital I/O interface, wearable ring type input, a keyboard, a mouse, an emotion recognition system, a gaze tracking system, a video interface, a microphone, a brain computer interface, a haptic interface, a wireless receiver, a high resolution camera, a thermal or infrared camera, a light sensor, a GPS, a wireless transmitter, a three-hundred and sixty degree camera, a digital optical interface, and a virtual reality interface.

9. The computer system architecture of claim 1 , wherein the output target includes a drone, a camera, or an augmented reality headset.

10. The computer system architecture of claim 1 , wherein:

the learning engine is further configured to receive user input from the user; and

the user input includes a selection of a data source from which the presented data originates and includes a selection of the output target.

11. A method of deep neural network training for application program generation, the method comprising:

presenting, by a trainer server, data about a particular subject to a user having knowledge about the particular subject;

receiving, by the trainer server, user sensory input via one or more sensors measuring activities and reactions of the user related to the presented data as the data is presented to the user;

creating, by the trainer server, an attention map based on the user sensory input, the attention map being indicative of portions of the presented data on which the user focused and including one or more data classification regions;

annotating, by the trainer server, the attention map with natural language input labels based on a natural language input of the user;

training, by the trainer server, a neural network based on the annotated attention map, the training including communicating to a differential memory subsystem neural network information and processed user sensory input for experimental data processing functions;

creating, by the trainer server, a model based on the trained neural network that is optimized for an anomalous activity of an end use application;

providing, by the trainer server, an application program for an output target of the end use application; and

instructing, by the trainer server, the output target via the application program to detect and remedy an anomalous activity.

12. The method of claim 11 , further comprising receiving user input from the user, wherein the user input includes a selection of a data source from which the presented data originates and includes a selection of the output target.

13. The method of claim 11 , wherein the differential memory subsystem is physically separate from and networked to a primary memory of the trainer server.

14. The method of claim 11 , further comprising:

receiving from the user a session token identifier;

authenticating the user based on the session token; and

storing information obtained from the user on a blockchain-style ledger.

15. The method of claim 11 , further comprising enabling a quality check of the application program, wherein the quality check includes:

receiving from a second user a second session token identifier;

authenticating the second user based on the second session token;

timestamping a quality check session of the second user; and

storing information obtained from the second user on a blockchain-style ledger.

16. The method of claim 11 , wherein:

the user includes a user with knowledge about agriculture;

the attention map includes a heat map, an image with a grid overlay, or an object segmentation selection;

the user sensory input includes user gaze data representative of a gaze of the agriculture as captured by a gaze sensor and physical touch data representative of a physical touch received on a touch screen; and

the natural language input of the agriculture user includes a voice input or a touch function.

17. The method of claim 11 , wherein:

the user includes a security user with knowledge about security;

the attention map includes historical footage;

the user sensory input includes user gaze data representative of a gaze of the security user as captured by a gaze sensor and an electroencephalogram and a camera representative of an emotional response; and

the natural language input of the security user includes a voice input.

18. The method of claim 11 , wherein the output target is a drone, a camera, or an augmented reality headset.

19. The method of claim 11 , wherein the one or more sensors includes one or more or a combination of a RBGd camera, a gesture capture sensor, a biometric capture system, wearable sensor system, digital I/O interface, wearable ring type input, a keyboard, a mouse, an emotion recognition system, a gaze tracking system, a video interface, a microphone, a brain computer interface, a haptic interface, a wireless receiver, a high resolution camera, a thermal or infrared camera, a light sensor, a GPS, a wireless transmitter, a three-hundred and sixty degree camera, a digital optical interface, and a virtual reality interface.

20. A non-transitory computer readable medium having stored therein executable code that, in response to being executed by a processor, causes a system to perform operations, the operations comprising:

presenting, by a trainer server, data about a particular subject to a user having knowledge about the particular subject;

receiving, by the trainer server, user sensory input via one or more sensors measuring activities and reactions of the user related to the presented data as the data is presented to the user;

creating, by the trainer server, an attention map based on the user sensory input, the attention map being indicative of portions of the presented data on which the user focused and including one or more data classification regions;

annotating, by the trainer server, the attention map with natural language input labels based on a natural language input of the user;

training, by the trainer server, a neural network based on the annotated attention map, the training including communicating to a differential memory subsystem neural network information and processed user sensory input for experimental data processing functions;

creating, by the trainer server, a model based on the trained neural network that is optimized for an anomalous activity of an end use application;

providing, by the trainer server, an application program for an output target of the end use application; and

instructing, by the trainer server, the output target via the application program to detect and remedy an anomalous activity.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 4, 2018
From: MONTANTES, JAMES
To: FUJITSU LIMITED
Reel/Frame 047225/0386 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 3, 2018
From: MONTANTES, JAMES
To: FUJITSU LIMITED
Reel/Frame 046084/0167 →
Continuity (1)
Related Publication 20190332931A1 · Oct 31, 2019
Cited By (3)
US 12,481,875 US 12,481,876 US 12,620,214