System and method for capturing, preserving, and representing human experiences and personality through a digital interface

1 . A method for capturing, preserving, and representing human experiences and personality through a digital interface, comprising:

digital activity recording (DAR) software installed on a personal computing device to capture and transmit digital device usage patterns;

a multimodal sensor device array (MSDA) which captures human activities and interactions within an environment;

an information storage device (ISD) which captures and cryptographically signs sensor information and cooperates with an interface to distribute the sensor information to a least one computer on a network that shares data and resources via wired or wireless technologies;

a methods control interface (MCI) that configures a plurality of modules;

a pattern recognition engine (PRE) which analyzes data and categorizes information with metadata based on detected interactions and events; and

a synthetic data generator (SDG) which utilizes a classification and categorization metadata created by the PRE to produce new data which is synthetic in nature but based on representative features present in end user data,

wherein the SDG is configured to produce synthetic data and data generation models, which are leveraged by the DAR, the MSDA, the ISD, the MCI, and the PRE,

and wherein the SDG uses a Generative Adversarial Network (GAN) process to develop data creation and evaluation models; the GANs utilize two distinct neural networks which leverage each other iteratively for mutual refinement;

the first neural network is known as a Discriminator, and

the second neural network is known as a Generator;

each neural network iteratively leverage each other for mutual training and refinement;

the role of the Discriminator is to predict whether data sample derived form a sensor or data within the ISD is real or synthetic;

alternating datasets are fed into the Discriminator;

the first dataset is a collection of labeled data which is the real data;

the second dataset is data from the Generator, which at first randomly generated data which matches the same or approximate characteristics of real data;

wherein using the first and second dataset, the Discriminator is trained to recognize whether the data is real or generated;

the discriminator outputs a 1 when the discriminator identifies real data, a 0 when the discriminator identifies generated data, and a floating-point number to reflect the Discriminator classification confidence; and

a low confidence of classification is 0.1 whereas a high confidence is between 0.9 to 1,

wherein the plurality of modules comprises the DAR, the MSDA, the ISD, the PRE, the SDG, the GAN, the Discriminator, and the Generator.

2 . The method of claim 1 , wherein

the role of the Generator is to produce data with the characteristics which the Discriminator will classify as real data and score as high of a confidence as possible;

random data samples, sometimes known as seeds, are inputted into the Generator, which uses a neural network to produce an output sample;

the output sample is fed into the input of Discriminator, as identified; and

the results of the Discriminator, between 0 and 1, are fed back into the Generator through a process known as backpropagation to update the Generator neural network weights, thus changing a future output.

3 . The method of claim 2 , wherein

the Discriminator and the Generator are trained iteratively;

with each training iteration, the Discriminator gets better at differentiating between real data and generated data;

conversely, with each iteration the Generator gets better are at producing outputs which are better at fooling the Discriminator; and

through an extensive set of iterations, the Generator is configured to produce synthetic data which is highly realistic, exceeding a threshold of confidence by the Discriminator.

4 . The method of claim 2 , wherein

the GAN is configured to reproduce synthetic data which simulates the sensor, end user generated, or any dataset contained within the ISD;

synthetic data achieves an acceptable threshold of realism to accurately simulate the data features of the original data to meet both Discriminator analysis and the end user subjective quality;

synthetic data generation is generated in accordance with parameters which manipulate the nature and the content of the GAN output; and

synthetic data specifically enables the representation of the visual, verbal, auditory appearance of the end user and the end user environment.

5 . The method of claim 4 , wherein

generating models for the personality simulation engine (PSE) and the synthetic likeness and voice engines (SLE and SVE);

such generative models generated within the SDG are exportable for use within the PSE, SLE, and SVE; and

these exportable models are used by the PSE, the SLE and SVE engines to produce new content to simulate biographical or interaction events.

6 . The method of claim 1 , further comprising

a personality simulation engine (PSE) which utilizes the classified metadata generated within the PRE to synthesize data which represents the reactions and modes of response to stimulus which is representationally like those of the end user;

a simulated personality produces sequences of data which represent actions, speech methods, motions, and events similar to the actions, speech methods, motions, and events of the end user;

the PSE creates these recognizable outputs based on input stimulus from a variety of different sources but is primarily driven by the human interaction interface (HII);

the PSE generates interaction information using multiple methods including: Text-based speech interaction, data and associated metadata extracts which align to the topics classified within the interaction, and results from the classification of data provided by a future user.

7 . The method of claim 6 , wherein

the PSE identifies the era(s) in a life of the end user to simulate based on life periods identified by the input and specifications of the future user;

wherein the three general settings for life scope specification are: time period-based specification, open specification, and aggregate life specification;

the PSE collects input stimulus and has several primary modes of interaction including conversational interactions between a future user and the PSE via the HII, replay or of data from specific periods and events; recreations of events which occurred, miming or re-enacting events; providing external data; and direct system to system queries;

the PSE permits system to system queries, providing an intermediate layer for another system to query data rapidly and request metadata types, periods of time, and data with greater specificity;

the PSE is configured to return fully synthetic data, a hybrid mix of real and synthetic data, or samples of original data captured by the end user and stored within the ISD; and

the PSE returns a metadata response which represents how the end user would likely interpret and respond to a query.

8 . The method of claim 7 , wherein

the PSE responds to queries derived from interactions with the future user within a time-constrained context of an engagement lasting a few seconds to several hours or longer;

within this constrained interaction window, the PSE generates a set of responses to external stimulus which are configured to be as highly representative of the reactions of the end user within a user-selected window of the end user biographical history; and

the PSE accepts these interactions via either the MCI interface, controlled by the end user or the end user delegate, or through the HII.

9 . The method of claim 8 , wherein

the PSE evaluates the metadata patterns, classifications, and narratives generated by the PRE and creates models capable of generating predictive behavior data;

using the classified biographical actions of the end user, the PSE evaluates the data and metadata which describes the environments and events which correlate as triggers to the end user response; wherein the triggers are the time of day, the nature of interaction, and stimulus;

the PSE analyzes multiple data streams to find which events may have triggered reactions from the end user; and

the PSE categorizes these events into triggering sets based on a broad spectrum of metadata, including textual analysis, human to human interaction patterns, environmental factors, cyclical patterns, to determine how the end user reacted to interaction, confrontation, positive and negative stimulus, environment, and sets of metadata derived by the PRE.

10 . The method of claim 9 , wherein

based on a subset recorded data that follows the triggers, the PSE uses predictive modelling to produce an array of possible responses that the end user would most likely demonstrate given all correlated triggers;

the PSE samples biographical data on what the end user said and did in response to a similar set of triggers as well as generates a list of alternate responses based on observed responses to other scenarios;

while the triggers of the original event and the HII interaction are unlikely to be identical, the PSE implements a cascading evaluation of all known triggers and gradually reduces the specificity of the triggers until a user-defined threshold is reached of possible predictions of the end user reactions; and

an action and response is then generated.

11 . The method of claim 6 , wherein

the PSE record of the HII interaction session adds additional inputs to help guide the consistency of the interaction;

as a future user interacts with the HII and the HII interactions are passed to the PSE, the responses outputs of the PSE are channeled back into the input for subsequent engagements to provide a feedback cycle;

the PSE utilizes these response records in formulating subsequent responses as to not backtrack onto previously discussed topics, as well as to build on the sentiment of the conversation; and

the PSE uses the session memory to avoid entering cyclical looping engagements which could result in unrealistic results including repeated conversation topics or continual looping engagement patterns.

12 . The method of claim 6 , wherein

the natural language outputs from the PRE are configured to identify the language and vocabulary as the language is spoken by the end user, not as grammatical rules dictate.

13 . The method of claim 6 , wherein

the PSE utilizes the MCI to engage the end user on a series of sessions to capture this information through a recorded interview process;

during the process, the end user body language is captured by the MCI and recorded as a training set for modelling and reference;

body posture and gross body movements are captured via motion capture;

finer movements comprising at least finger movements, facial expressions and minor motions of the head and neck, are all captured and analyzed via video;

wherein the body posture, the gross body movements and the finer movements are classified for intensity, motion and posture using external training data; and

to provide a representative set of samples, the recorded interview process must be repeated several times over the life of the end user to gather data on how they respond throughout the life of the end user.

14 . The method of claim 6 , wherein

the PRE and the DAR provide an opportunity to capture the end user topics of interest;

interest mapping is aligned to the period of life of the end user and the PRE and the DAR is configured to chart the interest mapping as progressions in interest and bias over time;

interests are derived based on the consistency and frequency of recognizable features in the end user datasets;

features with high degrees of reoccurrence over an extended time period or periodic reoccurrences after sporadic periods of low occurrence represent heightened interest and bias toward specific activities and interactions;

the interactions between the future user and the PSE references this subjective interest to demonstrate topics of specific interest in the end user recorded history;

based on the patterns of interactions by the future user, data which is indexed with heightened potential interest is configured to be prioritized for return; and

the PSE is configured to either reference the content or display the content as part of the interaction response through the HII and pantomime or paraphrase content from such sources as the content relates to the era of life of the end user that is the subject of the interaction.

15 . The method of claim 6 , wherein

the PSE is configured to sample a sliding window of time from biographical experiences of the end user, associated metadata, and models to represent likely responses to interactions suitable representing the personality of the end user.

16 . The method of claim 6 , wherein

data selection is weighed based on the indexing of uncommon events as key points of life;

periods of heightened novelty, comprising the presence of novel features within datasets, or change, when predictive or classification models experienced a degradation in accuracy or confidence, are the two primary means of indexing such-unique events;

the PRE creates metadata indices which identify moments or periods of time where models failed to accurately predict or classify the patterns of the end user;

by over-indexing on these periods of change, the PSE generates interactions and shares knowledge in these periods as primary areas of interest and potential interaction with a user;

the PSE is configured to draw from these areas in a weighted manner in preparing responses to user interaction.

17 . The method of claim 1 , further comprising

a synthetic likeness engine (SLE) which produces a likeness of the end user as representations of animated two-dimensional images or three-dimensional forms;

for each era of the end user biographical history, this likeness changes to represent the appearance of the end user during that period of life of the end user;

the likeness is a virtual avatar which is intended to walk, speak, gesture, pose, move, and act as closely as possible to the end user with the data collected during the lifetime of the end user.

18 . The method of claim 17 , wherein

the virtual avatar uses a blended method to make the virtual avatar realistic through the combination of gross animation of the 3D form combined with fine animation of the textures which are mapped on the virtual avatar to give specific features;

gross movements are informed by sensor data, image, and video materials which capture how the end user moved in life;

fine features are generated from likeness images produced by the synthetic data generator (SDG); and

these images are then texture mapped using common UV mapping methods to align to the geometry of the body of the 3D model where the combination of 3D form with applied texture mapping produces a realistic likeness of the human.

19 . The method of claim 18 , wherein

additional image or biometric reference data is added via the MCI to complete any required image datasets for a realistic and comprehensive image of the body, skin, and physical characteristics of the end user.

20 . The method of claim 18 , wherein

the SLE uses templated 3D forms of children and adults as the basis for the virtual avatars; wherein templates are generic gendered templates which are pre-built to represent common body shapes, body mass, musculatures, fats, limb lengths, flexibility, hair growth patterns, skin tones and shades, manual dexterity, facial features, and proportions;

templates are then adapted using data features recognized by the PRE; and

all the physical features of the body are used to adjust the templates with specific parameters to match the body geometry of the end user at specific times in the end user biographical history.

21 . The method of claim 20 , wherein

the SLE is capable of importing assets which represent the real assets and accessories of life and applying them to the model;

and wherein a combination of wear is configured to be detected by the PRE or user defined;

and wherein the end user optionally specifies a different set of clothes to represent specific eras or activities within the end user biographical history.

22 . The method of claim 20 , wherein

the combination of motion with speech from the PRE provides the basis for the degree and nature of hand gestures and movements.

23 . The method of claim 20 , wherein

the SDG references all available image data to reproduce the facial mapping of the end user with realistic blemishes, wrinkles, and micro-expressions which will significantly enhance the realism of the virtual avatar;

facial image data from the SDG is imported as image data by the SLE;

the facial geometries are mapped using a facial recognition model and applied against the 3D geometry of the virtual avatar;

as the virtual avatar speaks and moves, the mapping updates based on these characteristics; and

the facial geometries extracted by the PRE and generated by the SDG are applied to update the 3D virtual avatar.

24 . The method of claim 20 , wherein

the geometries of the virtual avatar are adjusted through measurements taken from image and video data;

the SLE uses photogrammetry to perform this function on the human body.

25 . The method of claim 17 , further comprising

a Synthetic Voice Engine (SVG) that synchronizes with the Synthetic Likeness Engine (SLE) to produce an integrated audio overlay which aligns to the animation and movement of the virtual avatar character;

the SVE uses the vocal characteristics of the end user that are identified in the PRE;

the SVE utilizes the voice generation models developed by the SDG and narrates the text that is created from the personality simulation engine (PSE) to produce a realistic audio that represents the end user at various ages and periods of the end user biographical history.

26 . The method of claim 25 , wherein

if the virtual avatar is recreating a conversation or a monologue that the end user delivered within the history, the SVE approximates delivery of the historical event with as much accuracy as possible; and

if the virtual avatar is narrating an event, describing what happened instead of re-enacting the event, then a separate model is configured to be used for clarity.

27 . The method of claim 26 , wherein

the SDG reproduces trained voice print models, and the personality simulation engine (PSE) produces the spoken language content and metadata;

together, the SVN combines the trained voice print models and the spoken language content and metadata outputs to generate acoustic information;

the SVN adapts the acoustic data output of the trained voice print models and the spoken language content and metadata models to adjust pitch, cadence, intonation, and other acoustic characteristics to align to the contextual parameters of the interaction and the relative age of the virtual avatar of the end user by applying audio adjustment filters which modulate the acoustic properties of the voice; and

the modulation of the acoustic properties of the voice is performed based on the output of the PSE and aligned to the timing of the SLE animation of the form of the virtual avatar.

28 . The method of claim 27 , wherein

the PRE performs a sentiment analysis of the vocal characteristics of the voice data within the ISD;

this sentiment data is linked as metadata to the original data samples; and

sentiment adjustments to the synthetic voice are generated by the PSE based on the predictive conversation models developed around the interactions of the end user.

29 . The method of claim 28 , wherein

for re-enactments of the voice, the characteristics of the virtual environment that the virtual avatar is occupying have an impact on the vocal properties that the virtual avatar is sharing; and

environmental filters are configured to apply to the voice to simulate the context of the various rooms and environments that the end user occupied to a more realistic interaction experience for the future user.

30 . The method of claim 29 , wherein

the SVE also utilizes the SDG to produce environmental sounds to layer over the vocal simulation to add context; and

the nature of the augmentation will vary depending on whether the sounds are contextual and secondary or related to the primary topic of the interaction.