Web browser with integrated vector database
A web browsing application can implement a vector database to store session data. A web browsing application can automatically embed loaded web content into an embedded data store that maintains session data for a number of web sessions. A user can query the web browser using simple instructions. The web browser can interpret the instructions and use the embedded data store to quickly search across multiple modalities of embedded data to retrieve relevant results. The web browser can use machine-learned models to answer queries or perform other tasks by performing vector-based queries over the content of visited web data.
1 . A computer-implemented method, the method comprising:
receiving input data describing a user interaction associated with a browser application operating on a user computing device;
retrieving a session data object associated with the input data based on searching a vector database representing session data associated with the browser application, session data descriptive of a web browsing history;
constructing, using the session data, conditioned input data for input to a machine-learned sequence processing model to perform a task associated with the input data, wherein the conditioned input data comprises the session data object and the input data, wherein the machine-learned sequence processing model attends over a plurality of input elements representing the conditioned input data to generate one or more output elements of an output sequence; and
generating, based on the output sequence, output data for rendering a response comprising the output data within the browser application or modifying, based on the output data comprising a command for causing the browser application to perform an operation, a state of the browser application.
2 . The method of claim 1 , wherein the input data comprises a query over content from one or more open tabs of the browser application.
3 . The method of claim 1 , wherein the input data is received via a context menu activated within the browser application.
4 . The method of claim 1 , wherein the machine-learned sequence processing model is configured to generate a response sequence that indicates one or more commands to adjust one or more tabs in the tabbed browsing interface based on relevance to the input data.
5 . The method of claim 1 , wherein the input data comprises a natural language query input via a unified input interface within the browser application, wherein the unified input interface is configured to determine a relevance of the natural language query to one or more functions of the browser application and render an interactive element associated with at least one relevant function.
6 . The method of claim 1 , wherein the conditioned input data comprises an interaction trajectory that characterizes one or more prior user interactions with the browser application.
7 . The method of claim 6 , wherein:
the interaction trajectory is generated by a machine-learned interaction trajectory generation system;
the interaction trajectory comprises data characterizing recorded user interactions; and
the machine-learned interaction trajectory generation system is configured to generate updated interaction trajectories responsive to state changes in the session data.
8 . The method of claim 1 , wherein constructing the conditioned input data comprises:
retrieving portions of the session data that are relevant to the input data using a similarity search over embeddings of the session data in the vector database.
9 . The method of claim 8 , wherein retrieving the session data object comprises:
classifying the input data to determine a corresponding retrieval precision; and
querying, using a query embedding, a subset of a data store of embedded session data, the subset characterized by the corresponding retrieval precision.
10 . The method of claim 1 , wherein the vector database comprises, for a respective item of session data:
a first embedding describing a portion of the respective item, the first embedding characterized by a first precision; and
a second embedding describing the portion of the respective item, the second embedding characterized by a second precision lower than the first precision.
11 . The method of claim 9 , wherein the session data comprises a data sketch, and wherein retrieving the session data object comprises:
retrieving, based on the corresponding retrieval precision, the data sketch.
12 . The method of claim 1 , wherein the output data comprises inputs for an application programming interface (API) of an operational environment to control an operation of the operational environment.
13 . A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations comprising:
receiving input data describing a user interaction associated with a browser application operating on a user computing device;
retrieving a session data object associated with the input data based on searching a vector database representing session data associated with the browser application, session data descriptive of a web browsing history;
constructing, using the session data, conditioned input data for input to a machine-learned sequence processing model to perform a task associated with the input data, wherein the conditioned input data comprises the session data object and the input data, wherein the machine-learned sequence processing model attends over a plurality of input elements representing the conditioned input data to generate one or more output elements of an output sequence; and
generating, based on the output sequence, output data for rendering a response comprising the output data within the browser application or modifying, based on the output data comprising a command for causing the browser application to perform an operation, a state of the browser application.
14 . The computing system of claim 13 , wherein:
the conditioned input data comprises an interaction trajectory that characterizes one or more prior user interactions with the browser application.
15 . The computing system of claim 14 , wherein:
the interaction trajectory is generated by a machine-learned interaction trajectory generation system;
the interaction trajectory comprises data characterizing recorded user interactions; and
the machine-learned interaction trajectory generation system is configured to generate updated interaction trajectories responsive to state changes in the session data.
16 . The computing system of claim 13 , wherein retrieving the session data object comprises:
classifying the input data to determine a corresponding retrieval precision; and
querying, using a query embedding, a subset of a data store of embedded session data, the subset characterized by the corresponding retrieval precision.
17 . The computing system of claim 13 , wherein the vector database comprises, for a respective item of session data:
a first embedding describing a portion of the respective item, the first embedding characterized by a first precision; and
a second embedding describing the portion of the respective item, the second embedding characterized by a second precision lower than the first precision.
18 . One or more non-transitory computer-readable media that store instructions that are executable by one or more processors to cause a computing system to perform operations comprising:
receiving input data describing a user interaction associated with a browser application operating on a user computing device;
retrieving a session data object associated with the input data based on searching a vector database representing session data associated with the browser application, session data descriptive of a web browsing history;
constructing, using the session data, conditioned input data for input to a machine-learned sequence processing model to perform a task associated with the input data, wherein the conditioned input data comprises the session data object and the input data, wherein the machine-learned sequence processing model attends over a plurality of input elements representing the conditioned input data to generate one or more output elements of an output sequence; and
generating, based on the output sequence, output data for rendering a response comprising the output data within the browser application or modifying, based on the output data comprising a command for causing the browser application to perform an operation, a state of the browser application.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein:
the conditioned input data comprises an interaction trajectory that characterizes one or more prior user interactions with the browser application.
20 . The one or more non-transitory computer-readable media of claim 19 , wherein:
the interaction trajectory is generated by a machine-learned interaction trajectory generation system;
the interaction trajectory comprises data characterizing recorded user interactions; and
the machine-learned interaction trajectory generation system is configured to generate updated interaction trajectories responsive to state changes in the session data.