Generating a hybrid search index for unified search
The present disclosure is directed toward systems, methods, and non-transitory computer readable media for generating and searching a hybrid search index. In some embodiments, the disclosed systems generate a hybrid search index that comprises one or more content items stored at a content management system or at external network locations linked to the content management system via software connectors along with world state data associated with the one or more content items. The disclosed systems can generate a search result from the hybrid search index in response to receiving a search query of the hybrid search index. In some cases, the disclosed systems can rank one or more content items included in the search result based on observation layer data of the one or more content items.
1. A computer-implemented method comprising:
identifying a first plurality of content items stored at a content management system and a second plurality of content items stored at external network locations associated with external computer applications, wherein the second plurality of content items is associated with the content management system via software connectors;
determining, for a client device accessing the first plurality of content items and the second plurality of content items, world state data defining:
device metrics exhibited by the client device while accessing the first plurality of content items and the second plurality of content items; and
environmental metrics exhibited by the client device while accessing the first plurality of content items and the second plurality of content items;
generating searchable data from the world state data for the first plurality of content items and the second plurality of content items; and
generating a hybrid search index comprising the searchable data, the first plurality of content items, and the second plurality of content items.
2. The computer-implemented method of claim 1 , wherein generating the searchable data further comprises:
generating, utilizing a large language model, a world state description based on the device metrics and the environmental metrics corresponding to the first plurality of content items and the second plurality of content items.
3. The computer-implemented method of claim 1 , wherein:
the device metrics comprise at least one of internal device temperature, device movement, device orientation, or device lighting settings of a client device; and
the environmental metrics comprise at least one of device location, environmental lighting conditions, ambient noise, or camera data indicating a position of a user relative to the client device.
4. The computer-implemented method of claim 1 , further comprising:
identifying a user account associated with a client device and the first plurality of content items and the second plurality of content items; and
generating the hybrid search index specific to the user account associated with the client device and the first plurality of content items and the second plurality of content items.
5. The computer-implemented method of claim 1 , further comprising:
detecting one or more changes to the world state data based on the one or more changes to the device metrics or the environmental metrics; and
updating the searchable data by utilizing a large language model to update a world state description based on the one or more changes to the world state data.
6. The computer-implemented method of claim 1 , further comprising:
generating additional searchable data from observation layer data for the first plurality of content items and the second plurality of content items; and
including the additional searchable data from the observation layer data in the hybrid search index.
7. The computer-implemented method of claim 1 , further comprising:
identifying a third plurality of content items stored locally at a client device; and
updating the searchable data by including the world state data defining the device metrics and the environmental metrics corresponding to the third plurality of content items.
8. A system comprising:
at least one processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the at least one processor, cause the system to:
identify a first plurality of content items stored at a content management system and a second plurality of content items stored at external network locations associated with external computer applications, wherein the second plurality of content items is associated with the content management system via software connectors;
determine, for the first plurality of content items and the second plurality of content items, observational layer data defining display of one or more content items corresponding to the first plurality of content items and the second plurality of content items provided for display within a graphical user interface of a client device by determining pixel coordinates of the one or more content items within the graphical user interface of the client device;
generate searchable data from the observational layer data for the first plurality of content items and the second plurality of content items; and
generate a hybrid search index comprising the searchable data, the first plurality of content items, and the second plurality of content items.
9. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to generate the searchable data by:
generating, utilizing a large language model, an observational layer description based on the display within the graphical user interface of the client device, of the one or more content items corresponding to the first plurality of content items and the second plurality of content items.
10. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate one or more observational metadata tags identifying one or more portions of the one or more content items corresponding to the first plurality of content items and the second plurality of content items provided for display in the graphical user interface of the client device; and
include the one or more observational metadata tags in the hybrid search index.
11. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to determine the observational layer data by:
monitoring the graphical user interface of the client device; and
identifying one or more portions of the one or more content items corresponding to the first plurality of content items and the second plurality of content items provided for display in the graphical user interface of the client device.
12. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
determine an interest of an entity based on the observational layer data; and
update the searchable data to include the interest of the entity based on the observational layer data.
13. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate one or more observational metadata tags corresponding to one or more content items based on detecting display of the one or more content items corresponding to the first plurality of content items and the second plurality of content items provided for display within a graphical user interface of a client device; and
add the one or more observational metadata tags to the hybrid search index.
14. The system of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the system to:
determine user interaction data for the first plurality of content items and the second plurality of content items;
generate additional searchable data from the user interaction data for the first plurality of content items and the second plurality of content items; and
include the additional searchable data from the user interaction data in the hybrid search index.
15. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to:
identify a first plurality of content items stored at a content management system and a second plurality of content items stored at external network locations associated with external computer applications, wherein the second plurality of content items is associated with the content management system via software connectors;
determine, for the first plurality of content items and the second plurality of content items, user interaction data defining detected interactions with one or more content items corresponding to the first plurality of content items and the second plurality of content items provided for display within a graphical user interface of a client device associated with a user account;
generate searchable data from the user interaction data for the first plurality of content items and the second plurality of content items by generating, utilizing a large language model, a user interaction description outlining the detected interactions of the user account with the one or more content items corresponding to the first plurality of content items and the second plurality of content items for a given time period; and
generate a hybrid search index comprising the searchable data, the first plurality of content items, and the second plurality of content items.
16. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, via the client device, one or more user interactions with at least one of one or more selectable elements, one or more text input fields, or one or more cursor locations corresponding to the first plurality of content items and the second plurality of content items provided for display within the graphical user interface of the client device;
generate one or more user interaction metadata tags corresponding to the first plurality of content items and the second plurality of content items based on the one or more user interactions; and
extract content from the first plurality of content items and the second plurality of content items based on the one or more user interaction metadata tags.
17. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to generate the searchable data from the user interaction data by:
generating, via the large language model, the user interaction description of one or more detected interactions with the first plurality of content items and the second plurality of content items while the client device associated with the user account accesses the first plurality of content items and the second plurality of content items.
18. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
determine an interest of a user account associated with the client device and the first plurality of content items and the second plurality of content items based on the user interaction data; and
update the searchable data from the user interaction data for the user account by including the interest of user account in the searchable data.
19. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
detect one or more changes to the user interaction data based on receiving one or more additional user interactions with the first plurality of content items and the second plurality of content items; and
update the searchable data comprising the user interaction data based on the one or more additional user interactions.
20. The non-transitory computer readable medium of claim 15 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
determine world state data for the first plurality of content items and the second plurality of content items;
generate additional searchable data from the world state data for the first plurality of content items and the second plurality of content items; and
include the additional searchable data from the user interaction data in the hybrid search index.