Obscuring objects in data streams using machine learning
Aspects of the present disclosure are directed to obscuring objects in data streams using machine learning. A data stream captured at a client device associated with a principal user (e.g., video stream, artificial reality session, and the like) can be processed to recognize objects in the stream. Based on user preferences that define object sharing rules, one or more of the recognized objects can be obscured from the data stream. For example, when the principal user's data stream is displayed to the participant users, such as during a video conference, objects in the principal user's data stream can be obscured according to the object sharing rules. Different groups of object sharing rules (e.g., profiles) can be used for different participant users or session types. Machine learning can be used to learn trends and predict a profile for use with a current or future shared streaming data session.
1. A method for obscuring objects in data streams using machine learning, the method comprising:
receiving user input that provides one or more explicit display rules, wherein,
a visual representation is displayed to a user comprising a field of view that includes the user and first objects recognized using one or more machine learning models,
the user input selects at least one of the first objects, and
a portion of the explicit display rules provided by the user input comprise a) a display status for the at least one selected first object, b) a category, recognized by the one or more machine learning models, for the at least one selected first object, and c) physical characteristics, recognized by the one or more machine learning models, for the at least one selected first object and/or a location, recognized by the one or more machine learning models, for the at least one selected first object;
recognizing second objects, including at least a category for each of the second objects and one or more of a location for each of the second objects or physical characteristics for each of the second objects, in a streaming data session using one or more machine learning models, the streaming data session comprising a video stream or a shared artificial reality session;
accessing rules, including the one or more explicit display rules, for displaying objects in shared streaming data sessions; and
causing one or more of the recognized second objects to be obscured in the streaming data session, wherein,
the causing is based on an application of the defined rules to the recognized category for the one or more second objects, and one or more of the recognized location of the one or more second objects or the recognized physical characteristics of the one or more second objects, and
at least one of the second objects is obscured based on application of the portion of the explicit display rules.
2. The method of claim 1 , wherein the at least one second object is obscured by causing a visual obscuring of the at least one second object from display when the streaming data session is displayed, the visual obscuring comprising visual blurring, visual blocking by a displayed obstruction, visual filter, or mask, or a combination thereof.
3. The method of claim 1 , wherein the at least one second object is obscured by obscuring first portions of the object and not obscuring second portions of the object.
4. The method of claim 1 , wherein the at least one second object comprises an alphanumeric character object and the portion of the explicit display rules define that alphanumeric character objects should not be displayed, or the at least one second object comprises a brand or logo object and the portion of the explicit display rules define that brand or logo objects should not be displayed.
5. The method of claim 1 , the portion of the explicit display rules define a predetermined list of objects that should be displayed, each object on the predetermined list is defined using a category for each object and at least one physical characteristic for each object, and the at least one second object is obscured when application of the portion of the explicit display rules to the at least one second object determines that the at least one second object is not part of the predetermined list of objects.
6. The method of claim 1 ,
wherein the rules are defined for a first user, the rules comprise first rules for displaying objects in streaming data sessions to a second user and second rules for displaying objects in streaming data sessions to a third user, and the second rules are different from the first rules, and
wherein the at least one second object is obscured by:
causing the streaming data session to be displayed to the second user and the third user, wherein the at least one second object is obscured from display to the second user and the at least one second object is not obscured from display to the third user.
7. The method of claim 6 , wherein the streaming data session is captured using one or more image capturing devices at a client system associated with the first user, and the streaming data session is caused to be displayed to the second user and the third user as part of a video teleconference, a video call, or an artificial reality session.
8. The method of claim 1 , wherein the streaming data session is captured using one or more image capturing devices at a client system, the streaming data session comprises a predefined region captured by the image capturing devices, and the rules define that objects located outside the predefined region should not be displayed.
9. The method of claim 1 , the streaming data session comprises a shared artificial reality session with a plurality of users.
10. The method of claim 1 , wherein,
the user input defines one or more explicit display rules for a list of objects that should be displayed, each object on the list is defined using a category, recognized by the one or more machine learning models, for each object and one or more of: a) a physical characteristic, recognized by the one or more machine learning models, for each object; or b) a location, recognized by the one or more machine learning models, for each object, and
the at least one second object is obscured when application of the one or more explicit rules to the at least one second object determines that the at least one second object is not part of the list of objects.
11. The method of claim 1 , further comprising:
suggesting, to a first user, a sharing profile comprising a set of rules for displaying objects in shared streaming data sessions, wherein:
the suggested sharing profile is generated by a predictive model configured using historic data, the historic data comprising instances of: a) users participating in the historic shared streaming data sessions, and b) historic explicit rules, based on user input, for displaying objects implemented during the historic shared streaming data sessions, and
the rules comprise the set of rules from the suggested sharing profile when the first user accepts the suggestion.
12. The method of claim 11 , wherein the predictive model comprises a machine learning model trained by the historic data, and the historic data comprises a) historic data for the first user and/or b) historic data for the first user and a plurality of additional users.
13. The method of claim 12 , wherein the suggested sharing profile generated by the predictive model is based on a set of user participants for the streaming data session.
14. A computing system for obscuring objects in data streams using machine learning, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
receiving user input that provides one or more explicit display rules, wherein,
a visual representation is displayed to a user comprising a field of view that includes the user and first objects recognized using one or more machine learning models,
the user input selects at least one of the first objects, and
a portion of the explicit display rules provided by the user input comprise a) a display status for the at least one selected first object, b) a category, recognized by the one or more machine learning models, for the at least one selected first object, and c) physical characteristics, recognized by the one or more machine learning models, for the at least one selected first object and/or a location, recognized by the one or more machine learning models, for the at least one selected first object;
recognizing second objects, including at least a category for each of the second objects and one or more of a location for each of the second objects or physical characteristics for each of the second objects, in a streaming data session using one or more machine learning models, the streaming data session comprising a video stream or a shared artificial reality session;
accessing rules, including the one or more explicit display rules, for displaying objects in shared streaming data sessions; and
causing one or more of the recognized second objects to be obscured in the streaming data session, wherein,
the causing is based on an application of the defined rules to the recognized category for the one or more second objects, and one or more of the recognized location of the one or more second objects or the recognized physical characteristics of the one or more second objects, and
at least one of the second objects is obscured based on application of the portion of the explicit display rules.
15. The computing system of claim 14 , wherein the at least one second object is obscured by causing a visual obscuring of the at least one object from display when the streaming data session is displayed, the visual obscuring comprising visual blurring, visual blocking by a displayed obstruction, visual filter, or mask, or a combination thereof.
16. The computing system of claim 14 , wherein the portion of the explicit display rules define a predetermined list of objects that should be displayed, each object on the predetermined list is defined using a category for each object and at least one physical characteristic for each object, and the at least one second object is obscured when application of the portion of the explicit display rules to the at least one second object determines that the at least one second object is not part of the predetermined list of objects.
17. The computing system of claim 14 ,
wherein the rules are defined for a first user, the rules comprise first rules for displaying objects in streaming data sessions to a second user and second rules for displaying objects in streaming data sessions to a third user, and the second rules are different from the first rules, and
wherein the at least one second object is obscured by:
causing the streaming data session to be displayed to the second user and the third user, wherein the at least one object is obscured from display to the second user and the at least one object is not obscured from display to the third user.
18. The computing system of claim 17 , wherein the streaming data session is captured using one or more image capturing devices at a client system associated with the first user, and the streaming data session is caused to be displayed to the second user and the third user as part of a video teleconference, a video call, or an artificial reality session.
19. The computing system of claim 14 , wherein the process further comprises:
suggesting, to a first user, a sharing profile comprising a set of rules for displaying objects in shared streaming data sessions, wherein:
the suggested sharing profile is generated by a predictive model configured using historic data, the historic data comprising instances of: a) users participating in the historic shared streaming data sessions, and b) historic explicit rules, based on user input, for displaying objects implemented during the historic shared streaming data sessions, and
the rules comprise the set of rules from the suggested sharing profile when the first user accepts the suggestion.
20. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for obscuring objects in data streams using machine learning, the process comprising:
receiving user input that provides one or more explicit display rules, wherein,
a visual representation is displayed to a user comprising a field of view that includes the user and first objects recognized using one or more machine learning models,
the user input selects at least one of the first objects, and
a portion of the explicit display rules provided by the user input comprise a) a display status for the at least one selected first object, b) a category, recognized by the one or more machine learning models, for the at least one selected first object, and c) physical characteristics, recognized by the one or more machine learning models, for the at least one selected first object and/or a location, recognized by the one or more machine learning models, for the at least one selected first object;
recognizing second objects, including at least a category for each of the second objects and one or more of a location for each of the second objects or physical characteristics for each of the second objects, in a streaming data session using one or more machine learning models, the streaming data session comprising a video stream or a shared artificial reality session;
accessing rules, including the one or more explicit display rules, for displaying objects in shared streaming data sessions;
causing one or more of the recognized second objects to be obscured in the streaming data session, wherein,
the causing is based on an application of the defined rules to the recognized category for the one or more second objects, and one or more of the recognized location of the one or more second objects or the recognized physical characteristics of the one or more second objects, and
at least one of the second objects is obscured based on application of the portion of the explicit display rules.