Detecting messages with offensive content
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on computer storage medium, to facilitate interception of messages that include offensive content. In one aspect, a method includes actions of receiving input on a user device that includes message content, determining, on the user device, whether the message content includes offensive content, and in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.
1. A method comprising:
receiving, by one or more computers, first data that represents message content obtained from a first user device;
determining, by one or more computers and prior to communication of second data to a second user device that, when processed by the second user device, causes the second user device to display the message content on a display of the second user device, whether the first data represents message content that is likely offensive content, wherein determining whether the received first data represents message content that is likely offensive content comprises:
providing, by one or more computers, at least a portion of the received first data as an input to a machine learning model that has been trained to detect a likelihood that a message includes offensive content; and
obtaining, by one or more computers, output data that was generated by the machine learning model based on the machine learning model processing the provided first data;
in response to determining, by one or more computers and based on the obtained output data, that the first data represents message content that is likely offensive content, generating, by one or more computers, alert data that, when processed by the first user device, causes the first user device to display a visualization that provides a user of the first user device with an opportunity to reconsider whether the message content is to be communicated to the second user device; and
providing, by one or more computers, the generated alert data to the first user device.
2. The method of claim 1 , wherein the one or more computers includes the first user device.
3. The method of claim 1 , wherein the first data and second data represent the same message content.
4. The method of claim 1 , the method comprising:
obtaining, by one or more computers, third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device; and
based on obtaining the third data, causing, by one or more computers, the second data to be communicated to the second user device.
5. The method of claim 4 , wherein obtaining third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device comprises:
obtaining, by one or more computers, data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be communicated to the second user device.
6. The method of claim 1 , the method comprising:
obtaining, by one or more computers, fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device; and
based on obtaining the fourth data, causing, by one or more computers, the second data to be discarded without communicating the second data to the second user device.
7. The method of claim 6 , wherein obtaining fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device comprises:
obtaining, by one or more computers, data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be discarded without transmitting the second data to the second user device.
8. A system comprising:
one or more computers; and
one or more computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations, the operations comprising:
receiving, by one or more computers, first data that represents message content obtained from a first user device;
determining, by one or more computers and prior to communication of second data to a second user device that, when processed by the second user device, causes the second user device to display the message content on a display of the second user device, whether the first data represents message content that is likely offensive content, wherein determining whether the received first data represents message content that is likely offensive content comprises:
providing, by one or more computers, at least a portion of the received first data as an input to a machine learning model that has been trained to detect a likelihood that a message includes offensive content; and
obtaining, by one or more computers, output data that was generated by the machine learning model based on the machine learning model processing the provided first data;
in response to determining, by one or more computers and based on the obtained output data, that the first data represents message content that is likely offensive content, generating, by one or more computers, alert data that, when processed by the first user device, causes the first user device to display a visualization that provides a user of the first user device with an opportunity to reconsider whether the message content is to be communicated to the second user device; and
providing, by one or more computers, the generated alert data to the first user device.
9. The system of claim 8 , wherein the one or more computers includes the first user device.
10. The system of claim 8 , wherein the first data and second data represent the same message content.
11. The system of claim 8 , the operations comprising:
obtaining, by one or more computers, third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device; and
based on obtaining the third data, causing, by one or more computers, the second data to be communicated to the second user device.
12. The system of claim 11 , wherein obtaining third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device comprises:
obtaining, by one or more computers, data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be communicated to the second user device.
13. The system of claim 8 , the operations comprising:
obtaining, by one or more computers, fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device; and
based on obtaining the fourth data, causing, by one or more computers, the second data to be discarded without communicating the second data to the second user device.
14. The system of claim 13 , wherein obtaining fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device comprises:
obtaining, by one or more computers, data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be discarded without transmitting the second data to the second user device.
15. One or more computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
receiving first data that represents message content obtained from a first user device;
determining, prior to communication of second data to a second user device that, when processed by the second user device, causes the second user device to display the message content on a display of the second user device, whether the first data represents message content that is likely offensive content, wherein determining whether the received first data represents message content that is likely offensive content comprises:
providing at least a portion of the received first data as an input to a machine learning model that has been trained to detect a likelihood that a message includes offensive content; and
obtaining, output data that was generated by the machine learning model based on the machine learning model processing the provided first data;
in response to determining, based on the obtained output data, that the first data represents message content that is likely offensive content, generating alert data that, when processed by the first user device, causes the first user device to display a visualization that provides a user of the first user device with an opportunity to reconsider whether the message content is to be communicated to the second user device; and
providing the generated alert data to the first user device.
16. The computer-readable media of claim 15 , wherein the one or more computers includes the first user device.
17. The computer-readable media of claim 15 , wherein the first data and second data represent the same message content.
18. The computer-readable media of claim 15 , the operations comprising:
obtaining third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device; and
based on obtaining the third data, causing the second data to be communicated to the second user device.
19. The computer-readable media of claim 18 , wherein obtaining third data that is indicative of a user interaction indicating that the user of the first user device affirmed the message content is to be communicated to the second user device comprises:
obtaining data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be communicated to the second user device.
20. The computer-readable media of claim 15 , the operations comprising:
obtaining fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device; and
based on obtaining the fourth data the second data to be discarded without communicating the second data to the second user device.
21. The computer-readable media of claim 20 , wherein obtaining fourth data that is indicative of a user interaction indicating that the user of the first user device affirmed that the message content is not to be communicated to the second user device comprises:
obtaining data that is indicative of a selection of a visualization of an icon on a display of the first user device that indicates the message content is to be discarded without transmitting the second data to the second user device.