Systems and methods for identifying problematic activity on messaging platforms

1 . A method for dynamic feature weighting in a system for mitigating distribution of offensive content, the method comprising:

obtaining, using one or more computers, data associated with a particular entity;

extracting, using one or more computers, a plurality of features from the obtained data, wherein the features are associated with behavioral or psychological indicators of a form of mental instability;

dynamically adjusting, using one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing the extracted features, wherein the adjustment is performed according to a predetermined algorithm or machine learning model;

providing, using one or more computers, the feature vector and the corresponding weights as input to a machine learning model that has been trained to determine a likelihood that a user is exhibiting one or more forms of mental instability based on processing usage data collected from a user device of an entity;

processing, using one or more computers, the feature vector and weights through the machine learning model to generate output data indicating a likelihood that the particular entity is exhibiting one or more forms of mental instability;

determining, using one or more computers, whether the particular entity is exhibiting one or more forms of mental instability based on the generated output data; and based on a determination that the particular entity is exhibiting one or more forms of mental instability, performing, using one or more computers, one or more operations to mitigate the impact of the one or more forms of mental instability on the particular entity or other entities, wherein the one or more operations comprise restricting, filtering, or modifying the distribution of offensive content associated with the particular entity.

2 . The method of claim 1 , wherein obtaining, using one or more computers, data associated with a particular entity comprises:

obtaining, using one or more computers, data from message content, social media post content, content referenced by a uniform resource locator (URL) shared by the user in a message or social media post, or content a user has viewed or read on a user device.

3 . The method of claim 1 , wherein adjusting, using one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

ranking, using one or more computers, the extracted features to emphasize or deemphasize certain features.

4 . The method of claim 1 , the method further comprising:

determining, using one or more computers and based on the obtained data, whether a user is trending in a particular direction of mental instability; and

based on determining that the user is trending towards mental instability, adjusting, using one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the determined trend.

5 . The method of claim 1 , the method further comprising:

determining, using one or more computers and based on the obtained data, that the user is consuming content related to suicide; and

based on determining that the user is consuming content related to suicide, adjusting, using one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on determining that the user is consuming content related to suicide.

6 . The method of claim 1 , the method further comprising:

mapping, using one or more computers, the extracted features to one or more symptoms of mental instability.

7 . The method of claim 6 , wherein adjusting, using one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

adjusting, using one or more computers, weights corresponding to one or more fields of the feature vector based on the mapping.

8 . A system for dynamic feature weighting in a system for mitigating distribution of offensive content, the system comprising:

one or more computers; and

one or more computer-readable storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

obtaining, using the one or more computers, data associated with a particular entity;

extracting, using the one or more computers, a plurality of features from the obtained data, wherein the features are associated with behavioral or psychological indicators of a form of mental instability:

dynamically adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing the extracted features, wherein the adjustment is performed according to a predetermined algorithm or machine learning model;

providing, using the one or more computers, the feature vector and the corresponding weights as input to a machine learning model that has been trained to determine a likelihood that a user is exhibiting one or more forms of mental instability based on processing usage data collected from a user device of an entity;

processing, using the one or more computers, the feature vector and weights through the machine learning model to generate output data indicating a likelihood that the particular entity is exhibiting one or more forms of mental instability:

determining, using the one or more computers, whether the particular entity is exhibiting one or more forms of mental instability based on the generated output data; and

based on a determination that the particular entity is exhibiting one or more forms of mental instability, performing, using the one or more computers, one or more operations to mitigate the impact of the one or more forms of mental instability on the particular entity or other entities, wherein the one or more operations comprise restricting, filtering, or modifying the distribution of offensive content associated with the particular entity.

9 . The system of claim 8 , wherein obtaining, using the one or more computers, data associated with a particular entity comprises:

obtaining, using the one or more computers, data from message content, social media post content, content referenced by a uniform resource locator (URL) shared by the user in a message or social media post, or content a user has viewed or read on a user device.

10 . The system of claim 8 , wherein adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

ranking, using the one or more computers, the extracted features to emphasize or deemphasize certain features.

11 . The system of claim 8 , the operations further comprising:

mapping, using the one or more computers, the extracted features to one or more symptoms of mental instability.

12 . The system of claim 8 , wherein adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

adjusting, using the one or more computers, weights corresponding to one or more fields of the feature vector based on the mapping.

13 . The system of claim 8 , the operations further comprising:

determining, using the one or more computers and based on the obtained data, whether a user is trending in a particular direction of mental instability; and

based on determining that the user is trending towards mental instability, adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the determined trend.

14 . The system of claim 8 , the operations further comprising:

determining, using the one or more computers and based on the obtained data, that the user is consuming content related to suicide; and

based on determining that the user is consuming content related to suicide, adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on determining that the user is consuming content related to suicide.

15 . One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

obtaining, using the one or more computers, data associated with a particular entity;

extracting, using the one or more computers, a plurality of features from the obtained data, wherein the features are associated with behavioral or psychological indicators of a form of mental instability;

dynamically adjusting, using the one or more computers and based on the obtained data, weights corresponding to one or more fields of a feature vector representing the extracted features, wherein the adjustment is performed according to a predetermined algorithm or machine learning model;

providing, using the one or more computers, the feature vector and the corresponding weights as input to a machine learning model that has been trained to determine a likelihood that a user is exhibiting one or more forms of mental instability based on processing usage data collected from a user device of an entity;

processing, using the one or more computers, the feature vector and weights through the machine learning model to generate output data indicating a likelihood that the particular entity is exhibiting one or more forms of mental instability;

determining, using the one or more computers, whether the particular entity is exhibiting one or more forms of mental instability based on the generated output data; and

based on a determination that the particular entity is exhibiting one or more forms of mental instability, performing, using the one or more computers, one or more operations to mitigate the impact of the one or more forms of mental instability on the particular entity or other entities, wherein the one or more operations comprise restricting, filtering, or modifying the distribution of offensive content associated with the particular entity.

16 . The computer readable-storage media of claim 15 , wherein obtaining data associated with a particular entity comprises:

obtaining data from message content, social media post content, content referenced by a uniform resource locator (URL) shared by the user in a message or social media post, or content a user has viewed or read on a user device.

17 . The computer readable-storage media of claim 15 , wherein adjusting, based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

ranking the extracted features to emphasize or deemphasize certain features.

18 . The computer readable-storage media of claim 15 , the operations further comprising:

determining, based on the obtained data, whether a user is trending in a particular direction of mental instability; and

based on determining that the user is trending towards mental instability, adjusting, based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the determined trend.

19 . The computer readable-storage media of claim 15 , the method further comprising:

determining, based on the obtained data, that the user is consuming content related to suicide; and

based on determining that the user is consuming content related to suicide, adjusting, based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on determining that the user is consuming content related to suicide.

20 . The computer readable-storage media of claim 15 , the operations further comprising:

mapping the extracted features to one or more symptoms of mental instability.

21 . The computer readable-storage media of claim 20 , wherein adjusting, based on the obtained data, weights corresponding to one or more fields of a feature vector representing features extracted from user content based on the obtained data comprises:

adjusting weights corresponding to one or more fields of the feature vector based on the mapping.