Determining safety risk using natural language processing
A network system coordinates users who provide geographical location-based services to users. Users can provide textual feedback regarding the services provided by other users. To create a safe user experience and encourage users to continue using the services, the network system determines safety risks of the users by analyzing the textual feedback from users along with metadata associated with the textual feedback. For example, the network system determines a safety score for a user based on classifying textual feedback according to different types of safety risk related to topics such as vehicle operation or interpersonal behavior. The network system can train classifiers using feature vectors derived from textual feedback previously received from users of the network system, which may be labeled in a supervised training process. The network system may provide interventions to high-risk users to help mitigate their safety risk and prevent safety incidents from occurring.
1. A method comprising:
receiving, from a client device of a user, textual feedback associated with a provider, the textual feedback including a plurality of words;
determining a feature vector for the textual feedback by characterizing at least the plurality of words using a plurality of dimensions, each dimension of the plurality of dimensions associated with one of a plurality of categories of safety risk, the plurality of categories of safety risk including at least a first category of safety risk and a second category of safety risk different than the first category of safety risk;
determining a first safety sub-score by applying the feature vector to a first classifier associated with the first category of safety risk, the first classifier being a machine learning model trained using a first training feature vector characterizing a first set of textual feedback associated with the first category of safety risk;
determining a second safety sub-score by applying the feature vector to a second classifier associated with the second category of safety risk, the second classifier being a different machine learning model trained using a second training feature vector characterizing a second set of textual feedback associated with the second category of safety risk; and
determining a safety score for the provider using the first safety sub-score and the second safety sub-score.
2. The method of claim 1 , wherein the first safety sub-score denotes a confidence level that the textual feedback belongs to the first category of safety risk and the second safety sub-score denotes another confidence level that the textual feedback belongs to the second category of safety risk.
3. The method of claim 1 , further comprising:
receiving, from the client device of the user, metadata associated with the textual feedback; and
wherein determining the feature vector for the textual feedback further comprises characterizing the metadata using the plurality of dimensions.
4. The method of claim 1 , wherein determining the safety score for the provider comprises:
determining, by a third classifier, a weighted average of the first safety sub-score and the second safety sub-score.
5. The method of claim 4 , wherein determining the weighted average comprises:
determining, by the third classifier, to boost the first safety sub-score based on a plurality of training feature vectors.
6. The method of claim 4 , wherein determining the weighted average comprises:
determining, by the third classifier, to boost the first safety sub-score by using metadata to verify veracity of the plurality of words.
7. The method of claim 6 , wherein the first category of safety risk is associated with vehicle operation and the metadata includes telematics data indicating a speed of a vehicle during service provided by the provider to the user.
8. The method of claim 1 , wherein the first classifier is further trained using a third training feature vector characterizing a third set of textual feedback associated with a third category of safety risk of the plurality of categories of safety risk.
9. The method of claim 1 , wherein the feature vector indicates a level of linguistic similarity between the plurality of words and each of the plurality of categories of safety risk.
10. The method of claim 1 , further comprising:
providing an intervention for presentation on another client device of the provider based on the safety score.
11. A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
receive, from a client device of a user, textual feedback associated with a provider, the textual feedback including a plurality of words;
determine a feature vector for the textual feedback by characterizing at least the plurality of words using a plurality of dimensions, each dimension of the plurality of dimensions associated with one of a plurality of categories of safety risk, the plurality of categories of safety risk including at least a first category of safety risk and a second category of safety risk different than the first category of safety risk;
determine a first safety sub-score by applying the feature vector to a first classifier associated with the first category of safety risk, the first classifier being a machine learning model trained using a first training feature vector characterizing a first set of textual feedback associated with the first category of safety risk;
determine a second safety sub-score by applying the feature vector to a second classifier associated with the second category of safety risk, the second classifier being a different machine learning model trained using a second training feature vector characterizing a second set of textual feedback associated with the second category of safety risk; and
determine a safety score for the provider using the first safety sub-score and the second safety sub-score.
12. The non-transitory computer readable storage medium of claim 11 , wherein the first safety sub-score denotes a confidence level that the textual feedback belongs to the first category of safety risk and the second safety sub-score denotes another confidence level that the textual feedback belongs to the second category of safety risk.
13. The non-transitory computer readable storage medium of claim 11 , having further instructions that when executed by the one or more processors cause the one or more processors to:
receive, from the client device of the user, metadata associated with the textual feedback; and
wherein determining the feature vector for the textual feedback further comprises characterizing the metadata using the plurality of dimensions.
14. The non-transitory computer readable storage medium of claim 11 , wherein determine the safety score for the provider comprises:
determine, by a third classifier, a weighted average of the first safety sub-score and the second safety sub-score.
15. The non-transitory computer readable storage medium of claim 14 , wherein determine the weighted average comprises:
determine, by the third classifier, to boost the first safety sub-score based on a plurality of training feature vectors.
16. The non-transitory computer readable storage medium of claim 14 , wherein determine the weighted average comprises:
determine, by the third classifier, to boost the first safety sub-score by using metadata to verify veracity of the plurality of words.
17. The non-transitory computer readable storage medium of claim 16 , wherein the first category of safety risk is associated with vehicle operation and the metadata includes telematics data indicating a speed of a vehicle during service provided by the provider to the user.
18. The non-transitory computer readable storage medium of claim 11 , having further instructions that when executed by the one or more processors cause the one or more processors to:
provide an intervention for presentation on another client device of the provider based on the safety score.
19. A system comprising:
one or more processors; and
a computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
receive, from a client device of a user, textual feedback associated with a provider, the textual feedback including a plurality of words;
determine a feature vector for the textual feedback by characterizing at least the plurality of words using a plurality of dimensions, each dimension of the plurality of dimensions associated with one of a plurality of categories of safety risk, the plurality of categories of safety risk including at least a first category of safety risk and a second category of safety risk different than the first category of safety risk;
determine a first safety sub-score by applying the feature vector to a first classifier associated with the first category of safety risk, the first classifier being a machine learning model trained using a first training feature vector characterizing a first set of textual feedback associated with the first category of safety risk;
determine a second safety sub-score by applying the feature vector to a second classifier associated with the second category of safety risk, the second classifier being a different machine learning model trained using a second training feature vector characterizing a second set of textual feedback associated with the second category of safety risk; and
determine a safety score for the provider using the first safety sub-score and the second safety sub-score.
20. The system of claim 19 , wherein determine the safety score for the provider comprises:
determine to boost the first safety sub-score based on a plurality of training feature vectors; and
determine, by a third classifier, a weighted average of the boosted first safety sub-score and the second safety sub-score.