System and method for quality evaluation of collaborative text inputs

1. A processor-implemented method for quality evaluation of collaborative text input, comprising:

receiving, via one or more hardware processors, an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;

extracting a plurality of features from the input data, via the one or more hardware processors;

constructing, via the one or more hardware processors, a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and

modelling, via the one or more hardware processors, the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.

2. The method of claim 1 , wherein the post-edit submission is determined to be improved over prior post-edit submissions if the first binary classifier determines a binary class as 1, and post-edit submission is determined not to be improved if the first binary classifier determines the binary class as 0.

3. The method of claim 1 , wherein the task comprises a translation of text, for each post-edit submission, the post-edit feature vector comprises an edit vector of a current submission, and wherein the post-edit feature vector comprises of ratios of word-wise inserts, deletions and substitutions made by a contributor to a selected parent submission with total number of words, word length of a source sentence associated with the task, and quantized and normalized time taken by the contributor to perform the task.

4. The method of claim 1 , wherein for every post-edit submission, the contributor representation feature vector comprises one of one-hot vector encoding of dimensions of total number of contributors and learned contributor embedding by feeding the one-hot-vectors associated with the contributors to a neural network (NN) with a single linear hidden layer.

5. The method of claim 4 , further comprising learning the contributor embedding during the training of the LSTM network.

6. The method of claim 1 , further comprising training a second binary classifier, by the LSTM network to predict a probability of a contributor improving upon prior submissions.

7. The method of claim 6 , wherein predicting the probability comprises:

mapping a history of prior post-edit submissions into an intermediate embedding, the history of prior post-edit submissions obtained from the input data;

applying output of the LSTM networks and current contributor embedding to a neural network (NN), wherein the output of the LSTM networks comprises the history embedding at the last time stamp; and

predicting the probability of a contributor improving upon the prior submissions based on the output of the NN.

8. The method of claim 1 , further comprising training an encoder-decoder architecture to predict sequence of ranks for each of the plurality of sequence of submissions based on the plurality of sequences of the concatenated feature vectors.

9. The method of claim 8 , wherein training the encoder-decoder architecture comprises:

applying the sequence of concatenated feature vectors into the encoder-decoder architecture, the encoder-decoder architecture predicts a sequence of binary labels generated from selected expert provided ranks of the post-edit submissions as ordinal labels;

obtaining, from the encoder, the embedding of the last hidden state which represents the history of post-edit;

feeding the embedding to the decoder which learns the sequence of binary labels for each time step to provide a binary sequence; and

calculating ranks based on the binary sequence obtained from the encoder-decoder.

10. A system for quality evaluation of collaborative text input, the system comprising:

at least one memory storing instructions; and

one or more hardware processors coupled to said at least one memory, wherein said one or more hardware processors are configured by said instructions to:

receive an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;

extract a plurality of features from the input data;

construct a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and

model the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.

11. The system of claim 10 , wherein the one or more hardware processors are configured by the instructions to determine if the post-edit submission is improved over prior post-edit submissions when the first binary classifier determines a binary class as 1, and determines the post-edit submission not to be improved when the first binary classifier determines the binary class as 0.

12. The system of claim 10 , wherein the task comprises a translation of text, for every post-edit submission, the post-edit feature vector comprises an edit vector of a current submission, and wherein the post-edit feature vector comprises of ratios of word-wise inserts, deletions and substitutions made by a contributor to a selected parent submission with total number of words; word length of a source sentence, associated with the task, and quantized and normalized time taken by the contributor to perform the task.

13. The system of claim 10 , wherein for every post-edit submission, the contributor representation feature vector comprises one of one-hot vector encoding of dimensions of total number of contributors and learned contributor embedding by feeding the one-hot-vectors associated with the contributors to a neural network (NN) with a single linear hidden layer.

14. The system of claim 13 , wherein the one or more hardware processors are configured by the instructions to learn the contributor embedding during the training of the LSTM network.

15. The system of claim 10 , wherein the one or more hardware processors are configured by the instructions to train a second binary classifier, by the LSTM network to predict a probability of a contributor improving upon prior submissions.

16. The system of claim 15 , wherein to predict the probability, the one or more hardware processors are configured by the instructions to:

map a history of prior post-edit submissions into an intermediate embedding, the history of prior post-edit submissions obtained from the input data;

apply output of the LSTM networks and current contributor embedding to a neural network (NN), wherein the output of the LSTM networks comprises the history embedding at the last time stamp; and

predict the probability of a contributor improving upon the prior submissions based on the output of the NN.

17. The system of claim 10 , wherein the one or more hardware processors are configured by the instructions to train an encoder-decoder architecture to predict sequence of ranks for each of the plurality of sequence of submissions based on the plurality of sequences of the concatenated feature vectors.

18. The system of claim 17 , wherein to train the encoder-decoder architecture, the one or more hardware processors are configured by the instructions to:

apply the sequence of concatenated feature vectors into the encoder-decoder architecture, the encoder-decoder architecture predicts a sequence of binary labels generated from selected expert provided ranks of the post-edit submissions as ordinal labels;

obtain, from the encoder, the embedding of the last hidden state which represents the history of post-edit;

feed the embedding to the decoder which learns the sequence of binary labels for each time step to provide a binary sequence; and

calculate ranks based on the binary sequence obtained from the encoder-decoder.

19. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method for quality evaluation of collaborative text input, the method comprising:

receiving, via one or more hardware processors, an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;

extracting a plurality of features from the input data, via the one or more hardware processors;

constructing, via the one or more hardware processors, a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and

modelling, via the one or more hardware processors, the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.