Systems and methods for detecting character encodings of text streams

1. A computer-implemented method for detecting character encodings of text streams, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

identifying a request to identify a character encoding of a text stream;

dividing the text stream to identify a plurality of n-grams within the text stream;

identifying a plurality of vectors within a vector space, each vector representing a potential character encoding by a plurality of expected n-grams within the potential character encoding;

generating, based at least in part on the plurality of n-grams within the text stream and based at least in part on an inverse frequency of appearance of each of the plurality of n-grams within the plurality of vectors, a representative vector describing the text stream;

identifying the character encoding of the text stream based on identifying a similarity between the representative vector and a neighboring vector within the plurality of vectors representing the character encoding.

2. The computer-implemented method of claim 1 , wherein generating the representative vector describing the text stream comprises, for each n-gram within the plurality n-grams, generating an importance value for the n-gram by:

identifying a term frequency with which the n-gram appears in the text stream;

identifying the inverse frequency of appearance of the n-gram by calculating a logarithm of an inverse of a proportion of vectors within the plurality of vectors including the n-gram;

weighting the term frequency by the inverse frequency of appearance.

3. The computer-implemented method of claim 1 , wherein:

generating the representative vector describing the text stream comprises generating the representative vector based on the plurality of vectors being represented with a first dimensionality;

identifying the similarity between the representative vector and the neighboring vector comprises identifying the similarity between the representative vector and the neighboring vector based on the plurality of vectors being represented with a second dimensionality that is smaller than the first dimensionality.

4. The computer-implemented method of claim 1 , wherein generating the representative vector describing the text stream comprises:

identifying a target dimensionality of the representative vector;

identifying a weighted term frequency with which each n-gram within the plurality of n-grams appears in the text stream;

identifying a subset of the plurality of n-grams with highest weighted term frequencies;

populating the representative vector with the subset of the plurality of n-grams.

5. The computer-implemented method of claim 1 , wherein identifying the request to identify the character encoding of the text stream comprises at least one of:

determining that the text stream lacks a metadata indicator of a defined character encoding for the text stream;

determining that the metadata indicator of the of the defined character encoding for the text stream is incorrect.

6. The computer-implemented method of claim 1 , wherein identifying the request to identify the character encoding of the text stream comprises determining that a probability that the text stream comprises more than one language is above a predetermined threshold.

7. The computer-implemented method of claim 1 , further comprising at least one of:

rendering the text stream according to the character encoding of the text stream;

analyzing textual content of the text stream according to the character encoding of the text stream.

8. The computer-implemented method of claim 1 , wherein identifying the character encoding of the text stream based on identifying the similarity between the representative vector and the neighboring vector comprises determining that the neighboring vector is the nearest neighbor to the representative vector within the vector space.

9. The computer-implemented method of claim 1 , wherein identifying the character encoding of the text stream comprises identifying a character encoding that comprises:

a first character set used within a first language but not a second language;

a second character set used within the second language but not the first language.

10. A system for detecting character encodings of text streams, the system comprising:

an identification module programmed to identify a request to identify a character encoding of a text stream;

a division module programmed to divide the text stream to identify a plurality of n-grams within the text stream;

a vector module programmed to identify a plurality of vectors within a vector space, each vector representing a potential character encoding by a plurality of expected n-grams within the potential character encoding;

a generation module programmed to generate, based at least in part on the plurality of n-grams within the text stream and based at least in part on an inverse frequency of appearance of each of the plurality of n-grams within the plurality of vectors, a representative vector describing the text stream;

a similarity module programmed to identify the character encoding of the text stream based on identifying a similarity between the representative vector and a neighboring vector within the plurality of vectors representing the character encoding;

at least one processor configured to execute the identification module, the division module, the vector module, the generation module, and the similarity module.

11. The system of claim 10 , wherein the generation module is programmed to generate the representative vector describing the text stream by, for each n-gram within the plurality n-grams, generating an importance value for the n-gram by:

identifying a term frequency with which the n-gram appears in the text stream;

identifying the inverse frequency of appearance of the n-gram by calculating a logarithm of an inverse of a proportion of vectors within the plurality of vectors including the n-gram;

weighting the term frequency by the inverse frequency of appearance.

12. The system of claim 10 , wherein:

the generation module is programmed to generate the representative vector describing the text stream by generating the representative vector based on the plurality of vectors being represented with a first dimensionality;

the similarity module is programmed to identify the similarity between the representative vector and the neighboring vector by identifying the similarity between the representative vector and the neighboring vector based on the plurality of vectors being represented with a second dimensionality that is smaller than the first dimensionality.

13. The system of claim 10 , wherein the generation module is programmed to generate the representative vector describing the text stream by:

identifying a target dimensionality of the representative vector;

identifying a weighted term frequency with which each n-gram within the plurality of n-grams appears in the text stream;

identifying a subset of the plurality of n-grams with highest weighted term frequencies;

populating the representative vector with the subset of the plurality of n-grams.

14. The system of claim 10 , wherein the identification module is programmed to identify the request to identify the character encoding of the text stream by at least one of:

determining that the text stream lacks a metadata indicator of a defined character encoding for the text stream;

determining that the metadata indicator of the of the defined character encoding for the text stream is incorrect.

15. The system of claim 10 , wherein the identification module is programmed to identify the request to identify the character encoding of the text stream by determining that a probability that the text stream comprises more than one language is above a predetermined threshold.

16. The system of claim 10 , further comprising an encoding module programmed to at least one of:

rendering the text stream according to the character encoding of the text stream;

analyzing textual content of the text stream according to the character encoding of the text stream.

17. The system of claim 10 , wherein the similarity module is programmed to identify the character encoding of the text stream based on identifying the similarity between the representative vector and the neighboring vector by determining that the neighboring vector is the nearest neighbor to the representative vector within the vector space.

18. The system of claim 10 , wherein the similarity module is programmed to identify the character encoding of the text stream by identifying a character encoding that comprises:

a first character set used within a first language but not a second language;

a second character set used within the second language but not the first language.

19. A computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

identify a request to identify a character encoding of a text stream;

divide the text stream to identify a plurality of n-grams within the text stream;

identify a plurality of vectors within a vector space, each vector representing a potential character encoding by a plurality of expected n-grams within the potential character encoding;

generate, based at least in part on the plurality of n-grams within the text stream and based at least in part on an inverse frequency of appearance of each of the plurality of n-grams within the plurality of vectors, a representative vector describing the text stream;

identify the character encoding of the text stream based on identifying a similarity between the representative vector and a neighboring vector within the plurality of vectors representing the character encoding.

20. The computer-readable storage medium of claim 19 , wherein the one or more computer-executable instructions cause the computing device to generate the representative vector describing the text stream by causing the device to, for each n-gram within the plurality n-grams, generate an importance value for the n-gram by:

identifying a term frequency with which the n-gram appears in the text stream;

identifying the inverse frequency of appearance of the n-gram by calculating a logarithm of an inverse of a proportion of vectors within the plurality of vectors including the n-gram;

weighting the term frequency by the inverse frequency of appearance.