Adaptive window screening for large text content security
Systems and methods are disclosed for detecting and remediating safety issues in electronic text received by an electronic platform. The electronic text is screened using an adaptive text window that splits the text into segments. Each segment is tested for safety issues, e.g., for content moderation or computer security. Any segment that is identified as containing a potential safety issue is iteratively split and tested using progressively smaller window sizes. The testing of the segments includes generating a safety score, e.g., using a content safety model, which is compared to an adaptive threshold. The threshold may be adjusted in each iteration based on the change in window size. The dynamic screening approach continues until all segments are determined to be safe or until a minimum window size is reached and the electronic text is accepted or rejected based on the safety scores for the segments of the electronic text.
1 . A computer-implemented method for detecting safety issues in electronic text, the method comprising:
receiving electronic text in an electronic platform, the electronic platform receives electronic text from a plurality of sources;
splitting the electronic text into a plurality of segments with a window size;
testing each segment for safety issues, comprising:
determining a safety score for each segment in the plurality of segments with a content safety model;
identifying any segments with potential safety issues based on the safety score for each segment;
iteratively splitting any segment with potential safety issues with progressively smaller window sizes and testing each segment for the potential safety issues; and
accepting the electronic text or rejecting the electronic text for the electronic platform based on the safety scores for the segments of the electronic text.
2 . The method of claim 1 , wherein identifying any segments with potential safety issues comprises comparing the safety score for each segment to a safety threshold, the method further comprising adjusting the safety threshold based on the window size in each iteration.
3 . The method of claim 2 , further comprising comparing the safety score for each segment to an unsafe threshold that identifies segments as unsafe, wherein iteratively splitting and testing is performed until any segment is identified as unsafe.
4 . The method of claim 1 , wherein any segment that is not identified with potential safety issues is deemed safe and is not split in further iterations.
5 . The method of claim 1 , wherein iteratively splitting and testing is performed until no segments are identified with potential safety issues or a minimum window size is reached.
6 . The method of claim 1 , wherein accepting the electronic text or rejecting the electronic text for the electronic platform is based on whether any segment is identified with potential safety issues after all iterations are performed.
7 . The method of claim 1 , wherein accepting the electronic text or rejecting the electronic text for the electronic platform is based on an aggregate safety score for all segments.
8 . The method of claim 1 , wherein splitting the electronic text into the plurality of segments and splitting any segment with potential safety issues with progressively smaller window sizes uses overlapping segments.
9 . The method of claim 1 , wherein splitting any segment with potential safety issues with progressively smaller window sizes comprises overlapping the any segments with a portion of one or more segments that are not identified with potential safety issues.
10 . A system for detecting safety issues in electronic text, the system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, causes the system to perform operations comprising:
receiving electronic text in an electronic platform, the electronic platform receives electronic text from a plurality of sources;
splitting the electronic text into a plurality of segments with a window size;
testing each segment for safety issues, comprising:
determining a safety score for each segment in the plurality of segments with a content safety model;
identifying any segments with potential safety issues based on the safety score for each segment;
iteratively splitting any segment with potential safety issues with progressively smaller window sizes and testing each segment for the potential safety issues; and
accepting the electronic text or rejecting the electronic text for the electronic platform based on the safety scores for the segments of the electronic text.
11 . The system of claim 10 , wherein identifying any segments with potential safety issues is performed by comparing the safety score for each segment to a safety threshold, the operations further comprise adjusting the safety threshold based on the window size in each iteration.
12 . The system of claim 11 , the operations further comprise comparing the safety score for each segment to an unsafe threshold that identifies segments as unsafe, wherein iteratively splitting and testing is performed until any segment is identified as unsafe.
13 . The system of claim 10 , wherein any segment that is not identified with potential safety issues is deemed safe and is not split in further iterations.
14 . The system of claim 10 , wherein iteratively splitting and testing is performed until no segments are identified with potential safety issues or a minimum window size is reached.
15 . The system of claim 10 , wherein accepting the electronic text or rejecting the electronic text for the electronic platform is based on whether any segment is identified with potential safety issues after all iterations are performed.
16 . The system of claim 10 , wherein accepting the electronic text or rejecting the electronic text for the electronic platform is based on an aggregate safety score for all segments.
17 . The system of claim 10 , wherein splitting the electronic text into the plurality of segments and splitting any segment with potential safety issues with progressively smaller window sizes uses overlapping segments.
18 . The system of claim 10 , wherein splitting any segment with potential safety issues with progressively smaller window sizes comprises overlapping the any segments with a portion of one or more segments that are not identified with potential safety issues.