Resisting anonymous scraping
View Patent ↗A system, method, and apparatus are provided for applying a technique for resisting or hindering scraping of a website or other repository of electronic data. When a connection from an entity is received at the website, if no signal or information is received that identifies the entity (e.g., a user identifier, a cookie), or the information is insufficient to discriminate the entity from other entities (e.g., an IP address that is or may be shared), one or more techniques are applied, such as: delaying loading of a page or page component, rendering a page (or page component) as an image, rendering only a portion of a page, applying a CAPTCHA, redirecting the entity to a login page, and/or others. Thus, an anti-scraping technique is activated for a connection from what could be a scraper that has purged its browser data of some or all identifying information.
1. A method comprising:
at a computer system supporting a website, establishing a first communication connection with a client device;
determining that, during establishment of the first communication connection, no signal was received that identifies the client device or a user of the client device;
responsive to determining that no signal was received that identifies the client device or a user of the client device, transmitting to the client device, during the first communication connection, a token configured to identify the client device or a user of the client device, wherein the token is configured to be returned to the computer system when a subsequent communication is established between the client device and the computer system;
at the computer system, after termination of the first communication connection, establishing a second communication connection with the client device;
determining that the token was not received from the client device during establishment of the second communication connection; and
responsive to determining that the token was not received during establishment of the second communication connection, activating one or more techniques that enable resisting scraping of the website.
2. The method of claim 1 , wherein:
the token is a cookie; and
the token, if received during establishment of the second communication connection, is a signal that identifies the client device or the user of the client device.
3. The method of claim 1 , wherein each of the following is a signal that identifies the client device or the user of the client device:
the token;
an identifier of the user assigned by the website;
a cookie issued to the client device by the website; and
a fingerprint of the client device.
4. The method of claim 1 , wherein techniques for resisting scraping of the website include:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.
5. The method of claim 4 , wherein techniques for resisting scraping of the website further include:
a CAPTCHA; and
redirecting the client device to a login page.
6. An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
at a computer system supporting a website, establish a first communication connection with a client device;
determine that, during establishment of the first communication connection, no signal was received that identifies the client device or a user of the client device;
responsive to determining that no signal was received that identifies the client device or a user of the client device, transmit to the client device, during the first communication connection, a token configured to identify the client device or a user of the client device, wherein the token is configured to be returned to the computer system when a subsequent communication is established between the client device and the computer system;
at the computer system, after termination of the first communication connection, establish a second communication connection with the client device;
determine that the token was not received from the client device during establishment of the second communication connection; and
responsive to determining that the token was not received during establishment of the second communication connection, activate one or more techniques that enable resisting scraping of the website.
7. The apparatus of claim 6 , wherein:
the token is a cookie; and
the token, if received during establishment of the second communication connection, is a signal that identifies the client device or the user of the client device.
8. The apparatus of claim 6 , wherein each of the following is a signal that identifies the client device or the user of the client device:
the token;
an identifier of the user assigned by the website;
a cookie issued to the client device by the website; and
a fingerprint of the client device.
9. The apparatus of claim 6 , wherein techniques for resisting scraping of the website include:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.
10. The apparatus of claim 9 , wherein techniques for resisting scraping of the website further include:
a CAPTCHA; and
redirecting the client device to a login page.
11. A system, comprising:
one or more processors;
a communication module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
establish a first communication connection with a client device, wherein the system supports a website; and
after termination of the first communication connection, establish a second communication connection with the client device; and
an anti-scraping module comprising a non-transitory computer-readable medium storing instructions that, when executed, cause the system to:
determine that, during establishment of the first communication connection, no signal was received that identifies the client device or a user of the client device;
responsive to determining that no signal was received that identifies the client device or a user of the client device, transmit to the client device, during the first communication connection, a token configured to identify the client device or a user of the client device, wherein the token is configured to be returned to the computer system when a subsequent communication is established between the client device and the computer system;
determine that the token was not received from the client device during establishment of the second communication connection; and
responsive to determining that the token was not received during establishment of the second communication connection, activate one or more techniques that enable resisting scraping of the website.
12. The system of claim 11 , wherein:
the token is a cookie; and
the token, if received during establishment of the second communication connection, is a signal that identifies the client device or the user of the client device.
13. The system of claim 11 , wherein each of the following is a signal that identifies the client device or the user of the client device:
the token;
an identifier of the user assigned by the website;
a cookie issued to the client device by the website; and
a fingerprint of the client device.
14. The system of claim 11 , wherein techniques for resisting scraping of the website include:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.
15. The system of claim 14 , wherein techniques for resisting scraping of the website further include:
a CAPTCHA; and
redirecting the client device to a login page.
16. The method of claim 1 , wherein techniques for resisting scraping of the website include:
requesting the user of the client device to login or register; and
when the user of the client device fails to login or register as requested, performing one or more of:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.
17. The method of claim 16 , wherein causing the client device to render only a portion of a page of the website instead of an entirety of the page comprises:
causing the same portion of the page of the website to be rendered each time only a portion of a page of the website is rendered instead of an entirety of the page.
18. The apparatus of claim 6 , wherein techniques for resisting scraping of the website include:
requesting the user of the client device to login or register; and
when the user of the client device fails to login or register as requested, performing one or more of:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.
19. The apparatus of claim 18 , wherein causing the client device to render only a portion of a page of the website instead of an entirety of the page comprises:
causing the same portion of the page of the website to be rendered each time only a portion of a page of the website is rendered instead of an entirety of the page.
20. The system of claim 11 , wherein techniques for resisting scraping of the website include:
requesting the user of the client device to login or register; and
when the user of the client device fails to login or register as requested, performing one or more of:
causing loading of a page of the website on the client device to be delayed;
causing a page of the website that comprises text to be rendered on the client device as an image; and
causing the client device to render only a portion of a page of the website instead of an entirety of the page.