OXYLABS, UAB

Systems and methods to intelligently optimize data collection requests are disclosed. In one embodiment, systems are configured to identify and select a complete set of suitable parameters to execute the data collection requests. In another embodiment, systems are configured to identify and select a partial set of suitable parameters to execute the data collection requests. The present embodiments can implement machine learning algorithms to identify and select the suitable parameters according to the nature of the data collection requests and the targets. Moreover, the embodiments provide systems and methods to generate feedback data based upon the effectiveness of the data collection parameters. Furthermore, the embodiments provide systems and methods to score the set of suitable parameters based on the feedback data and the overall cost, which are then stored in an internal database.

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for proxy pool management on a client-specific basis. Each client has his own configuration for a proxy pool. While multiple clients may share a proxy pool and pool indexes, each client has his own cursor designating its own position within the pool index. When a request is made by a client, the proxy is selected at the position of the client's cursor. By having client-specific cursors, each client can take advantage of a greater level of unique IP addresses as offered by the pool.

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for caching media resources during a scraping operation. Web resources needed by webpage are stored in a cache that is used by multiple browsers that are scraping the webpage. When an unexpired entry for the web resource is present in the cache, a browser retrieves the web resource and cache instead of making a request from the webpage. This offers a technological improvement of reducing the traffic burden on proxy servers needed to forward the scraping requests and responses.

Systems and methods to intelligently optimize data collection requests are disclosed. In one embodiment, systems are configured to identify and select a complete set of suitable parameters to execute the data collection requests. In another embodiment, systems are configured to identify and select a partial set of suitable parameters to execute the data collection requests. The present embodiments can implement machine learning algorithms to identify and select the suitable parameters according to the nature of the data collection requests and the targets. Moreover, the embodiments provide systems and methods to generate feedback data based upon the effectiveness of the data collection parameters. Furthermore, the embodiments provide systems and methods to score the set of suitable parameters based on the feedback data and the overall cost, which are then stored in an internal database.

A computer-implemented method is provided that enables remote control of a headless browser. A message from a client to open a connection to a headless browser running on a device remote from the client is received. A command to provision the browser is sent to the headless browser. The command may configure the browser to appear as if the browser is controlled by a human. A command for controlling the headless browser is received from the client. The command is sent to the headless browser for execution. A response to the command is received from the headless browser. Finally, the response is forwarded to the client.

Systems and methods to manage and efficiently perform authorization of multiple proxy clients are disclosed. Furthermore, systems and methods to measure and check whether the web traffic of one or more client devices has reached a permissible limit of web traffic assigned by the proxy service provider. Specifically, a proxy is configured to gather and save authorization information of one or more clients within its memory. Therefore, the proxy server can verify and authorize one or more clients by utilizing the data from its memory. Furthermore, the proxy is configured to measure and report the utilized web traffic of one or more client devices to a messaging platform. In another aspect, systems and methods to check whether one or more client devices have reached a permissible amount of web traffic limit are disclosed.

Systems and methods to manage and efficiently implement functional proxy services are disclosed. In the proxy services, a single instance of exit-node is connected to at least two or multiple supernodes at any given time. One of the plurality of supernodes is configured to ping and send diagnostic requests to the connected exit-node through a network. The exit-node is directed to send the pong message and diagnostic response data to a different supernode from among the plurality of supernodes connected to the exit-node. Likewise, a client's request is received by an element of the proxy service provider and forwarded to a specific supernode capable of forwarding the client's request to the exit-node. After performing the client's request, the exit-node returns response data to a different supernode from among the plurality of supernodes connected to the exit-node.

Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by dynamically updating scraping parameters. A scrape system may retrieve a webpage addressed at a target URL. The scrape system may compile an object list from the webpage. The scrape system may determine a number of objects in the object list. Based on the determined number of objects, the scrape system may determine a next time to retrieve the webpage addressed at the target URL such that, when the determined number of objects is greater, the next time is sooner. When the determined next time occurs, the scrape system may re-retrieve the webpage addressed at the target URL.

Systems and methods herein provide for a proxy infrastructure. In the proxy infrastructure, a network element (e.g., a supernode) is connected with a plurality of exit nodes. At one of a plurality of messenger units of the proxy infrastructure, a proxy protocol request is received directly from a client computing device. The proxy protocol request specifies a request and a target. In response the proxy protocol request, a selection is made between one between one of the plurality of exit nodes. A message with the request is sent from the messenger to the supernode connected with the selected exit node. Finally, the message is sent from the supernode to the selected exit node to forward the request to the target.

System and method for efficiently implementing scalable, highly efficient decentralized proxy services through proxy infrastructures situated in different geo-locations. In one aspect, the systems and methods enable users from any geographical location to send requests to the geographically closest proxy infrastructure. One exemplary method described allows proxy infrastructures to gather, classify, and store metadata of exit nodes in its internal database. In another aspect, systems and methods described herein enable proxy infrastructures to select metadata of exit nodes from its internal database and forward requests from a user device to respective proxy servers or proxy supernodes to which the selected exit nodes are connected.