How to Activate a Proxy Package

In this article, we’ll show you how to activate your proxy package and start using it.

Step 1: Add the Test Package to Your Cart

If you've already paid or received a link to a free proxy package, you're ready to proceed. Otherwise, you can find the test proxy package link on this page: https://proxycompass.com/free-trial/.

Click the link to add the test proxy package to your cart, then click the “Checkout” button.

Step 2: Register on the ProxyCompass Service

Use your Google account or enter your valid email address. Click the “Register” button to complete the registration process.

Your registration is complete. The password for your new account has been sent to the email address you provided.

Step 3: Check Your Email

In the email you received from us, you will find an automatically generated password. You can change this password later.

Step 4: Log in to the Dashboard

Go to the link:
https://proxycompass.com/account/index.php?rp=/login

Enter your previously provided email address and the received password. Click the “Login” button.

Click the “Cart” button to proceed with the activation of the test proxy package.

Step 5 (Important): Enter Your Own IP Address in the “Bind IP” Field

Be sure to enter the IP address of the device where you'll use the proxies. The proxies will only be accessible from the device with the IP you specified in "Bind IP".

For example:

- For a home computer, enter your home computer’s IP.
- For a remote server or VPS, enter the server's IP.

Visit https://2ip.io/ to find your current device’s IP.

In most cases, the IP will auto-fill. Click “Set” to apply settings.

Note: Activation may take 5-10 minutes. Just be patient.

Step 6: Choose the Suitable Proxy Retrieval Option

- Download an HTTP or SOCKS proxy list in the IP:PORT format.
- Download an HTTP or SOCKS proxy list in the IP:PORT:Username:Password format.
- Get a random SOCKS proxy via the link.
- Get a random HTTP proxy via the link.
- Generate and download a proxy list in a custom format.

How to Download a Proxy List (Without Username & Password)

If your program uses proxies without authentication, download the text file in the IP:PORT format.

In this case, use port 8085 for HTTP or 1085 for SOCKS.

To download the list in this format, click the “TXT” link as shown in the screenshot.

You will receive proxy lists in the following format:

How to Download a Proxy List with Username and Password

Your proxy Login and Password can be found at the top of the page. It will be displayed as follows.

In our example, our Login is USK9MFARF, and the Password is pq94v42C.

If your program requires the proxy list in the format IP:Port:Username:Password, follow these steps:

Scroll to the bottom of the page to the "Proxy list designer" section.

Enter the following code in the "Template" field:

{ip}:{port}:USK9MFARF:pq94v42C

Replace this sample login and password with your actual credentials

where:

- {ip} - each IP address in the proxy list
- {port} - required port
- USK9MFARF - sample login
- pq94v42C - sample password

Select the desired option in "Proxy type" and click "Create" to generate the proxy list.

As a result, you will get a generated proxy list in the required format: IP:Port:Username:Password.

How to Get a Random Proxy from the List Without Downloading

If your program doesn’t support proxy lists and requires a direct link to a specific proxy server, you can do the following:

On the service page, find the section titled:

API for remote access to single random available proxy

You will see the following options there.

Click the “Get” link next to the option suitable for your program to obtain a random proxy from the list.

- Random HTTP Proxy without login/password:
https://proxycompass.com/api/getproxy/?r=1&format=txt&type=http_ip&login=YOURLOGIN&password=YOURPASSWORD
- Random SOCKS Proxy without login/password:
https://proxycompass.com/api/getproxy/?r=1&format=txt&type=socks_ip&login=YOURLOGIN&password=YOURPASSWORD
- Random HTTP Proxy with login/password:
https://proxycompass.com/api/getproxy/?r=1&format=txt&type=http_auth&login=YOURLOGIN&password=YOURPASSWORD
- Random SOCKS Proxy with login/password:
https://proxycompass.com/api/getproxy/?r=1&format=txt&type=socks_auth&login=YOURLOGIN&password=YOURPASSWORD

Replace YOURLOGIN and YOURPASSWORD with your actual login credentials.

https://proxycompass.com/knowledge-base/how-to-activate-a-proxy-package/

How to prevent and bypass CAPTCHA while web scraping?

Failing to bypass captcha during web scraping can lead to issues. Some of them can be delays, incomplete data extraction, and wasted resources. Since these website obstacles are frustrating and costly, you need to know how to avoid them.

In this article, we explore CAPTCHAs, how they function, and what activates them. Keep reading to discover ways to bypass these challenges and keep your projects running smoothly.

What are CAPTCHAS?

CAPTCHAs are a security measure implemented on web pages to distinguish human users from automated bots. They present a challenge that is easy for humans to solve. But, for computers and bots, it is difficult to interpret. 

The main purpose of CAPTCHAs is to prevent automated spam and bot attacks. They work by protecting websites from unauthorized access. 

So, they are a challenge for web scraping.

Types of CAPTCHA

There are several common types of CAPTCHA:

- Text-based CAPTCHA: displays a distorted text that users must decipher and write.
- Image-based CAPTCHA: requires users to identify or select specific images. For example, selecting all images containing a certain object.
- Audio CAPTCHA: Plays an audio clip that users must transcribe. It's an alternative for visually impaired users.
- Math Problems: Presents simple math operations for users to solve.
- reCAPTCHA: Google's advanced CAPTCHA system. It's a simple box users have to check. Uses behavioral analysis to determine if the user is human.

How CAPTCHA works

CAPTCHA systems work following a process. When the user  enters the site, it will pop up a challenge. The user has to complete the challenge and then, the system verifies the response.

If correct, the user can continue. But, if there's a mistake, a new challenge will pop up. 

But, traditional CAPTCHAs and advanced systems like reCAPTCHA work differently. 

For traditional CAPTCHAs, there are distorted forms of letters and numbers. These characters may be stretched, broken, or presented in any other way that makes it challenging to solve. It uses random strings of letters and numbers to prevent repetition.

While reCAPTCHA works analyzing user behavior patterns. These include cursor movements and device history. A human would behave somewhat randomly, not using straight lines. Unlike a bot. 

Also, this system assigns risk scores to determine the likelihood of the user being human or bot. 

How CAPTCHAs negatively affect web scrapers

CAPTCHAs are designed to prevent automated bots from accessing and interacting with websites. Thus, they significantly hinder web scraping efforts in several ways.

For humans, it may be easy to solve (kind of), but for machines it's complicated to understand. So, when a web scraper encounters a CAPTCHA, it cannot solve it automatically. This can bring the scraper process to a halt. 

And even if the scraper manages to access the site, it can encounter ongoing CAPTCHA tests. These could potentially block further activity.

Moreover, the increasing complexity of CAPTCHAs over time is also an issue. This is because you would need more sophisticated and costly solutions for scraping to bypass them. 

So, solving CAPTCHAs requires human intervention or specialized services like a web scraping captcha solver. All which can consume significant resources, both time and money. Ending up in a scraping process that is less efficient and more expensive. 

What triggers a CAPTCHA?

- Unusual patterns or a sudden spike in traffic from a particular IP address (or a range of IP addresses).
- Many failed login attempts.
- Suspicious behavior during the registration process.
- Rapid form submissions.
- Repetitive clicking.
- Submitting large amounts of data in a short period.
- No browsing history before the attempt.
- Anomalies in the loading sequence or behavior of website resources like scripts, CSS files, or other assets.

How to bypass CAPTCHA while web scraping

Since solving CAPTCHAs is not an easy task, then it's best to learn how to avoid captcha when scraping​. You can raise your trust score and bypass CAPTCHAS by implementing any of the following strategies:

1. Rotate IPs and User Agents

When a website detects many requests from the same IP address, it will often trigger a CAPTCHA. To avoid this, you use a rotating proxy server to change your IP address for each request. 

Similarly, you can also rotate User Agents to prevent triggering CAPTCHAs. The User Agent is a string used to identify the browser that is requesting the content. 

2. Use CAPTCHA solver services

You can integrate CAPTCHA-solving services like 2captcha.com, anti-captcha.com, and capsolver.com to your workflow. 

These services send the CAPTCHA challenge to human workers or sophisticated algorithms to solve them. 

Yet, this approach can be less efficient and costlier.

3. Cooldown logic and mimic human behavior

Cooldown logic refers to delays between requests to stimulate human browsing patterns. This will prevent the site  from flagging you as a bot because of rapid requests.

There are some tools that can help you mimic human behavior. With headless browsers like Puppeteer or Selenium, you can automate actions like natural mouse movements, clicks, and scrolling. 

This can help you bypass CAPTCHAs that rely on detecting unnatural patterns of interaction.

4. Use advanced web scraping APIs

Using a web scraping API like Scraping Robot can help you achieve a seamless experience. This tool has a specialized focus on handling CAPTCHA challenges and proxy rotation. Thus, it helps you scrape without worrying about triggering CAPTCHAs.

5. Avoid hidden traps and honeypot elements

A honeypot is a trap websites use to detect bots. They have hidden elements or fields that are invisible by users. For instance, it could be hidden links or buttons.

If the web scraper interacts with it, then the site can trigger CAPTCHAs and flag the IP as suspicious.

How can you avoid these traps? You need to interact with the site as a real browser would.

6. Use Machine Learning and OCR

Training models with frameworks like TensorFlow or PyTorch can help in recognizing and solving CAPTCHA patterns automatically.

OCR can sometimes work solving text-based CAPTCHAS. Still, it's not always effective due to CAPTCHA complexity.

7. Maintain a low profile

Ensure that your scraper's behavior closely resembles that of a real browser. This includes managing cookies, using appropriate TLS fingerprinting, and running JavaScript.

Moreover, limit the number of requests per IP, and don't scrap too quickly. Make organic timeouts to prevent triggering CAPTCHA challenges.

Here's how we can help

Learning how to bypass captcha while scraping​ isn't that simple. Fortunately, Proxy Compass offers robust solutions that could help you avoid CAPTCHAS during your scraping process.

Having a wide variety, and rotating your IP addresses will make it difficult for websites to detect and block you. It also reduces the likelihood of triggering CAPTCHA challenges.

Get a free trial today and enjoy a better web scraping experience. Stop relying on a single IP.

https://proxycompass.com/how-to-prevent-and-bypass-captcha-while-web-scraping/