Blog/
Setting up a proxy in Selenium in Python

Setting up a proxy in Selenium in Python

December 11, 2023
47 views

Selenium WebDriver is a browser automation tool designed for programmatically interacting with interactive websites and applications. It simulates human actions, including clicking, text entry, and page navigation. As an open-source tool, Selenium's code can be modified and enhanced to suit specific needs. For optimized results, integrating a proxy with Selenium is recommended.

Setting up a proxy for use in Selenium

Properly setting up proxy configurations in Selenium enables the routing of network traffic through proxy servers. This is particularly beneficial for automating web activities that require specific IP addresses or are linked to certain geographic regions.

How to set up Selenium in Python

Ensure you have a Python IDE installed. Integrate the Selenium library using pip by running pip install selenium in your command line or terminal.
```
pip install selenium
```
Download and install the WebDriver for your browser (e.g., Chrome, Firefox, Edge). Include the driver's path in your code. An example for Chrome WebDriver setup is provided.
```
from selenium import webdriver
driver = webdriver.Chrome(executable_path="/path/to/chromedriver")
```
Purchase a proxy package from a reliable provider. Input the proxy IP address, port, and authentication parameters into Selenium using the provided code lines.
```
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
```
In the ChromeOptions object, specify the proxy server using “proxy_server = 'ip_proxy:port'”, where 'ip_proxy' is the server's IP address, and 'port' is its port number.
```
# Replace 'ip_proxy' with the actual IP address of your proxy server.
# Replace 'port' with the port number of your proxy server.
proxy_server = 'ip_proxy:port'
```
Insert a new line of code into your command line or terminal to inform the browser of the new proxy connection, using the ‘proxy_server’ variable.
```
chrome_options.add_argument(f'--proxy-server={proxy_server}')
```
Replace “Path_to_driver” with the actual path of your WebDriver, typically found in the Chrome root folder at “C:\Program Files\Google\Chrome”.
```
driver = webdriver.Chrome(executable_path=driver_path, options=chrome_options)
```
If your proxy server requires authentication, add these parameters to Chrome or your chosen browser using the provided code.
```
chrome_options.add_argument('--proxy-auth={}:{}'.format(proxy_username, proxy_password))
```
Create a parameter in your setup to bypass local addresses, reducing proxy traffic and load.
```
chrome_options.add_argument('--proxy-bypass-list=*')
```
Complete the driver setup with the new configurations and pass debugging data from Chrome to the driver.
```
driver = webdriver.Chrome(options=chrome_options)
```
Test your setup on a target website for scraping. Initiate this with the “driver.get” command, passing the full URL of the site.
```
driver.get("https://www.example.com")
```

Remember, configuring proxies in Selenium not only maintains online privacy but also secures your data during scraping activities.

TOP 10 proxy providers for online testing

Proxy-Seller

The service provides proxy servers with support for various protocols (HTTP, HTTPS, SOCKS) and different geographical locations. There is a choice of static and dynamic proxies.
This provider offers a variety of proxy servers with high speeds and a variety of options, which can be useful when testing different scenarios and network conditions.
Provides personal and elite IPv4 and IPv6, mobile, ISP, and residential proxies. There is no demo version on the site.

BrightData

BrightData specializes in a peer-to-peer (P2P) proxy network, offering access to millions of authentic IP addresses globally. Users have the flexibility to choose specific geographical locations.
The network grants access to a diverse array of IP addresses, ideal for testing applications and services that are sensitive to geographic locations.
BrightData offers the purchase of a Selenium SOCKS5 proxy, optimized for IPv4 networks. This ensures a stable connection and high-speed data transfer, even during extensive scraping operations.

Geonix

Geonix provides high-speed proxies supporting HTTP, HTTPS, and SOCKS5 protocols.
The site features high-performance proxies, essential for testing the performance of web services and applications. It offers both free proxies and personal IPs available for rent over flexible durations.
Geonix also offers a range of additional webmaster tools, designed to enhance capabilities and boost productivity.

Stormproxies

Stormproxies provides a range of proxies supporting various protocols, including HTTP and SOCKS, available as both private and public connections.
They offer a diverse selection of proxy servers with different tariffs and packages, enabling customization to fit specific testing scenarios.
The bandwidth of Stormproxies' channels reaches 1 Gbit/sec, ensuring rapid request processing during scraping. For enhanced efficiency, customers can purchase a package of 5 proxy servers, with the minimum cost varying based on technical specifications.

Youproxy

Youproxy offers dedicated proxy servers compatible with HTTP, HTTPS, and SOCKS 4 and 5 protocols, catering to a variety of needs including testing.
The service includes personal proxies in both IPV4 and IPV6 networks, as well as mobile proxy connections, ensuring stable and reliable connectivity crucial for testing and data collection.
While Youproxy doesn't offer residential proxies, it guarantees secure connections, low ping, and high-speed data transmission and reception. An additional advantage is the consistent availability of servers, with proxy rental options available for up to one year.

Oxylabs

Oxylabs offers proxies supporting standard HTTP and SOCKS protocols, providing access to millions of authentic IP addresses. Their service includes an API for automation, with options for residential, mobile, private, and rotating ISP proxies.
Their tools facilitate deep and secure website analysis using Selenium or other scrapers, including the ability to use SSH connections.
Oxylabs' built-in Scraper API enhances the efficiency of technical research. Additionally, they offer Shared Proxy services suitable for collective use, like within an analytics company.

Webshare

You can purchase proxies with support for both HTTP and SOCKS5 protocols, including Anonymous Proxy, static, and residential addresses. These options provide everything you need to safeguard your identity and evade IP blocking.
There's a demo available with a brief testing period, offering clients 10 proxies to assess the service's functionality and explore its features. While API integration may be intricate, Python experts should find it manageable.
The provider guarantees speeds of up to 1 Gbit/sec, enabling simultaneous scraping of multiple resources.

Proxy-IPv4

Here, you can find proxies that support IPv4 and offer high connection speeds across HTTP, HTTPS, and SOCKS5 protocols.
Users have the option to choose between static and dynamic IPs, connecting to servers through various channels.
The presence of authentication guarantees connection security, and the provider's proxies are known for their impressive throughput. If you intend to test websites using Selenium, it's advisable to purchase an IP package without delay.

Soax

Soax offers proxies that support both HTTP and SOCKS5 protocols, including residential, mobile, and ISP proxies. They also offer separate rates for testers.
Soax proxies are known for their high performance and the option to select specific locations, which can be crucial for scraping geographically dependent websites.
Webmasters are recommended to consider packages with dynamic rotation. The total proxy pool comprises approximately 9 million proxies, and a single client can utilize up to 300 ports simultaneously.

Decodo

Decodo provides proxies with support for HTTP, HTTPS, and SOCKS5 protocols, and offers special plans tailored for testing purposes.
Buyers have access to residential, static, and mobile proxies, as well as dedicated proxies designed for data centers with high bandwidth.
The provider offers API interfaces for purchase on their website, which are ideal for collecting large amounts of data and facilitating analysis—a valuable asset for webmasters.
Buyers can independently customize the technical characteristics of a specific proxy package based on their budget and a suitable tariff plan.

When working with Selenium, selecting a proxy provider with extensive geolocation coverage is crucial for successful testing. We recommend considering Proxy-Seller (covering 220+ countries), Brightdata (195+ countries), Oxylabs (140+ countries), and Geonix (210+ countries). By using the services of these providers, you can eliminate issues related to selecting the correct location.

Furthermore, specific testing scenarios may require particular networks and subnets. In such cases, Proxy-Seller, Geonix, and Youproxy are your go-to options, as they are the only providers in our review that offer the ability to select the desired network or subnet when placing an order.

If you need to simulate a real user during testing, residential proxies are essential. We recommend providers like Proxy-Seller, Brightdata, Geonix, Oxylabs, and Webshare for purchasing this type of proxy. They offer attractive packages and trial versions, allowing you to choose the package that suits your needs, regardless of traffic consumption.

A well-chosen proxy server is a valuable addition to Selenium, and our review can assist you in selecting the right proxy provider to unlock its full potential.