How to Use cURL With Proxy?

Using environment variables

Another way to use proxy with curl is to set the environment variables http_proxy and https_proxy. 

Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc_curlrc file.

If you look at the first part of these variable names,  it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.

• http_proxy – the proxy will be used to access addresses that use http protocol

• https_proxy – the proxy will be used to access addresses that use https protocol

Simply set the variables http_proxy to http proxy address and https_proxy to set https proxy address. Open terminal and run these two commands.

export http_proxy="http://user:pwd@127.0.0.1:1234"
export https_proxy="http://user:pwd@127.0.0.1:1234"

After running these two commands, run curl normally.

curl "http://httpbin.org/ip"

If you see SSL Certificate errors, add -k to ignore these errors.

Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:

unset http_proxy
unset https_proxy

See the next section to set default proxy only for curl and not system wide.

Configure cURL to always use proxy

If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.

For Linux and MacOS, open terminal and navigate to your home directory. If there is already a .curlrc file, open it. If there is none, create a new file. Here are the set of commands that can be run:

cd ~
nano .curlrc

In this file, add this line:

proxy="http://user:pwd@127.0.0.1:1234"

Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from .curlrc file.

curl "http://httpbin.org/ip"

On Windows, the file is named _curlrc. This file can be placed in the %APPDATA% directory.

To find the exact path of %APPDATA%, open command prompt and run the following command:

echo %APPDATA%

This directory will be something like C:\Users\<your_user>\AppData\Roaming. Now go to this directory, and create a new file _curlrc, and set the proxy by adding this line:

proxy="http://user:pwd@127.0.0.1:1234"

This works exactly the same way in Linux, MacOS, and Windows.

Ignore or override proxy for one request

If the proxy is set globally, or by modifying the .curlrc file, this can still be overridden to set another proxy or even bypass it.

To override proxy for one request, set the new proxy using -x or –proxy switch as usual:

curl --proxy "http://user:pwd@1.0.0.1:8090" "http://httpbin.org/ip"

If you want to bypass proxy altogether for a request, you can use the cURL noproxy command by passing –noproxy followed by “*”. This instructs cURL to not use proxy for all URLs.

curl --noproxy "*" "http://httpbin.org/ip"

If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.

Bonus tip – turning proxies off and on quickly

This tip is dedicated only for advanced users. If you do not know what a .bashrc file is, you may skip this section.

You can create an alias in your .bashrc file to set proxies and unset proxies. For example, open .bashrc file using any editor and add these lines:

alias proxyon="export http_proxy=' http://user:pwd@127.0.0.1:1234';export https_proxy=' http://user:pwd@127.0.0.1:1234'"
alias proxyoff="unset http_proxy;unset https_proxy"

After adding these lines, save the .bashrc and update the shell to read this .bashrc. To do this, run this this command in the terminal:

. ~/.bashrc

Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:

proxyon
curl "http://httpbin.org/ip"
curl "http://google.com"
proxyoff 

cURL socks proxy

If the proxy server is using socks protocol, the syntax remains the same:

curl -x "socks5://user:pwd@127.0.0.1:1234" "http://httpbin.org/ip"

Similarly, socks4://, socks4a://, socks5:// or socks5h:// can be used depending on the socks version.

Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.

curl --socks5 "127.0.0.1:1234" "http://httpbin.org/ip" --proxy-user user:pwd

Again, –socks4, –socks4a or –socks5 can be used, depending on the version.

Summary

cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper. 

Click here and check out a repository on GitHub to find the complete code used in this article. You can also learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.