What is cURL and What Does It Mean?

History of cURL

Back in the dark ages of the 90s when everyone still used command line tools, Daniel Sterberg wanted to develop a simple IRC script that would convert currencies for chat room members. In 1997 there weren’t a lot of options to build a foundation for internet protocol data delivery, therefore Httpget, some few hundred lines of code for HTTP-based transfers, became the genesis for cURL. The cURL HTTP was first called, in honor of its foundation, HTTPGET 1.0.

Months later, FTP support had been developed and the name had to be dropped. Now it was being called urlget 2.0. After a few updates, on March 30 1998, the name was changed once again to the now well-known cURL 3.0.

cURL has its older cousin – wget. We won’t get too much into the details, but the main difference between wget and cURL are the download capabilities of each as the former can, for example, recover from broken transfers and continue to download files.

What does cURL do?

cURL is intended to transfer data through internet protocols. Everything else is outside its scope. It doesn’t even handle the data transferred, only performs the process itself. 

An example of what cURL might be used for is debugging. Using “curl -v https://oxylabs.io” shows the verbose output of one connection request, including details such as a user agent, handshake data, ports, etc.

There are far too many possible cURL command options to list and explain. Luckily, “curl –help” is an option that lists out all the cURL command option possibilities with short explanatory comments. Although without some background on how to use cURL the list won’t be all that helpful.

How to use cURL?

Nearly anyone with a relatively recent operating system can use cURL as it’s shipped as a default in Windows, MacOS, and most Linux distributions. For older systems such as any Windows operating system before 10, cURL might need to be downloaded and installed.

To use cURL, simply open up the terminal and type in “curl”. If everything is as it should be, the output should offer to use “curl –help”. “Help” will, as previously mentioned, list out all the command possibilities. A cURL command can be combined by adding the listed flags and typing in an URL. Flags can be either short (e.g. -o, -L, etc) or long (e.g. –verbose). These flags are differentiated by the use of single or double dashes.

Using cURL

cURL is a potent tool for data transfers through internet protocols. Detailing all the possible options on what cURL is used for would be nearly impossible. Although there are a few common use cases which we will expand upon. As cURL has initially been developed for HTTP, we can send all the usual requests (POST, GET, PUT, etc.).

Sending GET requests

To send a GET request using cURL, we will have to use the `curl` command from the command line or terminal, followed by the URL of the website. For example, to send a GET request to the URL "https://ip.oxylabs.io/location", we will have to use the following command:

curl https://ip.oxylabs.io/location

Once we run this command, it will send a GET request to the website: ip.oxylabs.io/location. And print the Response in the terminal. If it runs successfully we will see our current IP address as output.

Sending a POST request

In order to send a POST request to a URL, the -d (or –data) flag is used. Most websites will deny such requests from unauthorized users, therefore we will use a fake API for testing purposes.

Curl-d “name=something&value=somethingelse”https://jsonplaceholder.typicode.com/posts/

Sending such a request should return:

{
  "name": "something",
  "value": "somethingelse",
  "id": 101
}

There are a few things at play here:

• curl begins our command

• -d is the “data” flag for POST requests

• Quote marks (“”) begin our content statement. Note that some operating systems will only accept single quotes, others will accept double quotes.

• Finally, the destination. URL syntax should always be exact as cURL doesn’t automatically follow redirects

We can also send POST requests in the JSON format, although additional options will have to be provided in order to tell the server that we are sending a JSON. cURL does little interpretation on behalf of the user and would send the default Content-Type header of application/text, so we have to add the header Content-Type: application/json ourselves.

curl -H "Content-Type: application/json" --data "{\"data\":\"some data\"}"  https://jsonplaceholder.typicode.com/posts/

Saving URL content to a file

Curl also allows to download and upload files in local storage. The process is pretty simple. We will simply need to pass a couple of extra parameters. Let’s say we want to store our IP Address in a text file from the website https://ip.oxylabs.io/location.

We will only need to run the below command:

curl -o response.txt https://ip.oxylabs.io/location

Notice we are passing a new argument -o, which is the short form of --output. We are also mentioning a filename response.txt. Once we execute this command, curl will make a new request to the website and collect our IP address and then download it inside the current folder creating a new response.txt file. If the file already exists, it will overwrite it with new content.

Following redirects

cURL doesn’t automatically follow redirects. We should add an additional flag if we expect it to do so. Let’s take a look at an example:

curl https://google.com

Our browsers handle redirects by themselves, so we might not even notice an issue with such a request. Yet, if we send cURL in to do the work, we will receive a notice that the document has been moved as it tries to connect. To get cURL to follow redirects, we have to add a special flag “-L” (arguments are case sensitive).

curl -L https://google.com

We should now have received the regular answer from Google as cURL follows the redirect from https://google.com to https://www.google.com/.

Connecting through a proxy

cURL can be used to connect to any destination through a proxy. As with any other cURL statement, the URL, syntax and everything else stay the same except for the added flag and its attributes.

curl --proxy proxyaddress:port https://jsonplaceholder.typicode.com/

Entering the proxy and port after “–proxy” will route the connection through the inputted address. Proxies will often require credential details that can be sent through the -U flag.

curl --proxy proxy:port -U “username:password” https://jsonplaceholder.typicode.com/

Some websites will require an authentication method by themselves before they accept any connection request. A similar flag is used for server authentication: “-u”.

curl -u username:password https://jsonplaceholder.typicode.com/

You can read more on how to cURL with proxy in our blog post. Also, if you're searching for a reliable and fast proxy solution, check out our Residential Proxies.

Why use cURL?

cURL doesn’t only support multiple protocols, including HTTP, HTTPS, and DICT ( dictionary network protocol) but it also runs on essentially every single platform. As a result, cURL becomes the optimal choice when testing communication from almost any device. The only requirements are that there is a command line and network connectivity.

Yet, cURL benefits go beyond communication testing as you’ll see based on the brief list below.

• cURL can have its rate limited

• It’s a unique portable choice, as it’s compatible with most operating systems and connected devices

• Allows for easy testing of endpoints in order to see if they’re working

• cURL has great error logging

• It is also seen as detailed, showing exact information about what is sent and received, which makes it helpful in debugging.

Conclusion

cURL is a very powerful tool for internet protocol transfers. Mastering its use is definitely a challenge but it can become an irreplaceable tool in any developer’s toolkit. Fully expanding on all the possible cURL use cases is, frankly, impossible.

Want to find out more on interesting topics such as how do CAPTCHAs work? Peruse our blog as we’ve got plenty of posts to offer!