BASH

Linux curl Example

cURL is an incredibly flexible and powerful tool data transferring tool, that supports a wide variety of protocols, being the HTTP (and HTTPS) the most used. cURL library is available for many programming languages, but in this example we will see how to use it from the command line.

For this tutorial, Mint 17.3 has been used.

1. Installation

cURL can be installed easily via apt-get:

sudo apt-get update
sudo apt-get install curl

2. Basic usage

The simplest way to use cURL is just executing it, and specifying a site:

curl <site>

For example

curl google.com

And the terminal would show:

<HTML>
<HEAD>
    <meta http-equiv="content-type" content="text/html;charset=utf-8">
    <TITLE>301 Moved</TITLE>
</HEAD>
<BODY>
    <H1>301 Moved</H1>
    The document has moved
    <A HREF="http://www.google.es/">here</A>.
</BODY>
</HTML>

Which is not what we were expecting. This is because, by default, cURL doesn’t follow 3XX redirections. To make cURL follow redirections, we have to pass the -L option:

curl -L google.com

To save the HTTP response returned by cURL, we can always make a shell redirect with >, or use the -o option:

curl -o google.html -L google.com

When used, data about the receiving response is shown:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   219  100   219    0     0   2259      0 --:--:-- --:--:-- --:--:--  2281
  0     0    0 11675    0     0  50630      0 --:--:-- --:--:-- --:--:-- 10.5M

To see more information, including the HTTP headers, we have available the -v (--verbose) option:

curl -v google.com

Which will show something similar to the following before the HTTP response:

* Rebuilt URL to: google.com/
* Hostname was NOT found in DNS cache
*   Trying 130.206.193.53...
* Connected to google.com (130.206.193.53) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: google.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=UTF-8
< Location: http://www.google.es/?gfe_rd=cr&ei=OulzV836N9Op8weYzoxY
< Content-Length: 256
< Date: Wed, 29 Jun 2016 15:28:58 GMT
<
# HTML

3. Requests

3.1. GET

Making GET request with cURL is so simple. Actually, is as same as making it in the browser. For example:

curl -o systemcodegeeks.html https://www.systemcodegeeks.com?s=linux

We just have to add the query string in field=value& format after ?.

3.2. POST

To make a POST request, the usage is slightly different. The syntax is the following:

curl -d "field1=value1&field2=value2&fieldN=valueN" <site>

So, for making a search as the one done above, but supposing that the form method is POST, would be:

curl -d "s=linux" systemcodegeeks.com -o systemcodegeeks.html

Of course, we can make a POST request for uploading files:

curl -F "field=@/path/to/file.txt" awebsite.com/uploadfile

In this case, we have to specify the -F option for the file. And, the leading @ before the path to the file is necessary.

For making a POST request that combines both files and normal data, the data has to bet set also with -F, and not with -d. This is because the forms for file uploads sets the data encoding to -multipart/form-data, and this is what -F option does; whereas -d option sets the encoding to application/x-www-form-urlencoded. And all the data in a form has to be encoded under the same encoding. So, we have to specify also the data with -F option:

curl -F "field=value" -F "file=@/path/to/file.txt" awebsite.com/uploadfile

3.3. Custom requests: PUT and DELETE

PUT and DELETE are considered custom-methods. These requests are mostly used in CRUDs (Create, Read, Update, Delete), in RESTful services. To specify these commands, we have to use -X option.

For example:

curl -X PUT -d "data=stuff" http://myserver.com/rows/1
curl -X DELETE http://myserver.com/rows/1

3.4. Adding extra headers

For adding extra headers to a request, we just have to use the -H option (--header), specifying the header, and the value:

curl --header "X-ExtraHeader: myheader" http://myserver.com

4. FTP

We can also manage FTP server with cURL:

curl -u username:password ftp://myftp.com
# Or
curl fpt://username:password@myftp.com

We must take into account that, unless SSL is being used, the data will travel in plain text. If SSL is available, ftps should be use, instead of ftp.

We can upload files pretty simply, just specifying the local file with -T option:

curl -T file.txt ftps://username:password@myftp.com/path

Several files can be uploaded at once:

curl -T {file1.txt, file2.txt} ftps://username:password@myftp.com/path

Downloading files from FTP server is the same as downloading a simple HTML:

curl ftps://username:password@myftp.com/path/file.txt -o file.txt

5. Other interesting options

Apart from the common uses of cURL, let’s see other features that may result useful in some occasions.

5.1. HTTP authentication

With cURL is also possible to make HTTP authentication (that login prompts shown by the browser, not the authentication forms). For that, we have to use the -u (--user) option and specify the credentials:

curl -u username:password https://mylogin.com

5.2. Ignoring SSL errors

You will probably ever have encountered a site for which the browser shows a SSL certificate error, that it can be caused because it’s expired, because the certificate authority is not trusted by the browser, or any other reason. If we want to ignore the warning and access the site, we have to give explicit consent to the browser. By default, cURL does not accept the connections with SSL errors, but we can make it ignore them with -k (--insecure) option:

curl -k https://insecure-site.com

5.3. Cookies

Setting custom cookies with cURL is as easy as specifying them in name=value format, with -b (--cookie) option:

curl -b "mycookie=value" systemcodegeeks.com

We can even specify the cookies from a file!

curl -b cookies.txt systemcodegeeks.com

And, of course, we can save the cookies to a file, with -c (--cookie-jar) option:

curl -c cookies.txt systemcodegeeks.com

5.4. Making requests through proxy

Yes, cURL also allows to make requests through proxies. We just have to use the -x option, specifying the proxy:

curl -x https://aproxy.com systemcodegeeks.com

If the proxy server requires authentication, we have to specify as same as with HTTP authentication, but with --proxy-user option:

curl -x https://aproxy.com --proxy-user username:password systemcodegeeks.com

5.5. Download depending on the modification time

This is nice. We can make cURL download a file only if it has suffered modifications after the specified time, with -z (--time-cond) option:

curl -z 25 -Jun-16 systemcodegeeks.com

This feature can be specially useful for automating the task of having up-to-date documentation or software, for example.

6. Summary

This example has shown how to use cURL, from its simplest uses, which are downloading pages or making simple requests, to more advanced topics (which are actually easy with cURL), such us FTP connection, setting and saving cookies, or using proxies.

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button