As all bloggers eventually find out, two of the best reasons to write blog posts are either to document something for yourself for later or to force yourself to learn something well enough to explain it to others. That’s the impetus of this series that I plan on doing from time to time. I want to get more familiar with some of these core tools and also have a reference / resource available that is in “Pete Think” so I can quickly find what I need to use these tools if I forget.
The first tool I want to tackle is curl. In the world of command-line tools, curl shines as a flexible utility for transferring data over various network protocols. Whether you’re working on the development side, the network admin side, or the security side, learning how to use curl effectively can be incredibly beneficial. This blog post will guide you through the very basics of curl, cover some common use cases, and explain when curl might be a better choice than wget.
What is curl
curl (Client for URL) is a command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, SCP, TELNET, LDAP, IMAP, SMB, and many more. curl is known for its flexibility and is widely used for interacting with APIs, downloading files, and testing network connections.
Installing curl
Before diving into curl commands, you need to ensure it is installed on your system. Lots of operating systems come with it. In fact, even Windows has shipped with curl in Windows 10 and 11.
For Linux:
sudo apt-get install curl # Debian/Ubuntu sudo yum install curl # CentOS/RHEL
For macOS:
brew install curl
For Windows
You can download the installer from the official curl website if it isn’t already on your system. To check, just type curl –help at the command prompt and see if it understand the command. If you get something back like this, you’re all set.
C:\Users\peteonsoftware>curl --help Usage: curl [options...] <url> -d, --data <data> HTTP POST data -f, --fail Fail fast with no output on HTTP errors -h, --help <category> Get help for commands -i, --include Include response headers in output -o, --output <file> Write to file instead of stdout -O, --remote-name Write output to file named as remote file -s, --silent Silent mode -T, --upload-file <file> Transfer local FILE to destination -u, --user <user:password> Server user and password -A, --user-agent <name> Send User-Agent <name> to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit This is not the full help, this menu is stripped into categories. Use "--help category" to get an overview of all categories. For all options use the manual or "--help all".
The Most Simple Example
The simplest way to use curl is to fetch the contents of a URL. Here is a basic example that will will print the HTML content of the specified URL to the terminal:
c:\
λ curl https://hosthtml.live
<!doctype html>
<html data-adblockkey="MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANDrp2lz7AOmADaN8tA50LsWcjLFyQFcb/P2Txc58oYOeILb3vBw7J6f4pamkAQVSQuqYsKx3YzdUHCvbVZvFUsCAwEAAQ==_M6heeSY2n3p1IRsqfcIljkNrgqYXDBDFSWeybupIpyihjfHMZhFu8kniDL51hLxUnYHjgmcv2EYUtXfRDcRWZQ==" lang="en" style="background: #2B2B2B;">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12P4//8/AAX+Av7czFnnAAAAAElFTkSuQmCC">
<link rel="preconnect" href="https://www.google.com" crossorigin>
</head>
<body>
<div id="target" style="opacity: 0"></div>
<script>window.park = "eyJ1dWlkIjoiZDFhODUxY2ItOTUyZi00NGUyLTg4ZWMtMmU3ZGNhZmE1OTk0IiwicGFnZV90aW1lIjoxNzIwNzMyMzQxLCJwYWdlX3VybCI6Imh0dHBzOi8vaG9zdGh0bWwubGl2ZS8iLCJwYWdlX21ldGhvZCI6IkdFVCIsInBhZ2VfcmVxdWVzdCI6e30sInBhZ2VfaGVhZGVycyI6e30sImhvc3QiOiJob3N0aHRtbC5saXZlIiwiaXAiOiI3Mi4xMDQuMTY5LjE1NCJ9Cg==";</script>
<script src="/bwjblpHBR.js"></script>
</body>
</html>
Some Useful Examples to Actually Do Stuff
Downloading Files
# To download a file and save it with a specific name: curl -o curlypigtail.jpg https://peteonsoftware.com/images/202407/curlytail.jpg # If you want to save the file with the same name as in the URL: curl -O https://peteonsoftware.com/images/202407/curlytail.jpg
Sending HTTP Requests
# GET requests are used to retrieve data from a server. The basic example is already shown above.
# To include headers in the output, use the -i option:
curl -i https://hosthtml.live
# POST requests are used to send data to a server.
# This is particularly useful when interacting with APIs.
curl -X POST -d "param1=value1¶m2=value2" http://somereallygreat.net/api
# Many APIs now accept JSON. This is how you'd send that
curl -X POST -H "Content-Type: application/json" -d '{"key1":"value1", "key2":"value2"}' http://somereallygreat.net/api
# Without explaining it, we included a header above (-H). That added a Content-Type header.
# To add an Auth header, you might do something like this
curl -H "Authorization: Bearer token" http://asitethatneedsbearertokens.com
Cookies
# To save cookies from a response curl -c cookies.txt https://www.google.com # To send cookies with a request curl -b cookies.txt https://www.google.com
When to Use curl Over wget
While both curl and wget are used to transfer data over the internet, they have different strengths. Daniel Stenberg is the creator of curl (and also contributes to wget) and he’s published a more lengthy comparison here. I defer to the expert, but here are some of my big takeaways.
curl Advantages
- Flexibility: curl supports a wider range of protocols (like SCP, SFTP) and provides more options for customizing requests.
- Availability: curl comes preinstalled on macOS and Windows 10/11. wget doesn’t.
- Pipelining: curl can be used to chain multiple requests together, making it powerful for scripting complex interactions.
- Reuse: curl is a library (libcurl), while wget is just a command line tool.
wget Advantages
- Recursive Downloads: wget can download entire websites recursively, making it ideal for mirroring sites.
- Simplicity: For simple downloading tasks, wget might be more straightforward and easier to use.
curl is a versatile tool that – once mastered – can simplify many network-related tasks. From downloading files to interacting with APIs, curl provides the flexibility and functionality needed for a wide range of applications. While wget has its strengths, particularly for simple downloads and recursive website copying, curl shines in its versatility and extensive options for customizing requests.