Tag: General Tips

General Tips

Core Tools to Know: ffuf

ffuf run logo version, from the GitHub Repoffuf (“Fuzz Faster U Fool”) is a powerful, open-source tool designed for web application enumeration and fuzzing. Whether you’re performing vhost, directory, page, or parameter enumeration, ffuf can help you identify and exploit vulnerabilities effectively. In this overview tutorial, we’ll cover how to install ffuf and use it for various enumeration tasks. Right off the bat, I want to stress that you should never run tools like ffuf against any machine, network, system, etc that you either don’t own and control or have explicit permission to run these sorts of programs on. If you venture outside of that boundary, you get in pretty murky legal territory and I’m officially suggesting that you don’t do that.

Installing ffuf

Before diving into its usage, let’s install ffuf on your system. ffuf works on Linux, macOS, and Windows. According to the ffuf room on TryHackMe, ffuf is already included in the following Linux distributions: BlackArch, Pentoo, Kali, and Parrot. If you’re on Windows, macOS, or a Linux version without ffuf preinstalled, the easiest thing to do is download one of the prebuilt releases from ffuf’s GitHub. If you want to install ffuf from source, you’ll need to install Go 1.16 (or higher) and follow the instructions on ffuf’s readme on their GitHub.

Once you’re sure you’ve got the program, let’s dig in. We can make sure that ffuf is in our path by running the following command and getting the help menu in response. If it fails and you’re sure that you’ve got it installed, add its location to your path, or use the fully qualified path when calling it from the command line (like /opt/ffuf/ffuf or c:\Tools\ffuf.exe instead of just ffuf).

So How Does it Work?

Let’s take a look at the built-in help to understand what all it can do. Like many security tools, ffuf is command-line only, so pop open a terminal and issue the following command (the output is really long, feel free to skim it or just scroll past this block to not be overwhelmed).

$ ffuf -h
Fuzz Faster U Fool - v2.1.0-dev

HTTP OPTIONS:
  -H                  Header `"Name: Value"`, separated by colon. Multiple -H flags are accepted.
  -X                  HTTP method to use
  -b                  Cookie data `"NAME1=VALUE1; NAME2=VALUE2"` for copy as curl functionality.
  -cc                 Client cert for authentication. Client key needs to be defined as well for this to work
  -ck                 Client key for authentication. Client certificate needs to be defined as well for this to work
  -d                  POST data
  -http2              Use HTTP2 protocol (default: false)
  -ignore-body        Do not fetch the response content. (default: false)
  -r                  Follow redirects (default: false)
  -raw                Do not encode URI (default: false)
  -recursion          Scan recursively. Only FUZZ keyword is supported, and URL (-u) has to end in it. (default: false)
  -recursion-depth    Maximum recursion depth. (default: 0)
  -recursion-strategy Recursion strategy: "default" for a redirect based, and "greedy" to recurse on all matches (default: default)
  -replay-proxy       Replay matched requests using this proxy.
  -sni                Target TLS SNI, does not support FUZZ keyword
  -timeout            HTTP request timeout in seconds. (default: 10)
  -u                  Target URL
  -x                  Proxy URL (SOCKS5 or HTTP). For example: http://127.0.0.1:8080 or socks5://127.0.0.1:8080

GENERAL OPTIONS:
  -V                  Show version information. (default: false)
  -ac                 Automatically calibrate filtering options (default: false)
  -acc                Custom auto-calibration string. Can be used multiple times. Implies -ac
  -ach                Per host autocalibration (default: false)
  -ack                Autocalibration keyword (default: FUZZ)
  -acs                Custom auto-calibration strategies. Can be used multiple times. Implies -ac
  -c                  Colorize output. (default: false)
  -config             Load configuration from a file
  -json               JSON output, printing newline-delimited JSON records (default: false)
  -maxtime            Maximum running time in seconds for entire process. (default: 0)
  -maxtime-job        Maximum running time in seconds per job. (default: 0)
  -noninteractive     Disable the interactive console functionality (default: false)
  -p                  Seconds of `delay` between requests, or a range of random delay. For example "0.1" or "0.1-2.0"
  -rate               Rate of requests per second (default: 0)
  -s                  Do not print additional information (silent mode) (default: false)
  -sa                 Stop on all error cases. Implies -sf and -se. (default: false)
  -scraperfile        Custom scraper file path
  -scrapers           Active scraper groups (default: all)
  -se                 Stop on spurious errors (default: false)
  -search             Search for a FFUFHASH payload from ffuf history
  -sf                 Stop when > 95% of responses return 403 Forbidden (default: false)
  -t                  Number of concurrent threads. (default: 40)
  -v                  Verbose output, printing full URL and redirect location (if any) with the results. (default: false)

MATCHER OPTIONS:
  -mc                 Match HTTP status codes, or "all" for everything. (default: 200-299,301,302,307,401,403,405,500)
  -ml                 Match amount of lines in response
  -mmode              Matcher set operator. Either of: and, or (default: or)
  -mr                 Match regexp
  -ms                 Match HTTP response size
  -mt                 Match how many milliseconds to the first response byte, either greater or less than. EG: >100 or <100
  -mw                 Match amount of words in response

FILTER OPTIONS:
  -fc                 Filter HTTP status codes from response. Comma separated list of codes and ranges
  -fl                 Filter by amount of lines in response. Comma separated list of line counts and ranges
  -fmode              Filter set operator. Either of: and, or (default: or)
  -fr                 Filter regexp
  -fs                 Filter HTTP response size. Comma separated list of sizes and ranges
  -ft                 Filter by number of milliseconds to the first response byte, either greater or less than. EG: >100 or <100
  -fw                 Filter by amount of words in response. Comma separated list of word counts and ranges

INPUT OPTIONS:
  -D                  DirSearch wordlist compatibility mode. Used in conjunction with -e flag. (default: false)
  -e                  Comma separated list of extensions. Extends FUZZ keyword.
  -enc                Encoders for keywords, eg. 'FUZZ:urlencode b64encode'
  -ic                 Ignore wordlist comments (default: false)
  -input-cmd          Command producing the input. --input-num is required when using this input method. Overrides -w.
  -input-num          Number of inputs to test. Used in conjunction with --input-cmd. (default: 100)
  -input-shell        Shell to be used for running command
  -mode               Multi-wordlist operation mode. Available modes: clusterbomb, pitchfork, sniper (default: clusterbomb)
  -request            File containing the raw http request
  -request-proto      Protocol to use along with raw request (default: https)
  -w                  Wordlist file path and (optional) keyword separated by colon. eg. '/path/to/wordlist:KEYWORD'

OUTPUT OPTIONS:
  -debug-log          Write all of the internal logging to the specified file.
  -o                  Write output to file
  -od                 Directory path to store matched results to.
  -of                 Output file format. Available formats: json, ejson, html, md, csv, ecsv (or, 'all' for all formats) (default: json)
  -or                 Don't create the output file if we don't have results (default: false)

EXAMPLE USAGE:
  Fuzz file paths from wordlist.txt, match all responses but filter out those with content-size 42.
  Colored, verbose output.
    ffuf -w wordlist.txt -u https://example.org/FUZZ -mc all -fs 42 -c -v

  Fuzz Host-header, match HTTP 200 responses.
    ffuf -w hosts.txt -u https://example.org/ -H "Host: FUZZ" -mc 200

  Fuzz POST JSON data. Match all responses not containing text "error".
    ffuf -w entries.txt -u https://example.org/ -X POST -H "Content-Type: application/json" \
      -d '{"name": "FUZZ", "anotherkey": "anothervalue"}' -fr "error"

  Fuzz multiple locations. Match only responses reflecting the value of "VAL" keyword. Colored.
    ffuf -w params.txt:PARAM -w values.txt:VAL -u https://example.org/?PARAM=VAL -mr "VAL" -c

  More information and examples: https://github.com/ffuf/ffuf

That's quite a bit and everything that's there is what is output. Fortunately, you don't often need all of that. I'm going to go over some of the biggest use cases that you're likely to encounter. Honestly, anything that you can think of swapping out just one part of for another part works great, but these are the highlights.

Using ffuf for Enumeration

Vhost Enumeration

Virtual host (vhost) enumeration is essential for identifying additional subdomains or services hosted on the same server. Public servers can be discovered by DNS queries, but if you're fuzzing an internal application that you may not have access to full DNS, this is great.

I set up a stupid little node script to be a simple web server to work out some of these examples. So, some may seem a little contrived, but I also wanted to respect the rule about not fuzzing where you aren't supposed to. Also, ffuf is verbose when providing output (banner, parameter restatement, etc) and I will show it the first time, but subsequent samples will omit it. Just know that you'll see the banner et al every time when you're using the tool.

Our command is ffuf -w /usr/share/wordlists/seclists/Discovery/DNS/subdomains-top1million-5000.txt -u http://ffuf.hack:8000 -H "Host: FUZZ.ffuf.hack"

In this case, -w represents the wordlist to use when fuzzing. -u is the URL to hit (I just put ffuf.hack in my /etc/hosts pointed at my local machine), and -H is the HTTP header to send along, in this case, the Host that I'm trying to hit. Note that I put FUZZ in the spot where I'd like each word from the wordlist to go: foo.ffuf.hack, bar.ffuf.hack, baz.ffuf.hack, etc.

$ ffuf -w /usr/share/wordlists/seclists/Discovery/DNS/subdomains-top1million-5000.txt -u http://ffuf.hack:8000 -H "Host: FUZZ.ffuf.hack"

        /'___\  /'___\           /'___\
       /\ \__/ /\ \__/  __  __  /\ \__/
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/
         \ \_\   \ \_\  \ \____/  \ \_\
          \/_/    \/_/   \/___/    \/_/

       v2.1.0-dev
________________________________________________

 :: Method           : GET
 :: URL              : http://ffuf.hack:8000
 :: Wordlist         : FUZZ: /usr/share/wordlists/seclists/Discovery/DNS/subdomains-top1million-5000.txt
 :: Header           : Host: FUZZ.ffuf.hack
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200-299,301,302,307,401,403,405,500
________________________________________________

ns1                     [Status: 403, Size: 23, Words: 3, Lines: 1, Duration: 6ms]
mail                    [Status: 403, Size: 23, Words: 3, Lines: 1, Duration: 7ms]
// ****** SNIP ******
support                 [Status: 403, Size: 23, Words: 3, Lines: 1, Duration: 5ms]

I hit control-c here to break since we have our file size. One of the first tricks to learn about ffuf is to realize what it is going to consider a success or failure. You can see from the "Matcher" section that it will accept any of those response codes. For instance, 400 and 404 won't be returned. You can add additional codes or you can limit by file size (this version is the majority of the time for me). I snipped some out, but you can see that every negative response is 23 bytes. So, I then reissued the command with -fs 23 to filter size of 23 bytes and got the correct results. I left part of the header in this time so you can see that the response codes were the same, but a Filter was added for Response size.

$ ffuf -w /usr/share/wordlists/seclists/Discovery/DNS/subdomains-top1million-5000.txt -u http://ffuf.hack:8000 -H "Host: FUZZ.ffuf.hack" -fs 23

 :: Matcher          : Response status: 200-299,301,302,307,401,403,405,500
 :: Filter           : Response size: 23
________________________________________________

www                     [Status: 200, Size: 103732, Words: 14137, Lines: 948, Duration: 27ms]
lms                     [Status: 200, Size: 103732, Words: 14137, Lines: 948, Duration: 9ms]
:: Progress: [4989/4989] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

This is the correct result. I had set up my script to serve pages based on going to either ffuf.hack, www.ffuf.hack, or lms.ffuf.hack.

Directory Enumeration

Directory enumeration uncovers hidden directories on a web server. This one is even a more important use case for fuzzing, since there is nothing like DNS to otherwise find these directories. There are no new surprises with flags, just where I moved the FUZZ position to be replaced. I already knew the file size for a bad request and confirmed it, so that's included here right off the bat, too. In regular situations, you likely wouldn't even need the -H flag, but I'm using a janky simple web server script that I didn't want to keep changing, so I had to tag the Host header along, too. I'm also just enumerating the ffuf.hack domain itself, not any of the other vhosts. I am using a different word list, too. The correct wordlist to use is more art than science. You kind of get a feel for it, or you'll build up some favorites that always come through for you. All of the lists that I'm using come in the base Kali Linux install that I'm using.

$ ffuf -w /usr/share/wordlists/seclists/Discovery/Web-Content/raft-small-words-lowercase.txt -u http://ffuf.hack:8000/FUZZ -H "Host: ffuf.hack" -fs 23

________________________________________________

images                  [Status: 500, Size: 25, Words: 4, Lines: 1, Duration: 38ms]
.                       [Status: 500, Size: 25, Words: 4, Lines: 1, Duration: 26ms]
:: Progress: [38267/38267] :: Job [1/1] :: 3846 req/sec :: Duration: [0:00:10] :: Errors: 0 ::

In this case, this is also correct. I only made one directory in this pretend website and it was /images. If this was a penetration test, we'd probably wonder if the list was bad depending on what kind of site this was and how complex it seemed to be.

Page Enumeration

Page enumeration targets specific files or pages on the server. In this case, we use the -e flag (extension) and give it a comma-separated list of extensions for ffuf to tack onto our list. Here, I'm telling it to try .html, .jpg, and .js. I left the new part of the header in to show that ffuf has correctly understood what we're doing. Here, I used the same word list that I used for the directory enumeration.

$ ffuf -w /usr/share/wordlists/seclists/Discovery/Web-Content/raft-small-words-lowercase.txt -u http://ffuf.hack:8000/FUZZ -H "Host: ffuf.hack" -e ".html,.jpg,.js" -fs 23

 :: Extensions       : .html .jpg .js
________________________________________________

images                  [Status: 500, Size: 25, Words: 4, Lines: 1, Duration: 31ms]
index.html              [Status: 200, Size: 103732, Words: 14137, Lines: 948, Duration: 27ms]
main.js                 [Status: 200, Size: 59, Words: 8, Lines: 3, Duration: 22ms]
.                       [Status: 500, Size: 25, Words: 4, Lines: 1, Duration: 21ms]
server.js               [Status: 200, Size: 2294, Words: 594, Lines: 70, Duration: 30ms]
:: Progress: [153068/153068] :: Job [1/1] :: 4878 req/sec :: Duration: [0:00:41] :: Errors: 0 ::

Again, this is all correct. In fact, because I don't have this set up like a real node app or anything, it also found my server.js web server script. In a penetration test, if you could download this, you could read the source code for the application. It would be a huge finding as it should be excluded by web server configuration or by the way the application is set up.

Parameter Enumeration

Parameter enumeration identifies potential GET or POST parameters. ffuf can be used to find both parameter names and values. Which one you are trying to isolate depends on where you put the FUZZ. I'll show two examples here, first where I FUZZ where the parameter name is (with a junk value) and then fuzzing the values once I've identified the names. In this case, I'm fuzzing GET parameters. If you were fuzzing POST parameters, you'd just have the -u url be normal, and include a new flag of -d (data) like -d "param1=FUZZ" and then also add -X POST to the command for POST parameter fuzzing.

$ ffuf -w /usr/share/wordlists/seclists/Discovery/Web-Content/raft-small-words-lowercase.txt -H "Host: www.ffuf.hack" -u "http://ffuf.hack:8000/?FUZZ=foobar"
________________________________________________

first                   [Status: 200, Size: 119, Words: 20, Lines: 1, Duration: 12ms]
username                [Status: 200, Size: 119, Words: 20, Lines: 1, Duration: 8ms]
:: Progress: [38267/38267] :: Job [1/1] :: 904 req/sec :: Duration: [0:00:47] :: Errors: 0 ::

In this value fuzzing example, I made a short list to try it out. When fuzzing parameter values, you'll almost always want to have a "smart list" that you use. That could just be a list of numbers (like fuzzing an id parameter), a list of known usernames, a list of LFI escapes, etc. So in my case, I made a list of 14 values that would make sense to try to hack into an app written by the egotistical author of this blog 😉

// First Parameter Value Fuzzing
$ ffuf -w curated_values.txt -H "Host: www.ffuf.hack" -u "http://ffuf.hack:8000?username=FUZZ"
________________________________________________

peteonsoftware          [Status: 200, Size: 52, Words: 7, Lines: 1, Duration: 11ms]
:: Progress: [14/14] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::
// Second Parameter Value Fuzzing
$ ffuf -w curated_values.txt -H "Host: www.ffuf.hack" -u "http://ffuf.hack:8000?first=FUZZ"
________________________________________________

pete                    [Status: 200, Size: 42, Words: 7, Lines: 1, Duration: 10ms]
:: Progress: [14/14] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

So ffuf identified that my application takes two parameters: username and first. Valid values for username seem to only be peteonsoftware and the only valid value for the first parameter is pete. I can confirm that that's correct.

Parting Tips

If you don't care about how "loud" you are, you can increase how many threads that ffuf uses to bang on the server with -t. Also, if you're fuzzing an https URL and there are certificate issues, you use use -k to ignore the certificate problems.

As I touched on earlier, wordlists are key here, just like they are for password cracking and just about everything else. Daniel Miessler's SecLists can be a good place to get started.

Lastly, you are basically only limited by your imagination when using ffuf. If you can think of a creative way to build a request where you isolate one piece to constantly be replaced, you can use ffuf to do it. This overview only just gets you started. A good next place to go might be the TryHackMe ffuf room. It is a free room and anyone can do it (you don't have to have a paid subscription!). While I've had a paid subscription to TryHackMe for about a year and a half, they offer a ton of content for free and it is worth checking out!

Happy Fuzzing!

General Tips

Core Tools to Know: CyberChef

CyberChef LogoLast month, I started a series about tools and utilities that are good to know with a post about curl. In my most recent CTF post, I had to use CyberChef to help me with one of my steps. That post was already going to be very long, so I didn’t have a lot of time to explain what CyberChef was or how it worked if a reader was unfamiliar. It did, however, seem like a good topic for my next “Core Tools to Know” post, so here we are.

As some background, CyberChef was created by an anonymous worker at GCHQ (Government Communications Headquarters) in England. GCHQ is an intelligence agency and is the current version of the organization (then called GC&CS) that was at Bletchley Park and broke the Enigma Codes. So.. smart folks. Like a lot of high-tech places, GCHQ gives their employees 10% “Innovation Time” and this worker created this tool during that time. They shared it with their colleagues and eventually decided to open source it on GitHub for the world to benefit. So what is it? Let’s take a look.

The CyberChef UI

There is a lot going on here, but think of it like a factory or assembly line. In the upper right is where you put the input to the process. In the bottom right is the output. The center column is where you drag the “recipes”, which are the operations that you are going to conduct on the input. The left side shows all of the available recipes/operations that you can choose from. In the upper left is a search box to help you easily find what you were looking for. Let’s say you came across this string: UGV0ZSBvbiBTb2Z0d2FyZQ%3D%3D. If you recognize the end as looking to be URL encoded, let’s set up CyberChef to try to handle it. First, I put UGV0ZSBvbiBTb2Z0d2FyZQ%3D%3D in the input. It will immediately appear – unchanged – in the output because “Auto Bake” is selected at the bottom of the center column. If it isn’t, either check it or click Bake! to see something in the output. Next, let’s search for URL Decode in the search box and drag URL Decode to the Recipe column. After baking, you see that the output now says UGV0ZSBvbiBTb2Z0d2FyZQ== because each %3D was converted to an =. We now see that this looks like a base64 string.

Our input with the URL Decode Recipe Only

What is interesting is that CyberChef figured that out, too! You’ll notice that a magic wand appeared by the Output header and if you hover on it, it tells you that this is a Base64 string and what the decoding is.

Output Magic Suggestion

If you click it, it adds From Base64 to the recipe and our output now shows Pete on Software.

Our decoded string

It is important to note that order matters. You can drag the recipe steps up and down to perform them in a different order. If I From Base64 first and URL Decode second, we don’t get the exact right answer. If this was a more complicated recipe, small errors get compounded and might make this fail entirely.

Recipe Order Matters

As we move forward, it is important to note that you can remove a recipe step by either clicking and dragging it to the trash can in the upper right portion of the Recipe column, or by clicking the ⊘ on the operation. That will disable it. Clicking that symbol again will re-enable it.

By now, you can imagine all of the things you can do to text. CyberChef has encryption, hashing, encoding, and much, much more. However, you aren’t limited to text. You can perform network operations and even operate on Files. If you don’t enter text in the Input section, you have other options for input. If you hover on each of the icons, you can see that you can also open a Folder or a File as input.

CyberChef Input Options

I’m going to use CyberChef to remove EXIF data from an image of a moped. To start, here is the EXIF data that is on the image.

$ exif moped.jpg
EXIF tags in 'moped.jpg' ('Motorola' byte order):
--------------------+----------------------------------------------------------
Tag                 |Value
--------------------+----------------------------------------------------------
X-Resolution        |72
Y-Resolution        |72
Resolution Unit     |Inch
Date and Time (Origi|2018:11:08 19:49:33
User Comment        |Screenshot
Pixel X Dimension   |640
Pixel Y Dimension   |641
Exif Version        |Exif Version 2.1
FlashPixVersion     |FlashPix Version 1.0
Color Space         |Internal error (unknown value 65535)
--------------------+----------------------------------------------------------

I selected my file as input, added the Remove EXIF operation to the Recipe, made sure it Baked and I have file data in the output. If you click the Floppy Disk icon, you can save that output to a file. (Note: CyberChef didn’t blur the image. I did for privacy reasons)

CyberChef Remove EXIF Save File

$ exif moped_cyberchef.jpg
Corrupt data
The data provided does not follow the specification.
ExifLoader: The data supplied does not seem to contain EXIF data.

I will note that I was using a Linux utility to read that EXIF data, but CyberChef could have read the data in the first place. There is an Extract EXIF operation that you can add to the recipe.

As we wrap up, there are some things to consider. First, CyberChef is a client-side application. When you enter input (text, file, folder, etc), your data does not leave the browser. All of the recipes are written in JavaScript and are executed on the client. You don’t have to worry that some agency is getting your encryption keys, pictures, or other sensitive data. As a reminder, the code is open source and can be viewed and reviewed. Also, you don’t have to be online to use CyberChef. There is a download link at the top left that will let you download a ZIP file of the application to run locally. When you click the link, you get a popup that provides this message. It is important to note that this isn’t an installed application and there is no provision for keeping it updated unless you create those means.

CyberChef runs entirely within your browser with no server-side component, meaning that your Input data and Recipe configuration are not sent anywhere, whether you use the live, official version of CyberChef or a downloaded, standalone version (assuming it is unmodified).

There are three operations that make calls to external services, those being the 'Show on map' operation which downloads map tiles from wikimedia.org, the 'DNS over HTTPS' operation which resolves DNS requests using either Google or Cloudflare services, and the 'HTTP request' operation that calls out to the configured URL you enter. You can confirm what network requests are made using your browser's developer console (F12) and viewing the Network tab.

If you would like to download your own standalone copy of CyberChef to run in a segregated network or where there is limited or no Internet connectivity, you can get a ZIP file containing the whole web app below. This can be run locally or hosted on a web server with no configuration required.

Be aware that the standalone version will never update itself, meaning it will not receive bug fixes or new features until you re-download newer versions manually.

That’s it! I hope you got a good introduction to CyberChef and keep it in mind when you have little needs to manipulate data. If there is anything that you have found very handy to use it for or any good tips and tricks, I’d love to hear about it in the comments.

General Tips

Core Tools to Know: curl

'curl'-y pig's tailAs all bloggers eventually find out, two of the best reasons to write blog posts are either to document something for yourself for later or to force yourself to learn something well enough to explain it to others. That’s the impetus of this series that I plan on doing from time to time. I want to get more familiar with some of these core tools and also have a reference / resource available that is in “Pete Think” so I can quickly find what I need to use these tools if I forget.

The first tool I want to tackle is curl. In the world of command-line tools, curl shines as a flexible utility for transferring data over various network protocols. Whether you’re working on the development side, the network admin side, or the security side, learning how to use curl effectively can be incredibly beneficial. This blog post will guide you through the very basics of curl, cover some common use cases, and explain when curl might be a better choice than wget.

What is curl

curl (Client for URL) is a command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, SCP, TELNET, LDAP, IMAP, SMB, and many more. curl is known for its flexibility and is widely used for interacting with APIs, downloading files, and testing network connections.

Installing curl

Before diving into curl commands, you need to ensure it is installed on your system. Lots of operating systems come with it. In fact, even Windows has shipped with curl in Windows 10 and 11.

For Linux:

sudo apt-get install curl  # Debian/Ubuntu
sudo yum install curl      # CentOS/RHEL

For macOS:

brew install curl

For Windows
You can download the installer from the official curl website if it isn’t already on your system. To check, just type curl –help at the command prompt and see if it understand the command. If you get something back like this, you’re all set.

C:\Users\peteonsoftware>curl --help
Usage: curl [options...] <url>
 -d, --data <data>           HTTP POST data
 -f, --fail                  Fail fast with no output on HTTP errors
 -h, --help <category>       Get help for commands
 -i, --include               Include response headers in output
 -o, --output <file>         Write to file instead of stdout
 -O, --remote-name           Write output to file named as remote file
 -s, --silent                Silent mode
 -T, --upload-file <file>    Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>     Send User-Agent <name> to server
 -v, --verbose               Make the operation more talkative
 -V, --version               Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".

The Most Simple Example

The simplest way to use curl is to fetch the contents of a URL. Here is a basic example that will will print the HTML content of the specified URL to the terminal:

c:\
λ curl https://hosthtml.live
<!doctype html>
<html data-adblockkey="MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANDrp2lz7AOmADaN8tA50LsWcjLFyQFcb/P2Txc58oYOeILb3vBw7J6f4pamkAQVSQuqYsKx3YzdUHCvbVZvFUsCAwEAAQ==_M6heeSY2n3p1IRsqfcIljkNrgqYXDBDFSWeybupIpyihjfHMZhFu8kniDL51hLxUnYHjgmcv2EYUtXfRDcRWZQ==" lang="en" style="background: #2B2B2B;">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12P4//8/AAX+Av7czFnnAAAAAElFTkSuQmCC">
    <link rel="preconnect" href="https://www.google.com" crossorigin>
</head>
<body>
<div id="target" style="opacity: 0"></div>
<script>window.park = "eyJ1dWlkIjoiZDFhODUxY2ItOTUyZi00NGUyLTg4ZWMtMmU3ZGNhZmE1OTk0IiwicGFnZV90aW1lIjoxNzIwNzMyMzQxLCJwYWdlX3VybCI6Imh0dHBzOi8vaG9zdGh0bWwubGl2ZS8iLCJwYWdlX21ldGhvZCI6IkdFVCIsInBhZ2VfcmVxdWVzdCI6e30sInBhZ2VfaGVhZGVycyI6e30sImhvc3QiOiJob3N0aHRtbC5saXZlIiwiaXAiOiI3Mi4xMDQuMTY5LjE1NCJ9Cg==";</script>
<script src="/bwjblpHBR.js"></script>
</body>
</html>

Some Useful Examples to Actually Do Stuff

Downloading Files

# To download a file and save it with a specific name:
curl -o curlypigtail.jpg https://peteonsoftware.com/images/202407/curlytail.jpg

# If you want to save the file with the same name as in the URL:
curl -O https://peteonsoftware.com/images/202407/curlytail.jpg

Sending HTTP Requests

# GET requests are used to retrieve data from a server. The basic example is already shown above. 
# To include headers in the output, use the -i option:
curl -i https://hosthtml.live

# POST requests are used to send data to a server. 
# This is particularly useful when interacting with APIs. 
curl -X POST -d "param1=value1&param2=value2" http://somereallygreat.net/api

# Many APIs now accept JSON.  This is how you'd send that
curl -X POST -H "Content-Type: application/json" -d '{"key1":"value1", "key2":"value2"}' http://somereallygreat.net/api

# Without explaining it, we included a header above (-H).  That added a Content-Type header.  
# To add an Auth header, you might do something like this
curl -H "Authorization: Bearer token" http://asitethatneedsbearertokens.com

Cookies

# To save cookies from a response
curl -c cookies.txt https://www.google.com

# To send cookies with a request
curl -b cookies.txt https://www.google.com

When to Use curl Over wget

While both curl and wget are used to transfer data over the internet, they have different strengths. Daniel Stenberg is the creator of curl (and also contributes to wget) and he’s published a more lengthy comparison here. I defer to the expert, but here are some of my big takeaways.

curl Advantages

  • Flexibility: curl supports a wider range of protocols (like SCP, SFTP) and provides more options for customizing requests.
  • Availability: curl comes preinstalled on macOS and Windows 10/11. wget doesn’t.
  • Pipelining: curl can be used to chain multiple requests together, making it powerful for scripting complex interactions.
  • Reuse: curl is a library (libcurl), while wget is just a command line tool.

wget Advantages

  • Recursive Downloads: wget can download entire websites recursively, making it ideal for mirroring sites.
  • Simplicity: For simple downloading tasks, wget might be more straightforward and easier to use.

curl is a versatile tool that – once mastered – can simplify many network-related tasks. From downloading files to interacting with APIs, curl provides the flexibility and functionality needed for a wide range of applications. While wget has its strengths, particularly for simple downloads and recursive website copying, curl shines in its versatility and extensive options for customizing requests.

General Tips

Think Outside the Box

Entrepreneur on Fire Podcast Logo
I was listening to episode 682 of the Entrepreneur on Fire Podcast, where John Lee Dumas was interviewing Shane Snow. Shane has written a book called Smartcuts: How Hackers, Innovators, and Icons Accelerate Success. An abbreviated description from Amazon reads:

Entrepreneur and journalist Shane Snow analyzes the lives of people and companies that do incredible things in implausibly short time… One way or another, they do it like computer hackers. They employ what psychologists call “lateral thinking” to rethink convention and break “rules” that aren’t rules.

Shane Snow
During the show, Shane gave a small story to illustrate the general idea of the kind of lateral thinking that he is espousing.

Pretend that you are driving a car on a rainy night and you’re kind of in the middle of nowhere and windshield wipers are going like crazy and you see on the side of the road three people that are standing out there miserable in the rain. You slow down and the first one you notice is a friend of yours that saved your life one time. Next to him is a little old lady with a cane who definitely can’t make it back to town on her own. And next to them is the man or woman of your dreams and this is your only shot to ever meet them. You only have one seat in your car (meaning a car with a total of two seats), who do you pick up?

John Lee Dumas suggested that he would pick up the little old lady. That is just the “military man” in him. Shane’s point was that lateral thinking “hacks” the question and finds the answer by thinking outside the box. In Shane’s solution, you give the seat to the old lady, give your keys to your friend, and stay with the love of your life.

I love that illustration. Yes, it is contrived, but it is a perfect word picture of how creative problem solvers think. The problem that most people have with the problem is that they make assumptions. In this case, you’ve assumed that you must stay with your car and that you must only save one of the three people in distress. Lateral thinking throws out those assumptions and looks for a way for “everyone to win”. In real life, 100% satisfaction certainly isn’t always possible, but you will get a lot closer by throwing out assumptions and “hacking the question”.

Great reminder for the day.

General Tips

We have a lot of great content to show you

This is definitely one of those blog posts that will serve as a reminder to me when I forget the answers in the future.

I’m using Visual Studio 2013, when I noticed a message under the Product Videos header that said, “We have a lot of great content to show you, but we need your permission to keep it updated”.

We have a lot of great content to show you, but we need your permission to keep it updated

I am the kind of person who enjoys knowing what is going on and staying current and I think everyone knows that Microsoft really seems to care about developers. As such, they put out a lot of amazing content through their channels and partner channels. I wanted in on this gig. So, I did some digging and to “give permission”, I needed to go to the Tools menu and select Options.

Visual Studio 2013 Tools .. Options

Then, from the dialog box that popped up, I select Startup on the left and then put a checkmark in the box next to “Download content every:”. I left it at the default of 60 minutes and clicked OK.

Visual Studio 2013 Options .. Startup .. Download Every

My startup screen then immediately refreshed and now I have a list of videos that I can check out.

Visual Studio 2013 Product Videos Shown