Category: General Tips

General Tips

Core Tools to Know: CyberChef

CyberChef LogoLast month, I started a series about tools and utilities that are good to know with a post about curl. In my most recent CTF post, I had to use CyberChef to help me with one of my steps. That post was already going to be very long, so I didn’t have a lot of time to explain what CyberChef was or how it worked if a reader was unfamiliar. It did, however, seem like a good topic for my next “Core Tools to Know” post, so here we are.

As some background, CyberChef was created by an anonymous worker at GCHQ (Government Communications Headquarters) in England. GCHQ is an intelligence agency and is the current version of the organization (then called GC&CS) that was at Bletchley Park and broke the Enigma Codes. So.. smart folks. Like a lot of high-tech places, GCHQ gives their employees 10% “Innovation Time” and this worker created this tool during that time. They shared it with their colleagues and eventually decided to open source it on GitHub for the world to benefit. So what is it? Let’s take a look.

The CyberChef UI

There is a lot going on here, but think of it like a factory or assembly line. In the upper right is where you put the input to the process. In the bottom right is the output. The center column is where you drag the “recipes”, which are the operations that you are going to conduct on the input. The left side shows all of the available recipes/operations that you can choose from. In the upper left is a search box to help you easily find what you were looking for. Let’s say you came across this string: UGV0ZSBvbiBTb2Z0d2FyZQ%3D%3D. If you recognize the end as looking to be URL encoded, let’s set up CyberChef to try to handle it. First, I put UGV0ZSBvbiBTb2Z0d2FyZQ%3D%3D in the input. It will immediately appear – unchanged – in the output because “Auto Bake” is selected at the bottom of the center column. If it isn’t, either check it or click Bake! to see something in the output. Next, let’s search for URL Decode in the search box and drag URL Decode to the Recipe column. After baking, you see that the output now says UGV0ZSBvbiBTb2Z0d2FyZQ== because each %3D was converted to an =. We now see that this looks like a base64 string.

Our input with the URL Decode Recipe Only

What is interesting is that CyberChef figured that out, too! You’ll notice that a magic wand appeared by the Output header and if you hover on it, it tells you that this is a Base64 string and what the decoding is.

Output Magic Suggestion

If you click it, it adds From Base64 to the recipe and our output now shows Pete on Software.

Our decoded string

It is important to note that order matters. You can drag the recipe steps up and down to perform them in a different order. If I From Base64 first and URL Decode second, we don’t get the exact right answer. If this was a more complicated recipe, small errors get compounded and might make this fail entirely.

Recipe Order Matters

As we move forward, it is important to note that you can remove a recipe step by either clicking and dragging it to the trash can in the upper right portion of the Recipe column, or by clicking the ⊘ on the operation. That will disable it. Clicking that symbol again will re-enable it.

By now, you can imagine all of the things you can do to text. CyberChef has encryption, hashing, encoding, and much, much more. However, you aren’t limited to text. You can perform network operations and even operate on Files. If you don’t enter text in the Input section, you have other options for input. If you hover on each of the icons, you can see that you can also open a Folder or a File as input.

CyberChef Input Options

I’m going to use CyberChef to remove EXIF data from an image of a moped. To start, here is the EXIF data that is on the image.

$ exif moped.jpg
EXIF tags in 'moped.jpg' ('Motorola' byte order):
--------------------+----------------------------------------------------------
Tag                 |Value
--------------------+----------------------------------------------------------
X-Resolution        |72
Y-Resolution        |72
Resolution Unit     |Inch
Date and Time (Origi|2018:11:08 19:49:33
User Comment        |Screenshot
Pixel X Dimension   |640
Pixel Y Dimension   |641
Exif Version        |Exif Version 2.1
FlashPixVersion     |FlashPix Version 1.0
Color Space         |Internal error (unknown value 65535)
--------------------+----------------------------------------------------------

I selected my file as input, added the Remove EXIF operation to the Recipe, made sure it Baked and I have file data in the output. If you click the Floppy Disk icon, you can save that output to a file. (Note: CyberChef didn’t blur the image. I did for privacy reasons)

CyberChef Remove EXIF Save File

$ exif moped_cyberchef.jpg
Corrupt data
The data provided does not follow the specification.
ExifLoader: The data supplied does not seem to contain EXIF data.

I will note that I was using a Linux utility to read that EXIF data, but CyberChef could have read the data in the first place. There is an Extract EXIF operation that you can add to the recipe.

As we wrap up, there are some things to consider. First, CyberChef is a client-side application. When you enter input (text, file, folder, etc), your data does not leave the browser. All of the recipes are written in JavaScript and are executed on the client. You don’t have to worry that some agency is getting your encryption keys, pictures, or other sensitive data. As a reminder, the code is open source and can be viewed and reviewed. Also, you don’t have to be online to use CyberChef. There is a download link at the top left that will let you download a ZIP file of the application to run locally. When you click the link, you get a popup that provides this message. It is important to note that this isn’t an installed application and there is no provision for keeping it updated unless you create those means.

CyberChef runs entirely within your browser with no server-side component, meaning that your Input data and Recipe configuration are not sent anywhere, whether you use the live, official version of CyberChef or a downloaded, standalone version (assuming it is unmodified).

There are three operations that make calls to external services, those being the 'Show on map' operation which downloads map tiles from wikimedia.org, the 'DNS over HTTPS' operation which resolves DNS requests using either Google or Cloudflare services, and the 'HTTP request' operation that calls out to the configured URL you enter. You can confirm what network requests are made using your browser's developer console (F12) and viewing the Network tab.

If you would like to download your own standalone copy of CyberChef to run in a segregated network or where there is limited or no Internet connectivity, you can get a ZIP file containing the whole web app below. This can be run locally or hosted on a web server with no configuration required.

Be aware that the standalone version will never update itself, meaning it will not receive bug fixes or new features until you re-download newer versions manually.

That’s it! I hope you got a good introduction to CyberChef and keep it in mind when you have little needs to manipulate data. If there is anything that you have found very handy to use it for or any good tips and tricks, I’d love to hear about it in the comments.

General Tips

Core Tools to Know: curl

'curl'-y pig's tailAs all bloggers eventually find out, two of the best reasons to write blog posts are either to document something for yourself for later or to force yourself to learn something well enough to explain it to others. That’s the impetus of this series that I plan on doing from time to time. I want to get more familiar with some of these core tools and also have a reference / resource available that is in “Pete Think” so I can quickly find what I need to use these tools if I forget.

The first tool I want to tackle is curl. In the world of command-line tools, curl shines as a flexible utility for transferring data over various network protocols. Whether you’re working on the development side, the network admin side, or the security side, learning how to use curl effectively can be incredibly beneficial. This blog post will guide you through the very basics of curl, cover some common use cases, and explain when curl might be a better choice than wget.

What is curl

curl (Client for URL) is a command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, SCP, TELNET, LDAP, IMAP, SMB, and many more. curl is known for its flexibility and is widely used for interacting with APIs, downloading files, and testing network connections.

Installing curl

Before diving into curl commands, you need to ensure it is installed on your system. Lots of operating systems come with it. In fact, even Windows has shipped with curl in Windows 10 and 11.

For Linux:

sudo apt-get install curl  # Debian/Ubuntu
sudo yum install curl      # CentOS/RHEL

For macOS:

brew install curl

For Windows
You can download the installer from the official curl website if it isn’t already on your system. To check, just type curl –help at the command prompt and see if it understand the command. If you get something back like this, you’re all set.

C:\Users\peteonsoftware>curl --help
Usage: curl [options...] <url>
 -d, --data <data>           HTTP POST data
 -f, --fail                  Fail fast with no output on HTTP errors
 -h, --help <category>       Get help for commands
 -i, --include               Include response headers in output
 -o, --output <file>         Write to file instead of stdout
 -O, --remote-name           Write output to file named as remote file
 -s, --silent                Silent mode
 -T, --upload-file <file>    Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>     Send User-Agent <name> to server
 -v, --verbose               Make the operation more talkative
 -V, --version               Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".

The Most Simple Example

The simplest way to use curl is to fetch the contents of a URL. Here is a basic example that will will print the HTML content of the specified URL to the terminal:

c:\
λ curl https://hosthtml.live
<!doctype html>
<html data-adblockkey="MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANDrp2lz7AOmADaN8tA50LsWcjLFyQFcb/P2Txc58oYOeILb3vBw7J6f4pamkAQVSQuqYsKx3YzdUHCvbVZvFUsCAwEAAQ==_M6heeSY2n3p1IRsqfcIljkNrgqYXDBDFSWeybupIpyihjfHMZhFu8kniDL51hLxUnYHjgmcv2EYUtXfRDcRWZQ==" lang="en" style="background: #2B2B2B;">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12P4//8/AAX+Av7czFnnAAAAAElFTkSuQmCC">
    <link rel="preconnect" href="https://www.google.com" crossorigin>
</head>
<body>
<div id="target" style="opacity: 0"></div>
<script>window.park = "eyJ1dWlkIjoiZDFhODUxY2ItOTUyZi00NGUyLTg4ZWMtMmU3ZGNhZmE1OTk0IiwicGFnZV90aW1lIjoxNzIwNzMyMzQxLCJwYWdlX3VybCI6Imh0dHBzOi8vaG9zdGh0bWwubGl2ZS8iLCJwYWdlX21ldGhvZCI6IkdFVCIsInBhZ2VfcmVxdWVzdCI6e30sInBhZ2VfaGVhZGVycyI6e30sImhvc3QiOiJob3N0aHRtbC5saXZlIiwiaXAiOiI3Mi4xMDQuMTY5LjE1NCJ9Cg==";</script>
<script src="/bwjblpHBR.js"></script>
</body>
</html>

Some Useful Examples to Actually Do Stuff

Downloading Files

# To download a file and save it with a specific name:
curl -o curlypigtail.jpg https://peteonsoftware.com/images/202407/curlytail.jpg

# If you want to save the file with the same name as in the URL:
curl -O https://peteonsoftware.com/images/202407/curlytail.jpg

Sending HTTP Requests

# GET requests are used to retrieve data from a server. The basic example is already shown above. 
# To include headers in the output, use the -i option:
curl -i https://hosthtml.live

# POST requests are used to send data to a server. 
# This is particularly useful when interacting with APIs. 
curl -X POST -d "param1=value1&param2=value2" http://somereallygreat.net/api

# Many APIs now accept JSON.  This is how you'd send that
curl -X POST -H "Content-Type: application/json" -d '{"key1":"value1", "key2":"value2"}' http://somereallygreat.net/api

# Without explaining it, we included a header above (-H).  That added a Content-Type header.  
# To add an Auth header, you might do something like this
curl -H "Authorization: Bearer token" http://asitethatneedsbearertokens.com

Cookies

# To save cookies from a response
curl -c cookies.txt https://www.google.com

# To send cookies with a request
curl -b cookies.txt https://www.google.com

When to Use curl Over wget

While both curl and wget are used to transfer data over the internet, they have different strengths. Daniel Stenberg is the creator of curl (and also contributes to wget) and he’s published a more lengthy comparison here. I defer to the expert, but here are some of my big takeaways.

curl Advantages

  • Flexibility: curl supports a wider range of protocols (like SCP, SFTP) and provides more options for customizing requests.
  • Availability: curl comes preinstalled on macOS and Windows 10/11. wget doesn’t.
  • Pipelining: curl can be used to chain multiple requests together, making it powerful for scripting complex interactions.
  • Reuse: curl is a library (libcurl), while wget is just a command line tool.

wget Advantages

  • Recursive Downloads: wget can download entire websites recursively, making it ideal for mirroring sites.
  • Simplicity: For simple downloading tasks, wget might be more straightforward and easier to use.

curl is a versatile tool that – once mastered – can simplify many network-related tasks. From downloading files to interacting with APIs, curl provides the flexibility and functionality needed for a wide range of applications. While wget has its strengths, particularly for simple downloads and recursive website copying, curl shines in its versatility and extensive options for customizing requests.

General Tips

Think Outside the Box

Entrepreneur on Fire Podcast Logo
I was listening to episode 682 of the Entrepreneur on Fire Podcast, where John Lee Dumas was interviewing Shane Snow. Shane has written a book called Smartcuts: How Hackers, Innovators, and Icons Accelerate Success. An abbreviated description from Amazon reads:

Entrepreneur and journalist Shane Snow analyzes the lives of people and companies that do incredible things in implausibly short time… One way or another, they do it like computer hackers. They employ what psychologists call “lateral thinking” to rethink convention and break “rules” that aren’t rules.

Shane Snow
During the show, Shane gave a small story to illustrate the general idea of the kind of lateral thinking that he is espousing.

Pretend that you are driving a car on a rainy night and you’re kind of in the middle of nowhere and windshield wipers are going like crazy and you see on the side of the road three people that are standing out there miserable in the rain. You slow down and the first one you notice is a friend of yours that saved your life one time. Next to him is a little old lady with a cane who definitely can’t make it back to town on her own. And next to them is the man or woman of your dreams and this is your only shot to ever meet them. You only have one seat in your car (meaning a car with a total of two seats), who do you pick up?

John Lee Dumas suggested that he would pick up the little old lady. That is just the “military man” in him. Shane’s point was that lateral thinking “hacks” the question and finds the answer by thinking outside the box. In Shane’s solution, you give the seat to the old lady, give your keys to your friend, and stay with the love of your life.

I love that illustration. Yes, it is contrived, but it is a perfect word picture of how creative problem solvers think. The problem that most people have with the problem is that they make assumptions. In this case, you’ve assumed that you must stay with your car and that you must only save one of the three people in distress. Lateral thinking throws out those assumptions and looks for a way for “everyone to win”. In real life, 100% satisfaction certainly isn’t always possible, but you will get a lot closer by throwing out assumptions and “hacking the question”.

Great reminder for the day.

General Tips

We have a lot of great content to show you

This is definitely one of those blog posts that will serve as a reminder to me when I forget the answers in the future.

I’m using Visual Studio 2013, when I noticed a message under the Product Videos header that said, “We have a lot of great content to show you, but we need your permission to keep it updated”.

We have a lot of great content to show you, but we need your permission to keep it updated

I am the kind of person who enjoys knowing what is going on and staying current and I think everyone knows that Microsoft really seems to care about developers. As such, they put out a lot of amazing content through their channels and partner channels. I wanted in on this gig. So, I did some digging and to “give permission”, I needed to go to the Tools menu and select Options.

Visual Studio 2013 Tools .. Options

Then, from the dialog box that popped up, I select Startup on the left and then put a checkmark in the box next to “Download content every:”. I left it at the default of 60 minutes and clicked OK.

Visual Studio 2013 Options .. Startup .. Download Every

My startup screen then immediately refreshed and now I have a list of videos that I can check out.

Visual Studio 2013 Product Videos Shown