Mastering Data Transfer with URLs: A Deep Dive into `curl` (and `httpie`)

Table of Contents#

1. Introduction to curl and httpie
2. Getting Started with curl
   • Basic HTTP Requests (GET, POST, PUT, DELETE)
   • Working with Headers
   • Authentication (Basic, Bearer)
   • File Downloads & Uploads
   • Cookie Management
3. Getting Started with httpie
   • Intuitive HTTP Requests
   • Headers, Data, and Authentication
   • File Operations (Download/Upload)
   • Sessions and Cookies
4. Comparing curl and httpie
5. Common Use Cases
   • API Testing
   • Debugging Web Services
   • Automation & Scripting
6. Best Practices
   • Security
   • Scripting & Automation
   • Performance
7. Advanced Features
   • curl: Tunneling, Proxies, and Beyond
   • httpie: Extensions & Plugins
8. Conclusion
9. References

1. Introduction to curl and httpie#

curl#

• What is it? A command-line tool and library for transferring data using URLs. It supports over 25 protocols, including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, SMB, SMTP, IMAP, POP3, LDAP, MQTT, WebSocket (WS/WSS), and more.
• Use Cases: Automating file transfers, API testing, debugging network requests, and scripting workflows.
• Ubiquity: Pre-installed on most Unix-like systems (Linux, macOS) and Windows 10+. The latest stable version is 8.21.0 (as of 2026).

httpie#

• What is it? A modern, user-friendly command-line HTTP client designed for humans. It prioritizes readability, simplicity, and JSON support. The latest stable version is 3.2.4 (as of 2024).
• Use Cases: Interactive API testing, debugging web services, and quick HTTP requests (especially with JSON).
• Philosophy: “CLI HTTP client for the API era.” It produces colored, formatted output (e.g., pretty-printed JSON) by default. Written in Python and uses the Requests library.

2. Getting Started with curl#

2.1 Basic HTTP Requests#

curl uses flags (-X, -d, -H, etc.) to customize requests.

GET Request (Default)#

Fetch a web page:

curl https://example.com

POST Request (Send Data)#

Send form data to a server:

curl -X POST -d "name=John&age=30" https://example.com/api/users

For JSON data:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"name":"John", "age":30}' \
  https://example.com/api/users

A simpler approach using curl's --json flag (available since version 7.82.0):

curl --json '{"name":"John", "age":30}' https://example.com/api/users

PUT Request (Update Resource)#

Update a resource (e.g., a user’s profile):

curl -X PUT \
  -H "Content-Type: application/json" \
  -d '{"name":"Updated Name"}' \
  https://example.com/api/users/123

DELETE Request (Remove Resource)#

Delete a resource:

curl -X DELETE https://example.com/api/users/123

2.2 Working with Headers#

Add Headers#

Include custom headers (e.g., authentication, content type):

curl -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  https://api.example.com/data

Show Response Headers#

Include headers in the output:

curl -i https://example.com  # Includes headers + body

Only Headers (HEAD Request)#

Fetch headers without the response body:

curl -I https://example.com  # Sends a HEAD request

2.3 Authentication#

Basic Authentication#

Pass a username and password (base64-encoded):

curl -u username:password https://example.com/secure

Bearer Token (OAuth/API Tokens)#

Include a bearer token in the Authorization header:

curl -H "Authorization: Bearer YOUR_TOKEN" https://api.example.com/secure

2.4 File Downloads & Uploads#

Download a File#

Save a file with a custom name:

curl -o output.html https://example.com/page.html

Resume an interrupted download:

curl -C - -o large_file.zip https://example.com/large_file.zip

Upload a File (Form Data)#

Send a file as part of a form:

curl -F "file=@/path/to/file.pdf" \
  -F "description=My Document" \
  https://example.com/upload

Upload JSON Data (File)#

Send a JSON file as the request body:

curl -X POST \
  -H "Content-Type: application/json" \
  -d @data.json \
  https://example.com/api/import

2.5 Cookie Management#

Send Cookies#

Include cookies in a request:

curl -b "session_id=123; user=john" https://example.com/dashboard

Save Cookies (and Reuse)#

Save cookies to a file, then reuse them:

# Save cookies
curl -c cookies.txt https://example.com/login
 
# Reuse cookies
curl -b cookies.txt https://example.com/dashboard

3. Getting Started with httpie#

httpie uses a concise, human-friendly syntax (no flags for basic requests).

3.1 Intuitive HTTP Requests#

GET Request (Default)#

Fetch a web page (output is formatted/colored):

http https://example.com

POST Request (Send Data)#

Send form data (auto-detects Content-Type):

http POST https://example.com/api/users name=John age=30

For JSON data (default for POST/PUT):

http POST https://example.com/api/users name=John age:=30  # `:=` for raw JSON values

3.2 Headers, Data, and Authentication#

Add Headers#

Include custom headers (e.g., authentication):

http GET https://api.example.com/data \
  "Authorization: Bearer YOUR_TOKEN" \
  "Content-Type: application/json"

Basic Authentication#

Prompt for a password (safer than plain text):

http -a username https://example.com/secure  # Prompts for password

3.3 File Operations#

Download a File#

Save a file (auto-names it nicely):

http --download https://example.com/file.zip

Upload a File#

Send a file as part of a request:

http POST https://example.com/upload file@/path/to/file.pdf

3.4 Sessions and Cookies#

Save a Session#

Create a session (saves cookies, headers, etc.):

http --session=my_session https://example.com/login username=john password=secret

Reuse a Session#

Use the saved session for subsequent requests:

http --session=my_session https://example.com/dashboard

4. Comparing curl and httpie#

5. Common Use Cases#

5.1 API Testing#

Test a REST API (e.g., create a user):

• curl:

  curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"name":"Alice", "role":"admin"}' \
    https://api.example.com/users

• httpie:

  http POST https://api.example.com/users name=Alice role=admin

5.2 Debugging Web Services#

View request/response details (verbose mode):

• curl:

  curl -v https://example.com  # Shows request/response headers

• httpie:

  http --verbose https://example.com

5.3 Automation & Scripting#

• curl is ideal for bash scripts (ubiquitous across systems):

  # Fetch a token and use it
  TOKEN=$(curl -u user:pass https://api.example.com/token)
  curl -H "Authorization: Bearer $TOKEN" https://api.example.com/data

• httpie can also be scripted, but curl is more portable.

6. Best Practices#

6.1 Security#

• Avoid Plaintext Credentials: Use environment variables, .netrc files, or httpie's password prompt (-a username without specifying the password).
• Verify Certificates: Use https and avoid --insecure (curl) or --verify no (httpie) in production.
• Protect Sensitive Data: Never hardcode API keys or tokens in scripts; use environment variables instead.

6.2 Scripting & Automation#

• Use --json for JSON APIs: curl --json simplifies JSON requests by setting the correct Content-Type header automatically.
• Reuse Connections: Use curl's --keepalive or httpie's --session to reduce overhead.
• Error Handling: Check exit codes (e.g., curl -f fails on 4xx/5xx, httpie exits non-zero on errors).
• Retry Logic: Use curl --retry for unreliable connections:

  curl --retry 3 --retry-delay 5 https://api.example.com/data

6.3 Performance#

• Limit Rate: Throttle requests (e.g., curl --limit-rate 100K to cap download speed).
• Parallel Requests: curl -Z (parallel transfers) or httpie with xargs for parallelism.
• Compression: Request compressed responses with curl --compressed to reduce bandwidth.

7. Advanced Features#

7.1 curl: Tunneling, Proxies, and Beyond#

• Proxy Support: Route traffic through a proxy:

  curl -x http://proxy:8080 https://example.com

• SSH Tunneling: Forward traffic over SSH (useful for private APIs):

  ssh -L 8080:internal-api:80 user@jump-server
  curl http://localhost:8080  # Access internal API

• URL Globbing: Fetch multiple URLs with ranges and sequences:

  curl https://example.com/file[1-5].txt
  curl https://example.com/{one,two,three}.html

• Parallel Transfers: Download multiple files concurrently:

  curl -Z https://example.com/file1.txt https://example.com/file2.txt

• AWS Signature V4: Authenticate with AWS services:

  curl --aws-sigv4 "aws:amz:us-east-1:s3" --user "KEY:SECRET" \
    https://s3.amazonaws.com/bucket

7.2 httpie: Extensions & Plugins#

httpie has a built-in plugin manager (since version 3.0.0) for installing and managing plugins:

httpie cli plugins install httpie-plugin-name
httpie cli plugins list

Notable plugins include:

• Auth Plugins: Extend authentication beyond Basic and Digest (e.g., OAuth, AWS Signature)
• Format Plugins: Support for additional output formats
• Transport Plugins: Unix sockets and other transport protocols

8. Conclusion#

• curl is a battle-tested, versatile tool for any protocol and scripting. It excels in automation, cross-platform compatibility, and performance. With support for HTTP/2, HTTP/3, WebSocket, and 25+ protocols, it remains the most capable command-line transfer tool available.
• httpie is a modern, intuitive alternative for HTTP(S) workflows, with beautiful output and easy-to-read syntax. It is ideal for quick, interactive API testing where readability matters most.

Choose curl for:

• Broad protocol support (FTP, SCP, SMTP, MQTT, WebSocket, etc.)
• Scripting and system-level tasks
• Performance-critical transfers
• HTTP/2 and HTTP/3 connections

Choose httpie for:

• Interactive API testing and debugging
• Human-readable output and simplicity
• Quick JSON-based requests
• Beginners learning HTTP concepts

Both tools are actively maintained and complement each other well. Many developers keep both installed and use each where it shines.

9. References#

• curl Official Documentation
• curl Man Page
• httpie Official Documentation
• "Everything curl" Book
• curl vs HTTPie — Daniel Stenberg's Comparison
• Stack Overflow: curl and httpie Tag

This blog provides a comprehensive guide to curl and httpie for data transfer with URLs. Experiment with both tools to find what works best for your workflow!