Skip to main content

4. Curl basics

About 7 minCLILinuxcrashcourseclishshellgnulinuxawk

4. Curl basics 관련

4. Curl basics
Mastering curl - interactive text guide

Now that we understand how curl handles URLs and options, let's talk about specific features.

Version

--version (-V) knows everything about the installed version of curl:

curl -V
# ➊ curl 8.2.1 (x86_64-alpine-linux-musl) libcurl/8.2.1 OpenSSL/3.1.2 zlib/1.2.13 brotli/1.0.9 libidn2/2.3.4 nghttp2/1.55.1
# ➋ Release-Date: 2023-07-26
# ➌ Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
# ➍ Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets

It lists (line by line)

  • ➊ versions of curl itself and its dependencies,
  • ➋ the release date,
  • ➌ available protocols, and
  • ➍ enabled features.

Verbose

verbose

--verbose (-v) makes curl verbose, which is useful for debugging:

curl -v http://httpbin/uuid
# {
#   "uuid": "dfc032c7-7452-4b21-aedb-3d80787e9946"
# }
# 
# * processing: http://httpbin/uuid
# *   Trying 172.19.0.2:80...
# * Connected to httpbin (172.19.0.2) port 80
# > GET /uuid HTTP/1.1
# > Host: httpbin
# > User-Agent: curl/8.2.1
# > Accept: */*
# > 
# < HTTP/1.1 200 OK
# < Access-Control-Allow-Credentials: true
# < Access-Control-Allow-Origin: *
# < Content-Type: application/json; charset=utf-8
# < Date: Fri, 15 Sep 2023 06:35:21 GMT
# < Content-Length: 53
# < 
# { [53 bytes data]
# * Connection #0 to host httpbin left intact

Use --write-out (-w) to extract specific information about the response. It supports over 50 variables. For example, here we extract the status code and response content type:

Case 1
curl
  -w "\nstatus: %{response_code}\ntype: %{content_type}"
  http://httpbin/status/429
#
# status: 429
# type: text/plain; charset=utf-8

Downloads

--remote-name (-O) tells curl to save the output to a file specified by the URL (specifically, by the part after the last /). It's often used together with --output-dir, which tells curl where exactly to save the file:

curl --output-dir /tmp -O http://httpbin/uuid

&& cat /tmp/uuid
# {
#   "uuid": "d07da12c-83e1-4b30-8840-059a6dcf1666"
# }

If the directory does not exist, --output-dir won't create it for you. Use --create-dirs for this:

curl --output-dir /tmp/some/place --create-dirs
  -O http://httpbin/uuid

&& cat /tmp/some/place/uuid
# {
#   "uuid": "e4e79cd5-053c-4d0c-834e-d9bddd7d00f2"
# }

You can use --max-filesize (in bytes) to limit the allowed response size, but often it isn't known in advance, so it may not work.

Retry

Sometimes the remote host is temporarily unavailable. To deal with these situations, curl provides the --retry [num] option. If a request fails, curl will try it again, but no more than num times:

curl -i --retry 3 http://httpbin/unstable
# HTTP/1.1 200 OK
# Access-Control-Allow-Credentials: true
# Access-Control-Allow-Origin: *
# Content-Type: text/plain; charset=utf-8
# Date: Fri, 15 Sep 2023 06:39:56 GMT
# Content-Length: 0

(this URL fails 50% of the time)

You can set the maximum time curl will spend retrying with --retry-max-time (in seconds) or the delay between retries with --retry-delay (also in seconds):

curl -i --retry 3 http://httpbin/unstable
  --retry-max-time 2
  --retry-delay 1
# HTTP/1.1 200 OK
# Access-Control-Allow-Credentials: true
# Access-Control-Allow-Origin: *
# Content-Type: text/plain; charset=utf-8
# Date: Fri, 15 Sep 2023 06:40:27 GMT
# Content-Length: 0

For curl, "request failed" means one of the following HTTP codes: 408, 429, 500, 502, 503 or 504. If the request fails with a "connection refused" error, curl will not retry. But you can change this with --retry-connrefused, or even enable retries for all kinds of problems with --retry-all-errors.

Uploads

Curl is often used to download data from the server, but you can also upload it. Use the --upload-file (-T) option:

echo hello > /tmp/hello.txt &&

curl -T /tmp/hello.txt http://httpbin/put
# {
#   "args": {},
#   "headers": {
#     "Accept": [
#       "*/*"
#     ],
#     "Content-Length": [
#       "6"
#     ],
#     "Host": [
#       "httpbin"
#     ],
#     "User-Agent": [
#       "curl/8.2.1"
#     ]
#   },
#   "method": "PUT",
#   "origin": "172.19.0.3:44058",
#   "url": "http://httpbin/put",
#   "data": "data:application/octet-stream;base64,aGVsbG8K",
#   "files": {},
#   "form": {},
#   "json": null
# }

For HTTP uploads, curl uses the PUT method.

Transfer controls

To stop slow transfers, set the minimum allowed download speed (in bytes per second) with --speed-limit. By default, curl checks the speed in 30 seconds intervals, but you can change this with --speed-time.

For example, allow no less than 10 bytes/sec during a 3-second interval:

curl -v --speed-limit 10 --speed-time 3 http://httpbin/get
# {
#   "args": {},
#   "headers": {
#     "Accept": [
#       "*/*"
#     ],
#     "Host": [
#       "httpbin"
#     ],
#     "User-Agent": [
#       "curl/8.2.1"
#     ]
#   },
#   "method": "GET",
#   "origin": "172.19.0.3:58696",
#   "url": "http://httpbin/get"
# }
# 
# * processing: http://httpbin/get
# *   Trying 172.19.0.2:80...
# * Connected to httpbin (172.19.0.2) port 80
# > GET /get HTTP/1.1
# > Host: httpbin
# > User-Agent: curl/8.2.1
# > Accept: */*
# > 
# < HTTP/1.1 200 OK
# < Access-Control-Allow-Credentials: true
# < Access-Control-Allow-Origin: *
# < Content-Type: application/json; charset=utf-8
# < Date: Fri, 15 Sep 2023 06:41:38 GMT
# < Content-Length: 236
# < 
# { [236 bytes data]
# * Connection #0 to host httpbin left intact

To limit bandwidth usage, set --limit-rate. It accepts anything from bytes to petabytes:

curl --limit-rate 3 http://httpbin/get
curl --limit-rate 3k http://httpbin/get
curl --limit-rate 3m http://httpbin/get
curl --limit-rate 3g http://httpbin/get
curl --limit-rate 3t http://httpbin/get
curl --limit-rate 3p http://httpbin/get

Another thing to limit is the number of concurrent requests (e.g. if you download a lot of files). Use --rate for this. It accepts seconds, minutes, hours or days:

curl --rate 3/s http://httpbin/anything/[1-9].txt
curl --rate 3/m http://httpbin/anything/[1-9].txt
curl --rate 3/h http://httpbin/anything/[1-9].txt
curl --rate 3/d http://httpbin/anything/[1-9].txt

Name resolving

By default, curl uses your DNS server to resolve hostnames to IP addresses. But you can force it to resolve to a specific IP with --resolve:

❌

(this one fails because no one is listening on 127.0.0.1)

curl --resolve httpbingo.org:8080:127.0.0.1
  http://httpbingo.org:8080/get
# curl: (7) Failed to connect to httpbingo.org port 8080 after 0 ms: Couldn't connect to server
# (exit status 7)

Connections

There are also some network connection-level settings.

--interface

Use a specific network interface

curl --interactive enp3s0 https://example.com

Timeouts

To limit the maximum amount of time curl will spend interacting with a single URL, use --max-time (in fractional seconds):

❌

(this one fails)

curl --max-time 0.5 http:/httpbin/delay/1
# curl: (28) Operation timed out after 500 milliseconds with 0 bytes received
#  (exit status 28)

Credentials

You almost never want to pass the username and password in the curl command itself. One way to avoid this is to use the .netrc file. It specifies hostnames and credentials for accessing them:

machine httpbin
login alice
password cheese

machine example.com
login bob
password nuggets

Pass the --netrc option to use the $HOME/.netrc file, or --netrc-file to use a specific one:

echo -e "machine httpbin\nlogin alice\npassword cheese" > /tmp/netrc &&

curl --netrc-file /tmp/netrc
  http://httpbin/basic-auth/alice/cheese
# {
#   "authorized": true,
#   "user": "alice"
# }

Exit status

When curl exits, it returns a numeric value to the shell. For success, it's 0, and for errors, there are about 100 different valuesopen in new window.

For example, here is an exit status 7 (failed to connect to host):

curl http://httpbin:1313/get
# curl: (7) Failed to connect to httpbin port 1313 after 1 ms: Couldn't connect to server
#  (exit status 7)

You can access the exit status through the $? shell variable.

이찬희 (MarkiiimarK)
Never Stop Learning.