How to urlencode data for curl command?

I am trying to write a bash script for testing that takes a parameter and sends it through curl to web site. I need to url encode the value to make sure that special characters are processed properly. What is the best way to do this? Here is my basic script so far:

#!/bin/bash host=$ value=$2 shift shift curl -v -d "param=$" http://$/somepath $@ 

39 Answers 39

Use curl --data-urlencode ; from man curl :

This posts data, similar to the other --data options with the exception that this performs URL-encoding. To be CGI-compliant, the part should begin with a name followed by a separator and a content specification.

curl \ --data-urlencode "paramName=value" \ --data-urlencode "secondParam=value" \ http://example.com 

See the man page for more info.

This requires curl 7.18.0 or newer (released January 2008). Use curl -V to check which version you have.

You can as well encode the query string:

curl --get \ --data-urlencode "p1=value 1" \ --data-urlencode "p2=value 2" \ http://example.com # http://example.com?p1=value%201&p2=value%202 

Seems to only work for http POST. Documentation here: curl.haxx.se/docs/manpage.html#--data-urlencode

@StanJames If you use it like so curl can also do the encoding for a GET request. curl -G --data-urlencode "blah=df ssdf sdf" --data-urlencode "blah2=dfsdf sdfsd " http://whatever.com/whatever

@kberg actually, this will only work for query data. curl will append a '?' followed by the urlencoded params. If you want to urlencode some url postfix (such as a CouchDB GET for some document id), then '--data-urlencode' won't work.

I want to URL encode the URL path (which is used as a parameter in a REST API endpoint). There is no query string parameters involved. How do I do this for a GET request?

Another option is to use jq :

$ printf %s 'input text'|jq -sRr @uri input%20text $ jq -rn --arg x 'input text' '$x|@uri' input%20text 

-r ( --raw-output ) outputs the raw contents of strings instead of JSON string literals. -n ( --null-input ) doesn't read input from STDIN.

-R ( --raw-input ) treats input lines as strings instead of parsing them as JSON, and -sR ( --slurp --raw-input ) reads the input into a single string. You can replace -sRr with -Rr if your input only contains a single line or if you don't want to replace linefeeds with %0A :

$ printf %s\\n multiple\ lines of\ text|jq -Rr @uri multiple%20lines of%20text $ printf %s\\n multiple\ lines of\ text|jq -sRr @uri multiple%20lines%0Aof%20text%0A 

Or this percent-encodes all bytes:

xxd -p|tr -d \\n|sed 's/../%&/g' 

Commented Nov 16, 2017 at 16:16

for anyone wondering the same thing as me: @uri is not some variable, but a literal jq filter used for formatting strings and escaping; see jq manual for details (sorry, no direct link, need to search for @uri on the page. )

Note, this is not suitable for binary data. jq can only operate on UTF-8 strings, so binary data which is not invalid UTF will be munged into valid UTF-8 before encoding. To test this, run: printf '\xAB\xCD\xEF' | jq -sRr @uri . There is an open PR which may fix this: github.com/stedolan/jq/pull/2314

This really saves the day, thanks. I’d suggest using Bash’s Commented Mar 31, 2023 at 10:35

Here is the pure BASH answer.

Update: Since many changes have been discussed, I have placed this on https://github.com/sfinktah/bash/blob/master/rawurlencode.inc.sh for anybody to issue a PR against.

Note: This solution is not intended to encode unicode or multi-byte characters - which are quite outside BASH's humble native capabilities. It's only intended to encode symbols that would otherwise ruin argument passing in POST or GET requests, e.g. '&', '=' and so forth.

Very Important Note: DO NOT ATTEMPT TO WRITE YOUR OWN UNICODE CONVERSION FUNCTION, IN ANY LANGUAGE, EVER. See end of answer.

rawurlencode() < local string="$" local strlen=$ local encoded="" local pos c o for (( pos=0 ; pos case "$c" in [-_.~a-zA-Z0-9] ) o="$" ;; * ) printf -v o '%%%02x' "'$c" esac encoded+="$" done echo "$" # You can either set a return variable (FASTER) REPLY="$" #+or echo the result (EASIER). or both. :p > 

You can use it in two ways:

easier: echo http://url/q?=$( rawurlencode "$args" ) faster: rawurlencode "$args"; echo http://url/q?$

Here's the matching rawurldecode() function, which - with all modesty - is awesome.

# Returns a string in which the sequences with percent (%) signs followed by # two hex digits have been replaced with literal characters. rawurldecode() < # This is perhaps a risky gambit, but since all escape characters must be # encoded, we can replace %NN with \xNN and pass the lot to printf -b, which # will decode hex for us printf -v REPLY '%b' "$" # You can either set a return variable (FASTER) echo "$" #+or echo the result (EASIER). or both. :p > 

With the matching set, we can now perform some simple tests:

$ diff rawurlencode.inc.sh \ <( rawurldecode "$( rawurlencode "$( cat rawurlencode.inc.sh )" )" ) \ && echo Matched Output: Matched 

And if you really really feel that you need an external tool (well, it will go a lot faster, and might do binary files and such. ) I found this on my OpenWRT router.

replace_value=$(echo $replace_value | sed -f /usr/lib/ddns/url_escape.sed) 

Where url_escape.sed was a file that contained these rules:

# sed url escaping s:%:%25:g s: :%20:g s::%3E:g s:#:%23:g s::%7D:g s:|:%7C:g s:\\:%5C:g s:\^:%5E:g s:~:%7E:g s:\[:%5B:g s:\]:%5D:g s:`:%60:g s:;:%3B:g s:/:%2F:g s. %3F:g s^:^%3A^g s:@:%40:g s:=:%3D:g s:&:%26:g s:\$:%24:g s:\!:%21:g s:\*:%2A:g 

While it is not impossible to write such a script in BASH (probably using xxd and a very lengthy ruleset) capable of handing UTF-8 input, there are faster and more reliable ways. Attempting to decode UTF-8 into UTF-32 is a non-trivial task to do with accuracy, though very easy to do inaccurately such that you think it works until the day it doesn't.

Even the Unicode Consortium removed their sample code after discovering it was no longer 100% compatible with the actual standard.

The Unicode standard is constantly evolving, and has become extremely nuanced. Any implementation you can whip together will not be properly compliant, and if by some extreme effort you managed it, it wouldn't stay compliant.

Unfortunately, this script fails on some characters, such as 'é' and '½', outputting 'e%FFFFFFFFFFFFFFCC' and '%FFFFFFFFFFFFFFC2', respectively (b/c of the per-character loop, I believe).

In that first block of code what does the last parameter to printf mean? That is, why is it double-quote, single-quote, dollar-sign, letter-c, double-quote? Does does the single-quote do?

@ColinFraizer the single quote serves to convert the following character into its numeric value. ref. pubs.opengroup.org/onlinepubs/9699919799/utilities/…

@Matthematics, @dmcontador, @Orwellophile: I was wrong in my previous comment. Solution using xxd is beter and works in any case (for any character). I have updated my script. Anyway, it looks like the rawurldecode() function works exceptionally well. :)

Use Perl's URI::Escape module and uri_escape function in the second line of your bash script:

. value="$(perl -MURI::Escape -e 'print uri_escape($ARGV[0]);' "$2")" . 

Edit: Fix quoting problems, as suggested by Chris Johnsen in the comments. Thanks!

I fixed this (use echo , pipe and <> ), and now it works even when $2 contains an apostrophe or double-quotes. Thanks!

Chris Johnsen's version is better. I had $ in my test expression and using this via echo tripped up uri_escape / Perl variable expansion.

@jrw32982 yeah, looking back at it, having another language with which to accomplish this task is good. If I could, I'd take back my downvote, but alas it is currently locked in.

One of variants, may be ugly, but simple:

urlencode() < local data if [[ $# != 1 ]]; then echo "Usage: $0 string-to-urlencode" return 1 fi data="$(curl -s -o /dev/null -w %--get --data-urlencode "$1" "")" if [[ $? != 3 ]]; then echo "Unexpected error" 1>&2 return 2 fi echo "$" return 0 > 

Here is the one-liner version for example (as suggested by Bruno):

# Oneliner updated for curl 7.88.1 date | < curl -Gs -w %--data-urlencode @- ./ ||: > | sed "s/%0[aA]$//;s/^[^?]*?\(.*\)/\1/" # Verification that it works on input without the trailing \n printf "%s" "$(date)" | < curl -Gs -w %--data-urlencode @- ./ ||: > | sed "s/%0[aA]$//;s/^[^?]*?\(.*\)/\1/" # Explanation of what the oneliner is doing date `# 1. Generate sample input data ` \ | \ < `# groups a set of commands as a unit` \ curl -Gs -w %--data-urlencode @- ./ `# 2. @- means read stdin` \ ||: `# since the curl command exits 6, add "OR true"` \ > \ | sed \ -e "s/%0[aA]$//" `# strip trailing \n if present` \ -e "s/^[^?]*?\(.*\)/\1/" `# strip leading chars up to and including 1st ?` 

This is absolutely brilliant! I really wish you had left it a one line so that people can see how simple it really is. To URL encode the result of the date command… date | curl -Gso /dev/null -w % --data-urlencode @- "" | cut -c 3- (You have to cut the first 2 chars off, because curl's output is a technically a relative URL with a query string.)

@BrunoBronosky Your one-liner variant is good but seemingly adds a "%0A" to the end of the encoding. Users beware. The function version does not seem to have this issue.

for the sake of completeness, many solutions using sed or awk only translate a special set of characters and are hence quite large by code size and also dont translate other special characters that should be encoded.

a safe way to urlencode would be to just encode every single byte - even those that would've been allowed.

echo -ne 'some random\nbytes' | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g' 

xxd is taking care here that the input is handled as bytes and not characters.

xxd comes with the vim-common package in Debian and I was just on a system where it was not installed and I didnt want to install it. The altornative is to use hexdump from the bsdmainutils package in Debian. According to the following graph, bsdmainutils and vim-common should have an about equal likelihood to be installed:

but nevertheless here a version which uses hexdump instead of xxd and allows to avoid the tr call:

echo -ne 'some random\nbytes' | hexdump -v -e '/1 "%02x"' | sed 's/\(..\)/%\1/g' 

@qdii why? that would not only make it impossible to urlencode newlines but it would also wrongly insert newlines created by xxd into the output.

@josch. This is just plain wrong. First, any \n characters will be translated by xxd -plain into 0a . Don’t take my word for it, try it yourself: echo -n -e '\n' | xxd -plain This proves that your tr -d '\n' is useless here as there cannot be any \n after xxd -plain Second, echo foobar adds its own \n character in the end of the character string, so xxd -plain is not fed with foobar as expected but with foobar\n . then xxd -plain translates it into some character string that ends in 0a , making it unsuitable for the user. You could add -n to echo to solve it.

@qdii indeed -n was missing for echo but the xxd call belongs in front of the tr -d call. It belongs there so that any newline in foobar is translated by xxd . The tr -d after the xxd call is to remove the newlines that xxd produces. It seems you never have foobar long enough so that xxd produces newlines but for long inputs it will. So the tr -d is necessary. In contrast to your assumption the tr -d was NOT to remove newlines from the input but from the xxd output. I want to keep the newlines in the input. Your only valid point is, that echo adds an unnecessary newline.

@qdii and no offence taken - I just think that you are wrong, except for the echo -n which I was indeed missing

I find it more readable in python:

encoded_value=$(python3 -c "import urllib.parse; print urllib.parse.quote('''$value''')") 

the triple ' ensures that single quotes in value won't hurt. urllib is in the standard library. It work for example for this crazy (real world) url:

"http://www.rai.it/dl/audio/" "1264165523944Ho servito il re d'Inghilterra - Puntata 7 

I had some trouble with quotes and special chars with the triplequoting, this seemed to work for basically everything: encoded_value="$( echo -n "$" | python -c "import urllib; import sys; sys.stdout.write(urllib.quote(sys.stdin.read()))" )";

Python 3 version would be encoded_value=$(python3 -c "import urllib.parse; print (urllib.parse.quote('''$value'''))") .

It would be much safer to refer to sys.argv rather than substituting $value into a string later parsed as code. What if value contained ''' + __import__("os").system("rm -rf ~") + ''' ?

I've found the following snippet useful to stick it into a chain of program calls, where URI::Escape might not be installed:

perl -p -e 's/([^A-Za-z0-9])/sprintf("%%%02X", ord($1))/seg' 

worked for me. I changed it to perl -lpe . (the letter ell). This removed the trailing newline, which I needed for my purposes.

FYI, to do the inverse of this, use perl -pe 's/\%(\w\w)/chr hex $1/ge' (source: unix.stackexchange.com/questions/159253/…)

Depending on specifically which characters you need to encode, you can simplify this to perl -pe 's/(\W)/sprintf("%%%02X", ord($1))/ge' which allows letters, numbers, and underscores, but encodes everything else.

Thanks for response above! Since the use case is for curl: That is: : and / does not need encoding, my final function in my bashrc/zshrc is: perl -lpe 's/([^A-Za-z0-9.\/:])/sprintf("%%%02X", ord($1))/seg

If you wish to run GET request and use pure curl just add --get to @Jacob's solution.

Here is an example:

curl -v --get --data-urlencode "access_token=$(cat .fb_access_token)" https://graph.facebook.com/me/feed 

This may be the best one:

after=$(echo -e "$before" | od -An -tx1 | tr ' ' % | xargs printf "%s") 

This works for me with two additions: 1. replace the -e with -n to avoid adding a newline to the end of the argument and 2. add '%%' to the printf string to put a % in front of each pair of hex digits.

This does not work with OS X's od because it uses a different output format than GNU od . For example printf aa|od -An -tx1 -v|tr \ - prints -----------61--61-------------------------------------------------------- with OS X's od and -61-61 with GNU od . You could use od -An -tx1 -v|sed 's/ */ /g;s/ *$//'|tr \ %|tr -d \\n with either OS X's od or GNU od . xxd -p|sed 's/../%&/g'|tr -d \\n does the same thing, even though xxd is not in POSIX but od is.

Here's a Bash solution which doesn't invoke any external programs:

uriencode() < s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" s="$" printf %s "$s" > 

This behaves differently between the bash versions. On RHEL 6.9 the bash is 4.1.2 and it includes the single quotes. While Debian 9 and bash 4.4.12 is fine with the single quotes. For me removing the single quotes made it work on both. s="$"

Direct link to awk version : http://www.shelldorado.com/scripts/cmds/urlencode
I used it for years and it works like a charm

: ########################################################################## # Title : urlencode - encode URL data # Author : Heiner Steven ([email protected]) # Date : 2000-03-15 # Requires : awk # Categories : File Conversion, WWW, CGI # SCCS-Id. : @(#) urlencode 1.4 06/10/29 ########################################################################## # Description # Encode data according to # RFC 1738: "Uniform Resource Locators (URL)" and # RFC 1866: "Hypertext Markup Language - 2.0" (HTML) # # This encoding is used i.e. for the MIME type # "application/x-www-form-urlencoded" # # Notes # o The default behaviour is not to encode the line endings. This # may not be what was intended, because the result will be # multiple lines of output (which cannot be used in an URL or a # HTTP "POST" request). If the desired output should be one # line, use the "-l" option. # # o The "-l" option assumes, that the end-of-line is denoted by # the character LF (ASCII 10). This is not true for Windows or # Mac systems, where the end of a line is denoted by the two # characters CR LF (ASCII 13 10). # We use this for symmetry; data processed in the following way: # cat | urlencode -l | urldecode -l # should (and will) result in the original data # # o Large lines (or binary files) will break many AWK # implementations. If you get the message # awk: record `. ' too long # record number xxx # consider using GNU AWK (gawk). # # o urlencode will always terminate it's output with an EOL # character # # Thanks to Stefan Brozinski for pointing out a bug related to non-standard # locales. # # See also # urldecode ########################################################################## PN=`basename "$0"` # Program name VER='1.4' : $ Usage () < echo >&2 "$PN - encode URL data, $VER usage: $PN [-l] [file . ] -l: encode line endings (result will be one line of output) The default is to encode each input line on its own." exit 1 > Msg () < for MsgLine do echo "$PN: $MsgLine" >&2 done > Fatal () < Msg "$@"; exit 1; >set -- `getopt hl "$@" 2>/dev/null` || Usage [ $# -lt 1 ] && Usage # "getopt" detected an error EncodeEOL=no while [ $# -gt 0 ] do case "$1" in -l) EncodeEOL=yes;; --) shift; break;; -h) Usage;; -*) Usage;; *) break;; # First file name esac shift done LANG=C export LANG $AWK ' BEGIN < # We assume an awk implementation that is just plain dumb. # We will convert an character to its ASCII value with the # table ord[], and produce two-digit hexadecimal output # without the printf("%02X") feature. EOL = "%0A" # "end of line" string (encoded) split ("1 2 3 4 5 6 7 8 9 A B C D E F", hextab, " ") hextab [0] = 0 for ( i=1; i < encoded = "" for ( i=1; ielse if ( c == " " ) < encoded = encoded "+" # special handling >else < # unsafe character, encode it as a two-digit hex-number lo = ord [c] % 16 hi = int (ord [c] / 16); encoded = encoded "%" hextab [hi] hextab [lo] >> if ( EncodeEOL ) < printf ("%s", encoded EOL) >else < print encoded >> END < #if ( EncodeEOL ) print "" >' "$@"