URL Encoding and Percent-Encoding

How characters that cannot appear in a URL are written as a percent sign and two hex digits, and where each rule applies.

Why URLs need encoding

A URL may only contain a small set of ASCII characters, and several of those characters already have a job. The slash separates path segments, the question mark starts the query, the ampersand separates query parameters and the hash starts the fragment. If a piece of data you want to put in a URL contains one of those characters, or a space, or a letter outside ASCII, it has to be written in a form that cannot be mistaken for structure.

That form is percent-encoding. Each byte that needs escaping is written as a percent sign followed by two hexadecimal digits giving the byte value. A space is byte 0x20, so it becomes %20. A percent sign itself must also be escaped, as %25, otherwise a decoder would read it as the start of an escape. If two-digit hex values are unfamiliar, the page on number bases covers them.

Reserved and unreserved characters

RFC 3986, the standard that defines URI syntax, splits the ASCII printable characters into groups.

  • Unreserved: letters A-Z and a-z, digits 0-9, and the four marks - . _ ~. These never need encoding, and encoding them gains nothing. A correct normaliser turns %41 back into A.
  • Reserved, general delimiters: : / ? # [ ] @. These mark the main parts of a URL.
  • Reserved, sub-delimiters: ! $ & ' ( ) * + , ; =. Schemes and applications use these inside components, such as & and = in a query string.

Everything else, including space, double quote, <, > and every non-ASCII character, must always be percent-encoded. A reserved character must be encoded when it is part of your data rather than part of the URL structure. The standard recommends uppercase hex digits, so %2F is preferred to %2f, though decoders accept both.

Common encodings

These values come from Python's urllib.parse.quote(c, safe=''), which escapes every reserved character.

CharacterEncoded
space%20
!%21
#%23
$%24
%%25
&%26
'%27
(%28
)%29
*%2A
+%2B
,%2C
/%2F
:%3A
;%3B
=%3D
?%3F
@%40
[%5B
]%5D
double quote%22

Worked examples

Python offers two encoders. quote follows URL path rules: it encodes a space as %20 and, by default, leaves / alone. quote_plus follows HTML form rules: it writes a space as + and encodes the slash. Passing safe='' to quote makes it encode the slash too.

Inputquote(s)quote(s, safe='')quote_plus(s)
hello worldhello%20worldhello%20worldhello+world
a&b=ca%26b%3Dca%26b%3Dca%26b%3Dc
50%50%2550%2550%25
cafécaf%C3%A9caf%C3%A9caf%C3%A9
€%E2%82%AC%E2%82%AC%E2%82%AC
/path/to file/path/to%20file%2Fpath%2Fto%20file%2Fpath%2Fto+file

Non-ASCII text is first converted to bytes, normally with UTF-8, and then each byte is escaped. The letter é is two bytes in UTF-8, C3 A9, so it becomes %C3%A9. The euro sign is three bytes, E2 82 AC, giving %E2%82%AC. One visible character can therefore turn into six or nine characters in the URL. The page on character encoding explains why those characters take two and three bytes.

Space: %20 or +?

The plus sign as a space is not part of RFC 3986. It comes from the application/x-www-form-urlencoded format that browsers use when they submit an HTML form. Inside a query string, most servers decode + as a space because they expect form data. Inside a path, a + is just a plus sign.

The safe rule: use %20 anywhere, and only produce + when you are deliberately building form-style query strings. If your data contains a real plus sign, it must be sent as %2B in a query string, or the server will see a space. For example, urlencode turns the parameters q=rock & roll and lang=fr-CA into:

q=rock+%26+roll&lang=fr-CA

The ampersand inside the value became %26, while the ampersand that separates the two parameters stayed literal. That distinction is the whole reason encoding exists.

Query string vs path

Each part of a URL has its own set of characters that are allowed literally.

  • In a path segment, a / inside your data must be encoded as %2F, or it will be read as a new segment. Some servers decode %2F early or reject it, so avoid slashes in identifiers when you can.
  • In a query value, &, =, + and # must be encoded. A slash or question mark is allowed literally there, although encoding it does no harm.
  • In a fragment, the browser never sends it to the server, but the same escaping applies for characters outside the allowed set.

Encode each component separately, then join the parts. Never take a finished URL and run it through an encoder as a whole: that escapes the delimiters you need and corrupts the structure.

Mistakes that break URLs

Double encoding. If a value is encoded twice, the percent signs from the first pass are themselves escaped. Encoding a b once gives a%20b; encoding that result again gives a%2520b. A server decodes only once, so it sees the literal text a%20b rather than a space. The usual cause is a library that encodes for you while you also encode by hand. Pick one layer to own the encoding.

Decoding twice. The reverse fault is just as harmful and is a classic security hole: a value like %252e%252e%252f passes a check for ../ after one decode and turns into ../ after the second.

Encoding with the wrong character set. Older systems sometimes encode é as the single Latin-1 byte %E9. A UTF-8 decoder will not accept that byte on its own. Use UTF-8 unless you have a specific reason not to.

Forgetting the plus sign. A phone number such as +44 in a query string arrives as 44 with a leading space unless you encode it as %2B44.