UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 12: ordinal not in range(128)

19 Jul 2013

The error in the title will occur on pre-0.12.0 versions of python’s Requests library. It seems to be fixed on later versions and maybe even earlier.

Here’s an example of how you save a zip file from behind a corporate proxy with python’s Requests library:

	sess = requests.session()
	# you may have other get/post requests before you can get to the download page for the file
	url = 'https://example.com/report.zip'
	# this may be different for you, i just copied this from an existing request (using fiddler)
	file_headers = {
	'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
	'Accept-Language': 'en-US,en;q=0.8',
	'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)',
	'Host': 'example.com',
	'Accept-Encoding': 'gzip,deflate,sdch',
	'Connection': 'Keep-Alive',
	'Content-Type': 'application/octet-stream;charset=text/html;charset=UTF-8',
	}
	with open('report.zip', mode='wb') as handle:
	report_file = sess.get(url, headers=file_headers, prefetch=False, verify=False)
	handle.write(report_file.content)

view raw gistfile1.py hosted with ❤ by GitHub

That’s it! Just use the write function with the content of the returned request.