Writing Python 2-3 compatible code
-
Copyright (c): 2013-2024 Python Charmers, Australia.
-
Author: Ed Schofield.
-
Licence: Creative Commons Attribution.
A PDF version is here: https://python-future.org/compatible_idioms.pdf
This notebook shows you idioms for writing future-proof code that is compatible with both versions of Python: 2 and 3. It accompanies Ed Schofield’s talk at PyCon AU 2014, “Writing 2/3 compatible code”. (The video is here: https://www.youtube.com/watch?v=KOqk8j11aAI&t=10m14s.)
Minimum versions:
-
Python 2: 2.7+
-
Python 3: 3.4+
The imports below refer to these pip-installable packages on PyPI:
import future # pip install future import builtins # pip install future import past # pip install future import six # pip install six
The following scripts are also pip-installable:
futurize # pip install future pasteurize # pip install future
See https://python-future.org and https://pythonhosted.org/six/ for more information.
# Python 2 only: print 'Hello'
# Python 2 and 3: print('Hello')
To print multiple strings, import print_function to prevent Py2 from interpreting it as a tuple:
# Python 2 only: print 'Hello', 'Guido'
# Python 2 and 3: from __future__ import print_function # (at top of module) print('Hello', 'Guido')
# Python 2 only: print >> sys.stderr, 'Hello'
# Python 2 and 3: from __future__ import print_function print('Hello', file=sys.stderr)
# Python 2 only: print 'Hello',
# Python 2 and 3: from __future__ import print_function print('Hello', end='')
# Python 2 only: raise ValueError, "dodgy value"
# Python 2 and 3: raise ValueError("dodgy value")
Raising exceptions with a traceback:
# Python 2 only: traceback = sys.exc_info()[2] raise ValueError, "dodgy value", traceback
# Python 3 only: raise ValueError("dodgy value").with_traceback()
# Python 2 and 3: option 1 from six import reraise as raise_ # or from future.utils import raise_ traceback = sys.exc_info()[2] raise_(ValueError, "dodgy value", traceback)
# Python 2 and 3: option 2 from future.utils import raise_with_traceback raise_with_traceback(ValueError("dodgy value"))
Exception chaining (PEP 3134):
# Setup: class DatabaseError(Exception): pass
# Python 3 only class FileDatabase: def __init__(self, filename): try: self.file = open(filename) except IOError as exc: raise DatabaseError('failed to open') from exc
# Python 2 and 3: from future.utils import raise_from class FileDatabase: def __init__(self, filename): try: self.file = open(filename) except IOError as exc: raise_from(DatabaseError('failed to open'), exc)
# Testing the above: try: fd = FileDatabase('non_existent_file.txt') except Exception as e: assert isinstance(e.__cause__, IOError) # FileNotFoundError on Py3.3+ inherits from IOError
# Python 2 only: try: ... except ValueError, e: ...
# Python 2 and 3: try: ... except ValueError as e: ...
Integer division (rounding down):
# Python 2 only: assert 2 / 3 == 0
# Python 2 and 3: assert 2 // 3 == 0
“True division” (float division):
# Python 3 only: assert 3 / 2 == 1.5
# Python 2 and 3: from __future__ import division # (at top of module) assert 3 / 2 == 1.5
“Old division” (i.e. compatible with Py2 behaviour):
# Python 2 only: a = b / c # with any types
# Python 2 and 3: from past.utils import old_div a = old_div(b, c) # always same as / on Py2
Short integers are gone in Python 3 and long has become int (without the trailing L in the repr).
# Python 2 only k = 9223372036854775808L # Python 2 and 3: k = 9223372036854775808
# Python 2 only bigint = 1L # Python 2 and 3 from builtins import int bigint = int(1)
To test whether a value is an integer (of any kind):
# Python 2 only: if isinstance(x, (int, long)): ... # Python 3 only: if isinstance(x, int): ... # Python 2 and 3: option 1 from builtins import int # subclass of long on Py2 if isinstance(x, int): # matches both int and long on Py2 ... # Python 2 and 3: option 2 from past.builtins import long if isinstance(x, (int, long)): ...
If you are upgrading an existing Python 2 codebase, it may be preferable to mark up all string literals as unicode explicitly with u prefixes:
# Python 2 only s1 = 'The Zen of Python' s2 = u'きたないのよりきれいな方がいい\n' # Python 2 and 3 s1 = u'The Zen of Python' s2 = u'きたないのよりきれいな方がいい\n'
The futurize and python-modernize tools do not currently offer an option to do this automatically.
If you are writing code for a new project or new codebase, you can use this idiom to make all string literals in a module unicode strings:
# Python 2 and 3 from __future__ import unicode_literals # at top of module s1 = 'The Zen of Python' s2 = 'きたないのよりきれいな方がいい\n'
See https://python-future.org/unicode_literals.html for more discussion on which style to use.
# Python 2 only s = 'This must be a byte-string' # Python 2 and 3 s = b'This must be a byte-string'
To loop over a byte-string with possible high-bit characters, obtaining each character as a byte-string of length 1:
# Python 2 only: for bytechar in 'byte-string with high-bit chars like \xf9': ... # Python 3 only: for myint in b'byte-string with high-bit chars like \xf9': bytechar = bytes([myint]) # Python 2 and 3: from builtins import bytes for myint in bytes(b'byte-string with high-bit chars like \xf9'): bytechar = bytes([myint])
As an alternative, chr() and .encode('latin-1') can be used to convert an int into a 1-char byte string:
# Python 3 only: for myint in b'byte-string with high-bit chars like \xf9': char = chr(myint) # returns a unicode string bytechar = char.encode('latin-1') # Python 2 and 3: from builtins import bytes, chr for myint in bytes(b'byte-string with high-bit chars like \xf9'): char = chr(myint) # returns a unicode string bytechar = char.encode('latin-1') # forces returning a byte str
# Python 2 only: a = u'abc' b = 'def' assert (isinstance(a, basestring) and isinstance(b, basestring)) # Python 2 and 3: alternative 1 from past.builtins import basestring # pip install future a = u'abc' b = b'def' assert (isinstance(a, basestring) and isinstance(b, basestring))
# Python 2 and 3: alternative 2: refactor the code to avoid considering # byte-strings as strings. from builtins import str a = u'abc' b = b'def' c = b.decode() assert isinstance(a, str) and isinstance(c, str) # ...
# Python 2 only: templates = [u"blog/blog_post_detail_%s.html" % unicode(slug)]
# Python 2 and 3: alternative 1 from builtins import str templates = [u"blog/blog_post_detail_%s.html" % str(slug)]
# Python 2 and 3: alternative 2 from builtins import str as text templates = [u"blog/blog_post_detail_%s.html" % text(slug)]
# Python 2 only: from StringIO import StringIO # or: from cStringIO import StringIO # Python 2 and 3: from io import BytesIO # for handling byte strings from io import StringIO # for handling unicode strings
Suppose the package is:
mypackage/ __init__.py submodule1.py submodule2.py
and the code below is in submodule1.py:
# Python 2 only: import submodule2
# Python 2 and 3: from . import submodule2
# Python 2 and 3: # To make Py2 code safer (more like Py3) by preventing # implicit relative imports, you can also add this to the top: from __future__ import absolute_import
heights = {'Fred': 175, 'Anne': 166, 'Joe': 192}
Iterable dict keys:
# Python 2 only: for key in heights.iterkeys(): ...
# Python 2 and 3: for key in heights: ...
Iterable dict values:
# Python 2 only: for value in heights.itervalues(): ...
# Idiomatic Python 3 for value in heights.values(): # extra memory overhead on Py2 ...
# Python 2 and 3: option 1 from builtins import dict heights = dict(Fred=175, Anne=166, Joe=192) for key in heights.values(): # efficient on Py2 and Py3 ...
# Python 2 and 3: option 2 from future.utils import itervalues # or from six import itervalues for key in itervalues(heights): ...
Iterable dict items:
# Python 2 only: for (key, value) in heights.iteritems(): ...
# Python 2 and 3: option 1 for (key, value) in heights.items(): # inefficient on Py2 ...
# Python 2 and 3: option 2 from future.utils import viewitems for (key, value) in viewitems(heights): # also behaves like a set ...
# Python 2 and 3: option 3 from future.utils import iteritems # or from six import iteritems for (key, value) in iteritems(heights): ...
dict keys as a list:
# Python 2 only: keylist = heights.keys() assert isinstance(keylist, list)
# Python 2 and 3: keylist = list(heights) assert isinstance(keylist, list)
dict values as a list:
# Python 2 only: heights = {'Fred': 175, 'Anne': 166, 'Joe': 192} valuelist = heights.values() assert isinstance(valuelist, list)
# Python 2 and 3: option 1 valuelist = list(heights.values()) # inefficient on Py2
# Python 2 and 3: option 2 from builtins import dict heights = dict(Fred=175, Anne=166, Joe=192) valuelist = list(heights.values())
# Python 2 and 3: option 3 from future.utils import listvalues valuelist = listvalues(heights)
# Python 2 and 3: option 4 from future.utils import itervalues # or from six import itervalues valuelist = list(itervalues(heights))
dict items as a list:
# Python 2 and 3: option 1 itemlist = list(heights.items()) # inefficient on Py2
# Python 2 and 3: option 2 from future.utils import listitems itemlist = listitems(heights)
# Python 2 and 3: option 3 from future.utils import iteritems # or from six import iteritems itemlist = list(iteritems(heights))
# Python 2 only class Upper(object): def __init__(self, iterable): self._iter = iter(iterable) def next(self): # Py2-style return self._iter.next().upper() def __iter__(self): return self itr = Upper('hello') assert itr.next() == 'H' # Py2-style assert list(itr) == list('ELLO')
# Python 2 and 3: option 1 from builtins import object class Upper(object): def __init__(self, iterable): self._iter = iter(iterable) def __next__(self): # Py3-style iterator interface return next(self._iter).upper() # builtin next() function calls def __iter__(self): return self itr = Upper('hello') assert next(itr) == 'H' # compatible style assert list(itr) == list('ELLO')
# Python 2 and 3: option 2 from future.utils import implements_iterator @implements_iterator class Upper(object): def __init__(self, iterable): self._iter = iter(iterable) def __next__(self): # Py3-style iterator interface return next(self._iter).upper() # builtin next() function calls def __iter__(self): return self itr = Upper('hello') assert next(itr) == 'H' assert list(itr) == list('ELLO')
# Python 2 only: class MyClass(object): def __unicode__(self): return 'Unicode string: \u5b54\u5b50' def __str__(self): return unicode(self).encode('utf-8') a = MyClass() print(a) # prints encoded string
# Python 2 and 3: from future.utils import python_2_unicode_compatible @python_2_unicode_compatible class MyClass(object): def __str__(self): return u'Unicode string: \u5b54\u5b50' a = MyClass() print(a) # prints string encoded as utf-8 on Py2
# Python 2 only: class AllOrNothing(object): def __init__(self, l): self.l = l def __nonzero__(self): return all(self.l) container = AllOrNothing([0, 100, 200]) assert not bool(container)
# Python 2 and 3: from builtins import object class AllOrNothing(object): def __init__(self, l): self.l = l def __bool__(self): return all(self.l) container = AllOrNothing([0, 100, 200]) assert not bool(container)
# Python 2 only: for i in xrange(10**8): ...
# Python 2 and 3: forward-compatible from builtins import range for i in range(10**8): ...
# Python 2 and 3: backward-compatible from past.builtins import xrange for i in xrange(10**8): ...
# Python 2 only mylist = range(5) assert mylist == [0, 1, 2, 3, 4]
# Python 2 and 3: forward-compatible: option 1 mylist = list(range(5)) # copies memory on Py2 assert mylist == [0, 1, 2, 3, 4]
# Python 2 and 3: forward-compatible: option 2 from builtins import range mylist = list(range(5)) assert mylist == [0, 1, 2, 3, 4]
# Python 2 and 3: option 3 from future.utils import lrange mylist = lrange(5) assert mylist == [0, 1, 2, 3, 4]
# Python 2 and 3: backward compatible from past.builtins import range mylist = range(5) assert mylist == [0, 1, 2, 3, 4]
# Python 2 only: mynewlist = map(f, myoldlist) assert mynewlist == [f(x) for x in myoldlist]
# Python 2 and 3: option 1 # Idiomatic Py3, but inefficient on Py2 mynewlist = list(map(f, myoldlist)) assert mynewlist == [f(x) for x in myoldlist]
# Python 2 and 3: option 2 from builtins import map mynewlist = list(map(f, myoldlist)) assert mynewlist == [f(x) for x in myoldlist]
# Python 2 and 3: option 3 try: import itertools.imap as map except ImportError: pass mynewlist = list(map(f, myoldlist)) # inefficient on Py2 assert mynewlist == [f(x) for x in myoldlist]
# Python 2 and 3: option 4 from future.utils import lmap mynewlist = lmap(f, myoldlist) assert mynewlist == [f(x) for x in myoldlist]
# Python 2 and 3: option 5 from past.builtins import map mynewlist = map(f, myoldlist) assert mynewlist == [f(x) for x in myoldlist]
# Python 2 only: from itertools import imap myiter = imap(func, myoldlist) assert isinstance(myiter, iter)
# Python 3 only: myiter = map(func, myoldlist) assert isinstance(myiter, iter)
# Python 2 and 3: option 1 from builtins import map myiter = map(func, myoldlist) assert isinstance(myiter, iter)
# Python 2 and 3: option 2 try: import itertools.imap as map except ImportError: pass myiter = map(func, myoldlist) assert isinstance(myiter, iter)
# Python 2 and 3: option 3 from six.moves import map myiter = map(func, myoldlist) assert isinstance(myiter, iter)
As above with zip and itertools.izip.
As above with filter and itertools.ifilter too.
# Python 2 only f = open('myfile.txt') data = f.read() # as a byte string text = data.decode('utf-8') # Python 2 and 3: alternative 1 from io import open f = open('myfile.txt', 'rb') data = f.read() # as bytes text = data.decode('utf-8') # unicode, not bytes # Python 2 and 3: alternative 2 from io import open f = open('myfile.txt', encoding='utf-8') text = f.read() # unicode, not bytes
# Python 2 only: assert reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) == 1+2+3+4+5
# Python 2 and 3: from functools import reduce assert reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) == 1+2+3+4+5
# Python 2 only: name = raw_input('What is your name? ') assert isinstance(name, str) # native str
# Python 2 and 3: from builtins import input name = input('What is your name? ') assert isinstance(name, str) # native str on Py2 and Py3
# Python 2 only: input("Type something safe please: ")
# Python 2 and 3 from builtins import input eval(input("Type something safe please: "))
Warning: using either of these is unsafe with untrusted input.
# Python 2 only: f = file(pathname)
# Python 2 and 3: f = open(pathname) # But preferably, use this: from io import open f = open(pathname, 'rb') # if f.read() should return bytes # or f = open(pathname, 'rt') # if f.read() should return unicode text
# Python 2 only: exec 'x = 10' # Python 2 and 3: exec('x = 10')
# Python 2 only: g = globals() exec 'x = 10' in g # Python 2 and 3: g = globals() exec('x = 10', g)
# Python 2 only: l = locals() exec 'x = 10' in g, l # Python 2 and 3: exec('x = 10', g, l)
# Python 2 only: execfile('myfile.py')
# Python 2 and 3: alternative 1 from past.builtins import execfile execfile('myfile.py')
# Python 2 and 3: alternative 2 exec(compile(open('myfile.py').read())) # This can sometimes cause this: # SyntaxError: function ... uses import * and bare exec ... # See https://github.com/PythonCharmers/python-future/issues/37
# Python 2 only: assert unichr(8364) == '€'
# Python 3 only: assert chr(8364) == '€'
# Python 2 and 3: from builtins import chr assert chr(8364) == '€'
# Python 2 only: intern('mystring')
# Python 3 only: from sys import intern intern('mystring')
# Python 2 and 3: alternative 1 from past.builtins import intern intern('mystring')
# Python 2 and 3: alternative 2 from six.moves import intern intern('mystring')
# Python 2 and 3: alternative 3 from future.standard_library import install_aliases install_aliases() from sys import intern intern('mystring')
# Python 2 and 3: alternative 2 try: from sys import intern except ImportError: pass intern('mystring')
args = ('a', 'b') kwargs = {'kwarg1': True}
# Python 2 only: apply(f, args, kwargs)
# Python 2 and 3: alternative 1 f(*args, **kwargs)
# Python 2 and 3: alternative 2 from past.builtins import apply apply(f, args, kwargs)
# Python 2 only: assert chr(64) == b'@' assert chr(200) == b'\xc8'
# Python 3 only: option 1 assert chr(64).encode('latin-1') == b'@' assert chr(0xc8).encode('latin-1') == b'\xc8'
# Python 2 and 3: option 1 from builtins import chr assert chr(64).encode('latin-1') == b'@' assert chr(0xc8).encode('latin-1') == b'\xc8'
# Python 3 only: option 2 assert bytes([64]) == b'@' assert bytes([0xc8]) == b'\xc8'
# Python 2 and 3: option 2 from builtins import bytes assert bytes([64]) == b'@' assert bytes([0xc8]) == b'\xc8'
# Python 2 only: assert cmp('a', 'b') < 0 and cmp('b', 'a') > 0 and cmp('c', 'c') == 0
# Python 2 and 3: alternative 1 from past.builtins import cmp assert cmp('a', 'b') < 0 and cmp('b', 'a') > 0 and cmp('c', 'c') == 0
# Python 2 and 3: alternative 2 cmp = lambda(x, y): (x > y) - (x < y) assert cmp('a', 'b') < 0 and cmp('b', 'a') > 0 and cmp('c', 'c') == 0
# Python 2 only: reload(mymodule)
# Python 2 and 3 from imp import reload reload(mymodule)
# Python 2 only import anydbm import whichdb import dbm import dumbdbm import gdbm # Python 2 and 3: alternative 1 from future import standard_library standard_library.install_aliases() import dbm import dbm.ndbm import dbm.dumb import dbm.gnu # Python 2 and 3: alternative 2 from future.moves import dbm from future.moves.dbm import dumb from future.moves.dbm import ndbm from future.moves.dbm import gnu # Python 2 and 3: alternative 3 from six.moves import dbm_gnu # (others not supported)
# Python 2 only from commands import getoutput, getstatusoutput # Python 2 and 3 from future import standard_library standard_library.install_aliases() from subprocess import getoutput, getstatusoutput
# Python 2 only from StringIO import StringIO from cStringIO import StringIO
# Python 2 and 3 from io import BytesIO # and refactor StringIO() calls to BytesIO() if passing byte-strings
# Python 2 only: import httplib import Cookie import cookielib import BaseHTTPServer import SimpleHTTPServer import CGIHttpServer # Python 2 and 3 (after ``pip install future``): import http.client import http.cookies import http.cookiejar import http.server
# Python 2 only: import DocXMLRPCServer import SimpleXMLRPCServer # Python 2 and 3 (after ``pip install future``): import xmlrpc.server
# Python 2 only: import xmlrpclib # Python 2 and 3 (after ``pip install future``): import xmlrpc.client
# Python 2 and 3: from cgi import escape # Safer (Python 2 and 3, after ``pip install future``): from html import escape # Python 2 only: from htmlentitydefs import codepoint2name, entitydefs, name2codepoint # Python 2 and 3 (after ``pip install future``): from html.entities import codepoint2name, entitydefs, name2codepoint
# Python 2 only: from HTMLParser import HTMLParser # Python 2 and 3 (after ``pip install future``) from html.parser import HTMLParser # Python 2 and 3 (alternative 2): from future.moves.html.parser import HTMLParser
urllib is the hardest module to use from Python 2/3 compatible code. You might want to switch to Requests (http://python-requests.org) instead.
# Python 2 only: from urlparse import urlparse from urllib import urlencode from urllib2 import urlopen, Request, HTTPError
# Python 3 only: from urllib.parse import urlparse, urlencode from urllib.request import urlopen, Request from urllib.error import HTTPError
# Python 2 and 3: easiest option from future.standard_library import install_aliases install_aliases() from urllib.parse import urlparse, urlencode from urllib.request import urlopen, Request from urllib.error import HTTPError
# Python 2 and 3: alternative 2 from future.standard_library import hooks with hooks(): from urllib.parse import urlparse, urlencode from urllib.request import urlopen, Request from urllib.error import HTTPError
# Python 2 and 3: alternative 3 from future.moves.urllib.parse import urlparse, urlencode from future.moves.urllib.request import urlopen, Request from future.moves.urllib.error import HTTPError # or from six.moves.urllib.parse import urlparse, urlencode from six.moves.urllib.request import urlopen from six.moves.urllib.error import HTTPError
# Python 2 and 3: alternative 4 try: from urllib.parse import urlparse, urlencode from urllib.request import urlopen, Request from urllib.error import HTTPError except ImportError: from urlparse import urlparse from urllib import urlencode from urllib2 import urlopen, Request, HTTPError
# Python 2 only: import Tkinter import Dialog import FileDialog import ScrolledText import SimpleDialog import Tix import Tkconstants import Tkdnd import tkColorChooser import tkCommonDialog import tkFileDialog import tkFont import tkMessageBox import tkSimpleDialog import ttk # Python 2 and 3 (after ``pip install future``): import tkinter import tkinter.dialog import tkinter.filedialog import tkinter.scrolledtext import tkinter.simpledialog import tkinter.tix import tkinter.constants import tkinter.dnd import tkinter.colorchooser import tkinter.commondialog import tkinter.filedialog import tkinter.font import tkinter.messagebox import tkinter.simpledialog import tkinter.ttk
# Python 2 only: import SocketServer # Python 2 and 3 (after ``pip install future``): import socketserver
# Python 2 only: import copy_reg # Python 2 and 3 (after ``pip install future``): import copyreg
# Python 2 only: from ConfigParser import ConfigParser # Python 2 and 3 (after ``pip install configparser``): from configparser import ConfigParser
# Python 2 only: from Queue import Queue, heapq, deque # Python 2 and 3 (after ``pip install future``): from queue import Queue, heapq, deque
# Python 2 only: from repr import aRepr, repr # Python 2 and 3 (after ``pip install future``): from reprlib import aRepr, repr
# Python 2 only: from UserDict import UserDict from UserList import UserList from UserString import UserString # Python 3 only: from collections import UserDict, UserList, UserString # Python 2 and 3: alternative 1 from future.moves.collections import UserDict, UserList, UserString # Python 2 and 3: alternative 2 from six.moves import UserDict, UserList, UserString # Python 2 and 3: alternative 3 from future.standard_library import install_aliases install_aliases() from collections import UserDict, UserList, UserString