Py3K: Solving the “outer scope” problem

I recently built the beta of Python 3000 – the upcoming total revamp of Python (due to be released in September – 992 years before they promised!) Because Py3K is unashamedly “backwards incompatible”, they are finally fixing all the major language flaws and making things “the way they should be!” (Note there will be a somewhat automated conversion process from Python 2 to 3 code).

And I love it! Everything is fixed the way I hoped. Hence this is the first in the “Py3K rox my sox” series of blog posts. You can see a summary of new features here.

OK, so one of the major problems I’ve complained about (and heard) in Python is the so-called “outer scope” problem. This is a very definite limitation of what you can do in Python. Read on!

How globals really work

First a bit of background you may not know. This applies to all versions of Python, not just 3.0.

In Python if you don’t declare a variable, Python figures out whether you’re referring to a local or global based on whether you write to it. For example:

x = 4
def f():
    return x

Here, Python figures out that the x you refer to is actually the global x, and returns 4. It figures this out because the function never writes to x, anywhere. Not just because it hasn’t written to x yet, but because it has no statement which assigns to x. (It figures this out statically, not at runtime). So, for example:

x = 4
def f():
    if True:
        return x
    else:
        x = 2

This would be a neat quiz question actually: What does f() evaluate to?

Answer: UnboundLocalError: local variable ‘x’ referenced before assignment.

The mere fact that x is assigned somewhere in the function (even somewhere which will never be executed) causes Python to treat it as a local, and hence it is undefined when you go to return it.

The correct solution is to declare it “global” explicitly, which is the only way to make a function which writes to a global.

x = 4
def f():
    global x
    if True:
        return x
    else:
        x = 2

This works well in practice, because you can define constants like MAX_FOO and use them all over the place without declaring them global, but you need to be explicit if you want to update a global (which is usually a good idea because it’s dangerous – see JavaScript for a counter-example).

The “outer scope” problem

On to the “outer scope” problem. Basically, Python lets you write nested functions, and the nested functions have access to the local variables of their containing code. For example:

def outer():
    x = 9
    def inner_read():
        return x
    return inner_read()

If you call outer(), it will return 9. The variable x is local to the outer function. But the inner function can read it, and return it.

The problem comes when you want to write to a non-local variable, like this:

def outer():
    x = 9
    def inner_read():
        return x
    def inner_write():
        x = 3
    inner_write()
    return inner_read()

As with global variables, Python can find outer scope variables if you only read them (as inner_read does), but if you write to them anywhere in the function, it assumes you are making a new local variable (as inner_write does). Hence inner_write creates a new local x, and assigns it 3, and the function outer returns 9. I would like for inner_write to update the existing x, and hence have outer return 3.

The solution is pretty simple: Have a keyword like global, but rather than going all the way to the top scope, it just tells Python to look for the innermost scope with a bound variable of that name.

Python 3.0 introduces exactly that: the nonlocal keyword. Let’s give it a try!

def outer():
    x = 9
    def inner_read():
        return x
    def inner_write():
        nonlocal x
        x = 3
    inner_write()
    return inner_read()

Woot! Python 3.0 compiles this code and the outer function returns 3.

The funny thing is, this problem seems to be specific to Python. In most static languages, all variables are declared. In Haskell, all variables are read-only. In Ruby, you refer to global variables by prefixing them with a $dollar. In JavaScript, it’s the inverse of Python: you declare all local variables and they default to global (which is a hideous idea – if you forget to declare a variable you implicitly start sharing where you didn’t expect to be sharing). Of course there are probably other languages with this problem but Python is the only one I’ve ever seen.

References

• PEP 3104 – Access to Names in Outer Scopes – Official proposal / discussion of this feature