Python Metaclasses and Metaprogramming

Imagine if you could have computer programs that wrote your code for you. It is possible, but the machines will not write all your code for you!

This technique, called metaprogramming, is popular with code framework developers. This is how you get code generation and smart features in many popular frameworks and libraries like Ruby On Rails or TensorFlow.

Functional programming languages like Elixir, Clojure, and Ruby are noted for their metaprogramming capabilities. In this guide, we show you how you can tap into the power of metaprogramming in Python. The code examples are written for Python 3, but will work for Python 2 with some adjustments.

What is a Metaclass in Python?

Python is an object-oriented language that makes working with classes easy.

Metaprogramming in Python relies on a special new type of class that is called the metaclass. This type of class, in short, holds the instructions about the behind-the-scenes code generation that you want to take place when another piece of code is being executed.

Wikipedia sums up metaclasses pretty well:

In object-oriented programming, a metaclass is a class whose instances are classes

When we define a class, the objects of that class are created using the class as the blueprint.

But what about the class itself? What is the blueprint of the class itself?

This is where a metaclass comes in. A metaclass is the blueprint of the class itself, just like a class is the blueprint for instances of that class. A metaclass is a class that defines properties of other classes.

With a metaclass, we can define properties that should be added to new classes that are defined in our code.

For example, the following metaclass code sample adds a hello property to each class which uses this metaclass as its template. This means, new classes that are instances of this metaclass will have a hello property without needing to define one themselves.

# hello_metaclass.py
# A simple metaclass
# This metaclass adds a 'hello' method to classes that use the metaclass
# meaning, those classes get a 'hello' method with no extra effort
# the metaclass takes care of the code generation for us
class HelloMeta(type):
    # A hello method
    def hello(cls):
        print("greetings from %s, a HelloMeta type class" % (type(cls())))

    # Call the metaclass
    def __call__(self, *args, **kwargs):
        # create the new class as normal
        cls = type.__call__(self, *args)

        # define a new hello method for each of these classes
        setattr(cls, "hello", self.hello)

        # return the class
        return cls

# Try out the metaclass
class TryHello(object, metaclass=HelloMeta):
    def greet(self):
        self.hello()

# Create an instance of the metaclass. It should automatically have a hello method
# even though one is not defined manually in the class
# in other words, it is added for us by the metaclass
greeter = TryHello()
greeter.greet()

The result of running this code is that the new TryHello class is able to print out a greeting that says:

greetings from <class '__main__.TryHello'>, a HelloMeta type class

The method responsible for this printout is not declared in the declaration of the class. Rather, the metaclass, which is HelloMeta in this case, generates the code at run time that automatically affixes the method to the class.

To see it in action, feel free to copy and paste the code in a Python console. Also, read the comments to better understand what we have done in each part of the code. We have a new object, named greeter, which is an instance of the TryHello class. However, we are able to call TryHello's self.hello method even though no such method was defined in the TryHello class declaration.

Rather than get an error for calling a method that does not exist, TryHello gets such a method automatically affixed to it due to using the HelloMeta class as its metaclass.

Metaclasses give us the ability to write code that transforms, not just data, but other code, e.g. transforming a class at the time when it is instantiated. In the example above, our metaclass adds a new method automatically to new classes that we define to use our metaclass as their metaclass.

This is an example of metaprogramming. Metaprogramming is simply writing code that works with metaclasses and related techniques to do some form of code transformation in the background.

The beautiful thing about metaprogramming is that, rather than output source code, it gives us back only the execution of that code. The end user of our program is unaware of the "magic" happening in the background.

Think about software frameworks that do code generation in the background to make sure you as a programmer have to write less code for everything. Here are some great examples:

• Django
• SQLAlchemy
• Flask
• Theano

Outside Python, other popular libraries such as Ruby On Rails (Ruby) and Boost(C++) are examples of where metaprogramming is used by framework authors to generate code and take care of things in the background.

The result is simplified end-user APIs that automate a lot of work for the programmer who codes in the framework.

Taking care of making that simplicity work behind the scenes, is a lot of metaprogramming baked into the framework source code.

Theory Section: Understanding How Metaclasses Work

To understand how Python metaclasses work, you need to be very comfortable with the notion of types in Python.

A type is simply the data or object nomenclature for an object in Python.

Finding the Type of an Object

Using the Python REPL, let's create a simple string object and inspect its type, as follows:

>>> day = "Sunday"
>>> print("The type of variable day is %s" % (type(day)))
The type of variable day is <type 'str'>

As you'd expect, we get a printout that the variable day is of type str, which is a string type. You can find the type of any object just using the built-in type function with one object argument.

Finding the Type of a Class

So, a string like "Sunday" or "hello" is of type str, but what about str itself? What is the type of the str class?

Again, type in the Python console:

>>> type(str)
<type 'type'>

This time, we get a printout that str is of type type.

Type and the Type of Type

But what about type itself? What is type's type?

>>> type(type)
<type 'type'>

The result is, once again, "type". Thus we find that type is not only the metaclass of classes such as int, it's also its own metaclass!

Special Methods Used by Metaclasses

At this point it may help to review the theory a bit. Remember that a metaclass is a class whose instances are themselves classes, and not just simple objects.

In Python 3 you can assign a metaclass to the creation of a new class by passing in the intended masterclass to the new class definition.

The type type, as the default metaclass in Python, defines special methods that new metaclasses can override to implement unique code generation behavior. Here is a brief overview of these "magic" methods that exist on a metaclass:

• __new__: This method is called on the Metaclass before an instance of a class based on the metaclass is created
• __init__: This method is called to set up values after the instance/object is created
• __prepare__: Defines the class namespace in a mapping that stores the attributes
• __call__: This method is called when the constructor of the new class is to be used to create an object

These are the methods to override in your custom metaclass to give your classes behavior different from that of type, which is the default metaclass.

Metaprogramming Practice 1: Using Decorators to Transform Function Behavior

Let's take a step back before we proceed with using metaclasses metaprogramming practice. A common usage of metaprogramming in Python is the usage of decorators.

A decorator is a function that transforms the execution of a function. In other words, it takes a function as input, and returns another function.

For example, here is a decorator that takes any function, and prints out the name of the function before running the original function as normal. This could be useful for logging function calls, for example:

# decorators.py

from functools import wraps

# Create a new decorator named notifyfunc
def notifyfunc(fn):
    """prints out the function name before executing it"""
    @wraps(fn)
    def composite(*args, **kwargs):
        print("Executing '%s'" % fn.__name__)
        # Run the original function and return the result, if any
        rt = fn(*args, **kwargs)
        return rt
    # Return our composite function
    return composite

# Apply our decorator to a normal function that prints out the result of multiplying its arguments
@notifyfunc
def multiply(a, b):
    product = a * b
    return product

You can copy and paste the code into a Python REPL. The neat thing about using the decorator is that the composite function is executed in place of the input function. The result of the above code is that the multiply function announces it is running before its computation runs:

>>> multiply(5, 6)
Executing 'multiply'
30
>>>
>>> multiply(89, 5)
Executing 'multiply'
445

In short, decorators achieve the same code-transformation behavior of metaclasses, but are much simpler. You would want to use decorators where you need to apply common metaprogramming around your code. For example, you could write a decorator that logs all database calls.

Metaprogramming Practice 2: Using Metaclasses like a Decorator Function

Metaclasses can replace or modify attributes of classes. They have the power to hook in before a new object is created, or after the new object is created. The result is greater flexibility regarding what you can use them for.

Below, we create a metaclass that achieves the same result as the decorator from the prior example.

To compare the two, you should run both examples side by side then follow along with the annotated source code. Note that you can copy the code and paste it straight into your REPL, if your REPL preserves the code formatting.

# metaclassdecorator.py
import types

# Function that prints the name of a passed in function, and returns a new function
# encapsulating the behavior of the original function
def notify(fn, *args, **kwargs):

    def fncomposite(*args, **kwargs):
        # Normal notify functionality
        print("running %s" % fn.__name__)
        rt = fn(*args, **kwargs)
        return rt
    # Return the composite function
    return fncomposite

# A metaclass that replaces methods of its classes
# with new methods 'enhanced' by the behavior of the composite function transformer
class Notifies(type):

    def __new__(cls, name, bases, attr):
        # Replace each function with
        # a print statement of the function name
        # followed by running the computation with the provided args and returning the computation result
        for name, value in attr.items():
            if type(value) is types.FunctionType or type(value) is types.MethodType:
                attr[name] = notify(value)

        return super(Notifies, cls).__new__(cls, name, bases, attr)

# Test the metaclass
class Math(metaclass=Notifies):
    def multiply(a, b):
        product = a * b
        print(product)
        return product

Math.multiply(5, 6)

# Running multiply():
# 30


class Shouter(metaclass=Notifies):
    def intro(self):
        print("I shout!")

s = Shouter()
s.intro()

# Running intro():
# I shout!

Classes that use our Notifies metaclass, for example Shouter and Math, have their methods replaced, at creation time, with enhanced versions that first notify us via a print statement of the name of the method now running. This is identical to the behavior we implemented before using a decorator function.

Metaclasses Example 1: Implementing a Class that can't be Subclassed

Common use cases for metaprogramming include controlling class instances.

For example, singletons are used in many code libraries. A singleton class controls instance creation such that there is only ever at most one instance of the class in the program.

A final class is another example of controlling class usage. With a final class, the class does not allow subclasses to be created. Final classes are used in some frameworks for security, ensuring the class retains its original attributes.

Below, we give an implementation of a final class using a metaclass to restrict the class from being inherited by another.

# final.py

# a final metaclass. Sub-classing a class that has the Final metaclass should fail
class Final(type):
    def __new__(cls, name, bases, attr):
        # Final cannot be subclassed
        # check that a Final class has not been passed as a base
        # if so, raise error, else, create the new class with Final attributes
        type_arr = [type(x) for x in bases]
        for i in type_arr:
            if i is Final:
                raise RuntimeError("You cannot subclass a Final class")
        return super(Final, cls).__new__(cls, name, bases, attr)


# Test: use the metaclass to create a Cop class that is final

class Cop(metaclass=Final):
    def exit():
        print("Exiting...")
        quit()

# Attempt to subclass the Cop class, this should ideally raise an exception!
class FakeCop(Cop):
    def scam():
        print("This is a hold up!")

cop1 = Cop()
fakecop1 = FakeCop()

# More tests, another Final class
class Goat(metaclass=Final):
    location = "Goatland"

# Sub-classing a final class should fail
class BillyGoat(Goat):
    location = "Billyland"

In the code, we've included class declarations for attempting to subclass a Final class. These declarations fail, resulting in exceptions being thrown. Using a metaclass that restricts subclassing its classes enables us to implement final classes in our codebase.

Metaclasses Example 2: Creating a Class Track Operation Execution Time

Profilers are used to take stock of resource usage in a computing system. A profiler can track things like memory usage, processing speed, and other technical metrics.

We can use a metaclass to keep track of code execution time. Our code example is not a full profiler, but is a proof of concept of how you can do the metaprogramming for profiler-like functionality.

# timermetaclass.py
import types

# A timer utility class
import time

class Timer:
    def __init__(self, func=time.perf_counter):
        self.elapsed = 0.0
        self._func = func
        self._start = None

    def start(self):
        if self._start is not None:
            raise RuntimeError('Already started')
        self._start = self._func()

    def stop(self):
        if self._start is None:
            raise RuntimeError('Not started')
        end = self._func()
        self.elapsed += end - self._start
        self._start = None

    def reset(self):
        self.elapsed = 0.0

    @property
    def running(self):
        return self._start is not None

    def __enter__(self):
        self.start()
        return self

    def __exit__(self, *args):
        self.stop()


# Below, we create the Timed metaclass that times its classes' methods
# along with the setup functions that rewrite the class methods at
# class creation times


# Function that times execution of a passed in function, returns a new function
# encapsulating the behavior of the original function
def timefunc(fn, *args, **kwargs):

    def fncomposite(*args, **kwargs):
        timer = Timer()
        timer.start()
        rt = fn(*args, **kwargs)
        timer.stop()
        print("Executing %s took %s seconds." % (fn.__name__, timer.elapsed))
        return rt
    # return the composite function
    return fncomposite

# The 'Timed' metaclass that replaces methods of its classes
# with new methods 'timed' by the behavior of the composite function transformer
class Timed(type):

    def __new__(cls, name, bases, attr):
        # replace each function with
        # a new function that is timed
        # run the computation with the provided args and return the computation result
        for name, value in attr.items():
            if type(value) is types.FunctionType or type(value) is types.MethodType:
                attr[name] = timefunc(value)

        return super(Timed, cls).__new__(cls, name, bases, attr)

# The below code example test the metaclass
# Classes that use the Timed metaclass should be timed for us automatically
# check the result in the REPL

class Math(metaclass=Timed):

    def multiply(a, b):
        product = a * b
        print(product)
        return product

Math.multiply(5, 6)


class Shouter(metaclass=Timed):

    def intro(self):
        print("I shout!")

s = Shouter()
s.intro()


def divide(a, b):
    result = a / b
    print(result)
    return result

div = timefunc(divide)
div(9, 3)

As you can see, we were able to create a Timed metaclass that rewrites its classes on-the-fly. Whenever a new class that uses the Timed metaclass is declared, its methods are rewritten to be timed by our timer utility class. Whenever we run computations using a Timed class, we get the timing done for us automatically, without needing to do anything extra.

Metaprogramming is a great tool if you are writing code and tools to be used by other developers, such as web frameworks or debuggers. With code-generation and metaprogramming, you can make life easy for the programmers that make use of your code libraries.

Mastering the Power of Metaclasses

Metaclasses and metaprogramming have a lot of power. The downside is that metaprogramming can get fairly complicated. In a lot of cases, using decorators provides a simpler way to get an elegant solution. Metaclasses should be used when circumstances demand generality rather than simplicity.

To make effective use of metaclasses, we suggest reading up in the official Python 3 metaclasses documentation.