Plane Tickets

A generator is a function or expression that returns a special type of iterator called a generator iterator. Generator-iterators are lazy: they do not store their values in memory, but generate their values when needed.

A generator function looks like any other function, but contains one or more yield expressions. Each yield will suspend code execution, saving the current execution state (including all local variables and try-statements). When the generator resumes, it picks up state from the suspension - unlike regular functions which reset with every call.

Constructing a generator

Generators are constructed much like other looping or recursive functions, but require a yield expression, which we will explore in depth a bit later.

An example is a function that returns the squares from a given list of numbers. As currently written, all input must be processed before any values can be returned:

>>> def squares(list_of_numbers):
...     squares = []
...     for number in list_of_numbers:
...         squares.append(number ** 2)
...     return squares

You can convert that function into a generator like this:

>>> def squares_generator(list_of_numbers):
...     for number in list_of_numbers:
...         yield number ** 2

The rationale behind this is that you use a generator when you do not need to produce all the values at once. This saves memory and processing power, since only the value you are currently working on is calculated.

Using a generator

Generators may be used in place of most iterables in Python. This includes functions or objects that require an iterable/iterator as an argument.

To use the squares_generator() generator:

>>> squared_numbers = squares_generator([1, 2, 3, 4])

>>> for square in squared_numbers:
...     print(square)
...
1
4
9
16

Values within a generator can also be produced/accessed via the next() function. next() calls the __next__() method of a generator-iterator object, "advancing" or evaluating the code up to its yield expression, which then "yields" or returns a value:

>>> squared_numbers = squares_generator([1, 2])

>>> next(squared_numbers)
1
>>> next(squared_numbers)
4

When a generator-iterator is fully consumed and has no more values to return, it throws a StopIteration error.

>>> next(squared_numbers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The yield expression

The yield expression is very similar to the return expression. Unlike the return expression, yield gives up values to the caller at a specific point, suspending evaluation/return of any additional values until they are requested. When yield is evaluated, it pauses the execution of the enclosing function and returns any values of the function at that point in time. The function then stays in scope, and when __next__() is called, execution resumes until yield is encountered again.

>>> def infinite_sequence():
...     current_number = 0
...     while True:
...         yield current_number
...         current_number += 1

>>> lets_try = infinite_sequence()
>>> lets_try.__next__()
0
>>> lets_try.__next__()
1

Why Create a Generator?

Generators are useful in a lot of applications.

When working with a potentially large collection of values, you might not want to put all of them into memory. A generator can be used to work on larger data piece-by-piece, saving memory and improving performance.

Generators are also very helpful when a process or calculation is complex, expensive, or infinite:

>>> def infinite_sequence():
...     current_number = 0
...     while True:
...         yield current_number
...         current_number += 1

Now whenever __next__() is called on the infinite_sequence object, it will return the previous number + 1.