Python Generators & Itertools 

Learn how generators make your Python programs memory-efficient and how the `itertools` module helps you build fast combinatorics and iterator-based solutions. Examples, tables, exercises, and FAQs included.

Quick intro:

Generators are simple functions that let you create iterators with yield — they produce values lazily, one at a time, saving memory for large or infinite sequences. The built-in itertools module complements generators with efficient iterator building blocks (combinations, permutations, chaining, and more).

Contents

• What are generators?
• Generator functions and expressions
• Iterators vs generators (table)
• Popular itertools functions and examples
• Real-world uses, exercises, FAQ, conclusion

What are generators?

In Python, a generator is a tool for creating iterators in a simple and readable way. Instead of building a full iterator class with __iter__ and __next__, you write a function that uses yield to produce values one at a time. When a generator yields a value, it preserves its state between calls — making it ideal for large datasets or infinite streams.

Why use generators?

• Memory efficient — produce items on demand instead of storing a full list.
• Cleaner code — simpler than creating custom iterator classes.
• Suitable for pipelines — compose with other iterators and functions.

Generator functions — yield explained

A generator function looks like a normal function but uses yield to return values one by one. Every time it yields, execution pauses and resumes when the next value is requested.

# simple generator that yields first n numbers
def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

# use it
for num in count_up_to(5):
    print(num)
    

Output:

1
2
3
4
5

Generator expressions

Generator expressions are like list comprehensions but lazy — they use parentheses instead of square brackets.

# generator expression
squares = (x*x for x in range(10))
print(next(squares))  # 0
print(next(squares))  # 1
# iterate the rest
for s in squares:
    print(s)
  

Yield vs Return (short)

return returns a value and exits the function. yield produces a value and pauses the function, keeping local state to resume later.

Iterators vs Generators (comparison)

Feature	Iterator	Generator
Creation	Class implementing __iter__ and __next__	Function with yield or generator expression
Memory	May store all items	Produces items lazily (low memory)
Reusability	Depends — can reset or create new instance	Single-use — once exhausted, it’s done
Use cases	Custom iteration logic	Streaming, large files, pipelines, infinite sequences

The itertools module — iterate like a pro

itertools is a standard Python module (no install required) that provides fast, memory-efficient tools for working with iterators. The functions are implemented in C and designed to be building blocks for efficient iteration.

Import pattern

from itertools import count, cycle, repeat, accumulate, chain, combinations, permutations, product, groupby

Common itertools functions (with examples)

1. count(start=0, step=1)

Creates an infinite counter that yields values indefinitely. Useful with zip or when you need indexes without building a list.

from itertools import count
for i in count(5):
    if i > 8:
        break
    print(i)
# prints 5 6 7 8
  

2. cycle(iterable)

Cycle through iterable elements infinitely. Combine with islice (from itertools) to limit the iterations.

from itertools import cycle, islice
colors = ['red','green','blue']
for c in islice(cycle(colors), 7):
    print(c)
# red green blue red green blue red
  

3. repeat(elem, times=None)

Repeat an element indefinite times or a fixed number of times.

from itertools import repeat
for x in repeat('hello', 3):
    print(x)
  

4. accumulate(iterable, func=None)

Return cumulative results. By default it provides running sums, but you can supply a custom function (like operator.mul).

from itertools import accumulate
import operator
nums = [1,2,3,4]
print(list(accumulate(nums)))            # [1, 3, 6, 10]
print(list(accumulate(nums, operator.mul)))  # [1,2,6,24]
  

5. chain(*iterables)

Combine multiple iterables sequentially without creating extra lists.

from itertools import chain
for x in chain([1,2], ['a','b']):
    print(x)
  

6. combinations(iterable, r) and permutations(iterable, r=None)

Return combinations (unordered r-length tuples) and permutations (ordered r-length tuples). Useful in combinatorics, feature selection, and tests.

from itertools import combinations, permutations
items = ['a','b','c']
print(list(combinations(items, 2)))
# [('a','b'), ('a','c'), ('b','c')]
print(list(permutations(items, 2)))
# [('a','b'),('a','c'),('b','a'),('b','c'),('c','a'),('c','b')]
  

7. product(*iterables, repeat=1)

Cartesian product. Equivalent to nested loops. Great for parameter grids.

from itertools import product
print(list(product([1,2], ['a','b'])))
# [(1,'a'),(1,'b'),(2,'a'),(2,'b')]
  

8. groupby(iterable, key=None)

Group consecutive items that share a key function. Note: data should be sorted by the same key for grouping all similar items together.

from itertools import groupby
data = [{'type':'A','val':1},{'type':'B','val':3},{'type':'A','val':2}]
# groupby groups only consecutive equal keys, so sort first
from operator import itemgetter
data_sorted = sorted(data, key=itemgetter('type'))
for k, group in groupby(data_sorted, key=itemgetter('type')):
    print(k, list(group))
  

Quick itertools reference

Function	Use
count()	Infinite counter
cycle()	Repeat iterable endlessly
repeat()	Repeat an element
accumulate()	Running totals or custom accumulation
chain()	Chain iterables
combinations(), permutations(), product()	Combinatorics & Cartesian product
groupby()	Group consecutive items

Real-world uses

• Reading large files: use a generator to yield lines that match a condition so you never hold the whole file in memory.
• Streaming data processing: pipeline data from a socket or API, transforming items with generator-based stages.
• Combinatorics for testing: use product or combinations to generate test inputs.
• Building infinite sequences: counters, timestamps, or probes that continue until a condition stops them.

Example: Process huge CSV lazily

def csv_row_generator(path):
    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            yield line.strip().split(',')

for row in csv_row_generator('bigfile.csv'):
    process(row)  # only one row in memory at a time
    

Exercises (try these)

1. Even numbers generator: Write a generator even_numbers(n) that yields even numbers from 0 up to n. Use it in a loop.
2. Fibonacci generator: Implement a generator that yields Fibonacci numbers indefinitely; stop after the first 10.
3. itertools combinations: Given a list of 6 items, produce all 3-item combinations and count them without converting to a list (use a loop).
4. groupby usage: Given a list of dictionaries with a 'category' key, sort and group them by category using groupby.

Hints

• Use yield and while True for infinite generators.
• Use islice from itertools to slice infinite iterators safely.
• Remember to sort data before using groupby if you want to group all equal items.

FAQ

Q: Can I reuse a generator after I iterate it once?

A: No — generators are single-use. If you need the sequence again, recreate the generator or save results to a list (which uses memory).

Q: How is yield different from return?

A: return sends a value (or None) and exits the function. yield sends a value and pauses the function, keeping local variables intact to resume later.

Q: Is itertools built-in?

A: Yes — it’s part of Python’s standard library (no external install needed).

Q: Are itertools functions generators?

A: Many itertools functions return iterators (not strictly generator objects), but they behave like generators: they produce values lazily.

Conclusion

Generators and itertools are powerful tools in Python for writing concise, memory-efficient code. Use generators when you need to process large or infinite streams and use itertools when you want tested, fast building blocks for iterator algebra (combinations, chaining, accumulating, and more).

Next steps: try the exercises, read the official itertools docs, and combine several itertools functions to build pipelines for real problems (for example: read data → filter → accumulate → summarize).

Internal links you can add:

• Python Programming: A Beginner’s Guide
• Python Variables
• Python Data Types
• Python Strings
• Python Operators
• Python Lists
  
• Python Tuples
• Python Sets
• Python Dictionaries
• Python If...Else Statements and Conditions
• Python Match Statement
• Python Functions
  
• Python Lambda
  
• Python Arrays
  
• Python Classes & Objects
  
• Python Inheritance
  
• Python Iterators
  
• Python Polymorphism
  
• Python Scope
  
• Python Modules
  
• Python Datetime
• Python Math
  
• Python Exception Handling
  
• Python File Handling

Continue Learning: Explore more beginner-friendly topics and practice materials in our Programming Resource Hub.

Tags: Python Generators Itertools