Millionen von Datensätzen auf einmal verarbeiten = Out of Memory. Chunking, Streaming und Parallelismus sind die Lösung.

Chunking¶

Python — Verarbeitung in 1000er-Blöcken¶

def process_in_chunks(query, chunk_size=1000): offset = 0 while True: chunk = db.execute(query.limit(chunk_size).offset(offset)).fetchall() if not chunk: break for row in chunk: process(row) db.commit() offset += chunk_size

Server-side Cursor (PostgreSQL)¶

SQLAlchemy — Server-side Cursor¶

with engine.connect().execution_options(stream_results=True) as conn: result = conn.execute(text(“SELECT * FROM big_table”)) for chunk in result.partitions(1000): for row in chunk: process(row)

Parallelismus¶

from concurrent.futures import ProcessPoolExecutor with ProcessPoolExecutor(max_workers=4) as executor: futures = [executor.submit(process_chunk, chunk) for chunk in chunks] results = [f.result() for f in futures]

Wichtigste Erkenntnis¶

Chunking für Speichereffizienz, Server-side Cursors für Streaming, ProcessPoolExecutor für CPU-bound Aufgaben.