Download 03-event-time-and-windows.ipynb
Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.
Event time and windows — practical
The grid report sums each site's energy in 5-minute windows. One hour of meter readings arrives, and site-N03's link drops for five minutes, so its readings arrive late. You build the report by arrival time and by event time, with different grace periods, and then meet a charger whose clock is three hours ahead.
Lab services: kafka-1, kafka-2, kafka-3 · Script: module-07/grid_report.py
· Lesson: modules/07-stream-processing/03-event-time-and-windows.md
# Setup: find the lab's shared helpers and check the lab is running import sys, pathlib lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir()) sys.path.insert(0, str(lab_root)) import subprocess from datetime import timedelta from common import settings, lab from common.scenario import Brightlane, BASE_TIME, iso, parse_iso, to_json lab.require_kafka() lab.print_versions()
Step 1: One hour of readings, and a link that drops
The readings from 17:30 to 18:30. Site-N03's mobile link drops from 18:05 to 18:10: its readings
from those five minutes arrive five minutes late. Each record's Kafka timestamp is the time it
arrived at the gateway; the reading's own ts is the time the charger measured it.
from confluent_kafka import Producer from confluent_kafka.admin import AdminClient TOPIC = "m07l03-telemetry" admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP}) readings = [(key, e) for key, e in Brightlane().stream(sessions=300) if "type" not in e and parse_iso(e["ts"]) < BASE_TIME + timedelta(hours=1)] outage = (BASE_TIME + timedelta(minutes=35), BASE_TIME + timedelta(minutes=40)) # 18:05 to 18:10 arrivals = Brightlane.delay(readings, "site-N03", *outage, delay_s=300) def write(topic, rows): lab.recreate_topic(admin, topic, partitions=1) producer = Producer(settings.PRODUCER_DEFAULTS) for arrived, key, reading in rows: producer.produce(topic, key=key, value=to_json(reading), timestamp=int(arrived.timestamp() * 1000)) producer.flush(30) write(TOPIC, arrivals) late = [(a, e) for a, _, e in arrivals if a != parse_iso(e["ts"])] print(f"{len(arrivals)} readings; {len(late)} from site-N03 arrive late, for example:") print(f" measured {late[0][1]['ts'][11:19]}, arrived {iso(late[0][0])[11:19]}")
Step 2: Windows by arrival time
report() runs the grid report script and prints its summary: totals, then site-N03 window by
window from 17:55 to 18:25.
def report(topic, *options): result = subprocess.run([sys.executable, "module-07/grid_report.py", "--topic", topic, *options], cwd=lab_root, capture_output=True, text=True) print(result.stdout.strip() or result.stderr[-800:]) report(TOPIC, "--time", "arrival")
Between 18:05 and 18:10, site-N03 seems to use no energy at all, and twice the usual amount in the next window.
Step 3: Windows by event time, with a 2-minute grace period
Now each reading counts in the window of its own ts. A window is reported once the stream time,
the latest event time seen, has passed its end by the grace period.
report(TOPIC, "--time", "event", "--grace", "120")
The window 18:05–18:10 has part of its readings. The rest came after it was reported: they are late, counted but not added.
Change it: a 6-minute grace period
report(TOPIC, "--time", "event", "--grace", "360")
Every window is complete, but every window is also reported 4 minutes later than before.
Break it: a clock three hours ahead
One reading from a charger whose clock jumped three hours ahead arrives at 18:07. Predict first: what happens to the windows after it?
middle = len(arrivals) // 2 arrived, key, reading = arrivals[middle] wrong_clock = (arrived, key, {**reading, "ts": iso(arrived + timedelta(hours=3))}) write(f"{TOPIC}-clock", arrivals[:middle] + [wrong_clock] + arrivals[middle:]) print("wrong clock at", iso(arrived)[11:19], "says", wrong_clock[2]["ts"][11:19]) report(f"{TOPIC}-clock", "--time", "event", "--grace", "120")
One reading pushed the stream time three hours ahead, every open window closed, and almost every reading after it was late. The guard: ignore event times that are far ahead of the arrival time.
report(f"{TOPIC}-clock", "--time", "event", "--grace", "120", "--max-ahead", "300")
Clean up
Delete the topics. If anything gets stuck, docker compose down -v in the code/ folder resets
the whole lab.
lab.delete_topics(admin, [t for t in admin.list_topics(timeout=10).topics if t.startswith(TOPIC)]) admin.delete_consumer_groups(["m07l03-report"]) print("cleaned up")