Notebook · code/module-07/03-event-time-and-windows.ipynb

Download 03-event-time-and-windows.ipynb

Open it in JupyterLab or VS Code from the lab folder to run it. This page is a read-only view.

Event time and windows — practical

The grid report sums each site's energy in 5-minute windows. One hour of meter readings arrives, and site-N03's link drops for five minutes, so its readings arrive late. You build the report by arrival time and by event time, with different grace periods, and then meet a charger whose clock is three hours ahead.

Lab services: kafka-1, kafka-2, kafka-3 · Script: module-07/grid_report.py · Lesson: modules/07-stream-processing/03-event-time-and-windows.md

# Setup: find the lab's shared helpers and check the lab is running
import sys, pathlib
lab_root = next(p for p in [pathlib.Path.cwd(), *pathlib.Path.cwd().parents] if (p / "common").is_dir())
sys.path.insert(0, str(lab_root))

import subprocess
from datetime import timedelta
from common import settings, lab
from common.scenario import Brightlane, BASE_TIME, iso, parse_iso, to_json

lab.require_kafka()
lab.print_versions()

The readings from 17:30 to 18:30. Site-N03's mobile link drops from 18:05 to 18:10: its readings from those five minutes arrive five minutes late. Each record's Kafka timestamp is the time it arrived at the gateway; the reading's own ts is the time the charger measured it.

from confluent_kafka import Producer
from confluent_kafka.admin import AdminClient

TOPIC = "m07l03-telemetry"
admin = AdminClient({"bootstrap.servers": settings.KAFKA_BOOTSTRAP})
readings = [(key, e) for key, e in Brightlane().stream(sessions=300)
            if "type" not in e and parse_iso(e["ts"]) < BASE_TIME + timedelta(hours=1)]
outage = (BASE_TIME + timedelta(minutes=35), BASE_TIME + timedelta(minutes=40))     # 18:05 to 18:10
arrivals = Brightlane.delay(readings, "site-N03", *outage, delay_s=300)

def write(topic, rows):
    lab.recreate_topic(admin, topic, partitions=1)
    producer = Producer(settings.PRODUCER_DEFAULTS)
    for arrived, key, reading in rows:
        producer.produce(topic, key=key, value=to_json(reading), timestamp=int(arrived.timestamp() * 1000))
    producer.flush(30)

write(TOPIC, arrivals)
late = [(a, e) for a, _, e in arrivals if a != parse_iso(e["ts"])]
print(f"{len(arrivals)} readings; {len(late)} from site-N03 arrive late, for example:")
print(f"  measured {late[0][1]['ts'][11:19]}, arrived {iso(late[0][0])[11:19]}")

Step 2: Windows by arrival time

report() runs the grid report script and prints its summary: totals, then site-N03 window by window from 17:55 to 18:25.

def report(topic, *options):
    result = subprocess.run([sys.executable, "module-07/grid_report.py", "--topic", topic, *options],
                            cwd=lab_root, capture_output=True, text=True)
    print(result.stdout.strip() or result.stderr[-800:])

report(TOPIC, "--time", "arrival")

Between 18:05 and 18:10, site-N03 seems to use no energy at all, and twice the usual amount in the next window.

Step 3: Windows by event time, with a 2-minute grace period

Now each reading counts in the window of its own ts. A window is reported once the stream time, the latest event time seen, has passed its end by the grace period.

report(TOPIC, "--time", "event", "--grace", "120")

The window 18:05–18:10 has part of its readings. The rest came after it was reported: they are late, counted but not added.

Change it: a 6-minute grace period

report(TOPIC, "--time", "event", "--grace", "360")

Every window is complete, but every window is also reported 4 minutes later than before.

Break it: a clock three hours ahead

One reading from a charger whose clock jumped three hours ahead arrives at 18:07. Predict first: what happens to the windows after it?

middle = len(arrivals) // 2
arrived, key, reading = arrivals[middle]
wrong_clock = (arrived, key, {**reading, "ts": iso(arrived + timedelta(hours=3))})
write(f"{TOPIC}-clock", arrivals[:middle] + [wrong_clock] + arrivals[middle:])
print("wrong clock at", iso(arrived)[11:19], "says", wrong_clock[2]["ts"][11:19])
report(f"{TOPIC}-clock", "--time", "event", "--grace", "120")

One reading pushed the stream time three hours ahead, every open window closed, and almost every reading after it was late. The guard: ignore event times that are far ahead of the arrival time.

report(f"{TOPIC}-clock", "--time", "event", "--grace", "120", "--max-ahead", "300")

Clean up

Delete the topics. If anything gets stuck, docker compose down -v in the code/ folder resets the whole lab.

lab.delete_topics(admin, [t for t in admin.list_topics(timeout=10).topics if t.startswith(TOPIC)])
admin.delete_consumer_groups(["m07l03-report"])
print("cleaned up")