Excel files in, partitioned parquet out — idempotent every time.

1 Vehicle IDs — core/vehicle_ids.py

The first foundational primitive in core/ we haven’t built yet. Vehicle IDs in this fleet follow a pattern: three-digit prefix + sequence number (e.g. 815001, prefix 815, number 001). The prefix classifies the vehicle:

• 815, 832 — AUTO fleet (city-owned automated collection trucks)
• Anything else — MANUAL or OTHER (contractor, hand-collection)

The module exposes a frozen VehicleId dataclass plus parse_vehicle_id() — tolerant of Excel-style noise (commas, trailing .0 from floats, whitespace).

v = parse_vehicle_id("815001")
v.prefix  # "815"
v.number  # "001"
v.kind    # "AUTO"

2 File registry — tonnage/registry.py

Tonnage files arrive daily-ish in a folder. Two sources, two naming conventions:

• RAD exports — the “ride-along data” tonnage records
• ARTS exports — the “automated routing tonnage system” records

The registry: scan the folder, parse source + date from each filename, attach the file’s mtime for tie-breaking. pick_latest_file(registry, source) returns the path of the most recent file for that source. Same input folder, same output — deterministic.

3 Cleaners — tonnage/cleaners.py

Each source has its own header conventions and quirks. The cleaners handle:

• Header resolution — given a list of acceptable header names, find which one this file uses. Legacy systems rename columns occasionally; the cleaner is tolerant.
• Type coercion — dates, floats, fleet IDs — all the “Excel exported numbers as strings or strings as numbers” mess.
• Material code normalization — the two systems use different vocabularies for the same materials (RAD: "RFS" for refuse; ARTS: "Refuse"). matl_norm_rad() and matl_norm_arts() map them to a canonical form.
• Top-level entry points: clean_arts_file(path) and clean_rad_file(path) — return a clean DataFrame + a metadata dict.

4 Hash keys — tonnage/keys.py

For dedup we need a key that’s deterministic from the row content alone — same row, same hash, every time, in every run. Trickier than it sounds because floats have rounding issues and datetimes have timezone issues.

The pattern: round tons to 3 decimals, floor datetimes to the minute, normalize text (strip + uppercase), then concatenate the key columns and hash. add_keys_arts() and add_keys_rad() attach the resulting row_hash column to each cleaned DataFrame.

Why the floors and rounds? Because 0.1 + 0.2 != 0.3 in floating point, and a datetime read back from parquet might lose microseconds it had in Excel. If the hash isn’t stable across round-trips, dedup falls apart.

5 Idempotent upsert — tonnage/upsert.py

One year, one parquet file per source: tonnage_out/<source>/<year>.parquet. The upsert step:

1. Read the existing year file (if it exists).
2. Concatenate new rows.
3. Drop duplicates by row_hash — first occurrence wins.
4. Write back to disk.

Running this on the same input twice does nothing on the second run, because every row hashes to a key that’s already in the file.

6 Pipeline — tonnage/pipeline.py

The one entry point most callers use:

from opentrash.tonnage.pipeline import run_ingest

run_ingest(
    data_dir="data/tonnage_in/",
    out_dir="data/tonnage_out/",
)

Internally: scan registry → pick latest of each source → clean → add keys → upsert into the year-partitioned parquet store. The whole job is a few hundred lines but reads top-to-bottom as five clean steps.

7 Docs deploy — mkdocs.yml + .github/workflows/docs.yml

Three small files turn on the live docs site at opentrash.app:

• mkdocs.yml at repo root — MkDocs Material configuration. Theme is the same charcoal-and-cyan palette as the v1 site. Nav has Home and Architecture for now; more pages added as the package grows.
• .github/workflows/docs.yml — GitHub Actions workflow. On every push to main: install mkdocs-material, run mkdocs gh-deploy --force, which builds the site to a gh-pages branch.
• docs/CNAME — one line: opentrash.app. Tells GitHub Pages to serve under the custom domain.

First push after this lesson, the docs site rebuilds with the new architecture page — replaces whatever was there before. Take a look at opentrash.app after CI is green.

8 Tests + push

pip install -e ".[dev]"
ruff check .
pytest -q
git add . && git commit -m "Lesson 5: tonnage pipeline + vehicle IDs + docs deploy"
git push origin main

Two workflows now trigger: ci.yml runs tests, docs.yml rebuilds and deploys the site.