CT Property Pulse

A civic-data pipeline that runs unchanged on CPU or GPU.

1.19 million Connecticut real-estate transactions (2001–2024), joined to municipal mill rates, aggregated on an NVIDIA GPU via cuDF, and explained in plain English by Gemma — with every generated number verified against the computed figures.

Built for the Google Cloud × NVIDIA Golden Ticket contest. Learning path: Speed Up Data Analytics on GPUs.

Per stage: 44.0× where it vectorises, 0.03× where it does not.	The gap widens with row count — 6.7× at 1.2M, 16.4× at 4.8M.

Just want to see it run? CT_Property_Pulse_executed_L4.ipynb is the notebook as executed on an NVIDIA L4 — every cell with its real output, the timings, the correctness check, the charts and the generated briefs. Nothing to install.

---

The problem

Connecticut publishes two datasets that belong together and never are:

Dataset	ID	Rows
Real Estate Sales 2001–2024	5mzw-sjtu	~1,189,000
Mill Rates FY2014–2026	emyx-j53e	~5,100

The sales file says what properties sold for. The mill-rate file says what each town taxes per $1,000 of assessed value. Neither answers the question a resident actually has — what does the median home in my town cost to own? — because answering it means cleaning a million-row file, filtering out the transfers that aren't market sales, aggregating by town, year and property type, and joining across 169 municipalities and 24 years.

That is a real data-engineering workload on a dataset small enough to fit on a free Colab T4. Which makes it a good place to test the claim the learning path makes.

The claim being tested

Accelerate analytics for large data sets on Google Cloud by tapping into NVIDIA GPUs — no code changes required.

So ct_pipeline.py contains zero GPU code. No CUDA, no device management, no .to_gpu(). It is ordinary pandas. The same file is executed twice:

python                ct_pipeline.py --out results_cpu.json    # CPU pandas
python -m cudf.pandas ct_pipeline.py --out results_gpu.json    # NVIDIA cuDF

Everything else in this repo is measurement.

Architecture

data.ct.gov ──> prepare_data.py ──> parquet
                                      │
                                      ▼
                            ct_pipeline.py  (12 timed stages)
                            ├── same source file
                            ├── run 1: CPU pandas
                            └── run 2: python -m cudf.pandas
                                      │
                     ┌────────────────┼────────────────┐
                     ▼                ▼                ▼
              fingerprint      per-stage timings   aggregate tables
              (must match)     (the benchmark)           │
                                                         ▼
                                              gemma_briefs.py
                                              ├── fact sheet from computed figures
                                              ├── Gemma writes the prose
                                              └── verify_brief() rejects any
                                                  number not in the fact sheet

Results

Hardware: NVIDIA L4  |  Rows in: 1,188,997  |  After cleaning: 1,174,232  |  Street groups: 62,694

Result fingerprints matched across engines (a30c81e53f704083): same keys, same row counts, same integer columns. Values are checked separately by compare_artifacts.py, which compares keys, integer columns and NaN patterns exactly and floats to rtol=1e-9; this run passed.

Stage	CPU (s)	GPU (s)	Speedup
string normalize	6.24	0.14	44.03×
dedupe	0.88	0.05	17.70×
aggregate street level high-cardinality groupby	0.62	0.10	6.25×
aggregate town year	0.50	0.21	2.40×
rolling trend groupby.rolling - may fall back	0.29	0.14	2.12×
load	0.72	0.38	1.90×
clean types	0.44	0.29	1.53×
feature engineer	0.23	0.44	0.52× ⚠️
join mill rates	0.01	0.05	0.29× ⚠️
row apply known CPU fallback - by design	0.35	2.10	0.17× ⚠️
rank towns	0.01	0.04	0.15× ⚠️
town similarity	0.02	0.62	0.03× ⚠️
Total	10.31	4.56	2.26×
Total (vectorised only)	9.96	2.46	4.05×

⚠️ marks stages that ran slower on the GPU. row_apply is a row-wise Python lambda and cannot be compiled to a kernel - it is in the pipeline deliberately, to show where the boundary is.

Scaling study

Rows	CPU (s)	GPU (s)	Speedup
1,188,997	10.07	1.50	6.71×
2,377,994	19.92	1.76	11.33×
4,755,988	40.22	2.45	16.42×

Replicated from the base dataset; dedupe and the high-cardinality groupby still do real work at every scale.

The correctness check runs before any timing is reported, in the notebook and in fill_readme.py alike. It deliberately does not ask for bit-identical output: float64 addition is not associative, a GPU reduction sums in a different order, and a hash over rounded values reports "different" about numbers that agree to every printed digit. So the fingerprint covers structure — keys, row counts, integer columns — and compare_artifacts.py compares the floats to a stated tolerance and reports the worst deviation as a number.

What the data says

Arms-length sales only — foreclosures and family transfers are filtered out, which is most of what makes a naive read of this dataset wrong.

The question this started with: the median West Hartford home sold for $564,100 in 2024, up 54% in five years, on a 44.78 mill rate — about $17,680 a year in property tax, or 3.13% of the sale price. Gemma writes one of these for each of the twelve busiest towns; every figure in every brief is computed by the pipeline, and checked.

What I learned

"No code changes required" is true, and that is the surprising part. python -m cudf.pandas in front of an existing script is the whole integration. For anyone with a pandas codebase already in production, that is a low-risk thing to try on a Friday afternoon.

The accelerator is only as good as the pandas you wrote. The single biggest win in this project was not the GPU. It was replacing agg(lambda s: s.quantile(0.25)) with groupby.quantile(0.25) — identical output, but a Python callback per group became one native kernel, and the aggregation stage dropped ~40× on CPU alone. Idiomatic vectorised pandas is the prerequisite; the GPU multiplies it. Write a Python lambda and you get a silent CPU fallback and no speedup at all.

Know where the fallback boundary is. 12_row_apply is in the pipeline deliberately and never accelerates — a row-wise Python lambda cannot be compiled to a kernel. groupby().rolling() sits near the boundary too. Timing every stage separately is what makes this visible; one end-to-end number would have averaged the truth away.

Small data does not need a GPU. At the base 1.19M rows several stages sit near parity, because fixed overhead (host-to-device transfer, kernel launch) dominates. The gap opens with volume. Recommending a GPU for a 50k-row job would be bad engineering advice, and the scaling chart is there to show where the crossover actually is.

The right division of labour with an open model. Gemma never sees a row of data. It sees a dozen figures cuDF computed and writes a paragraph. Then verify_brief() checks the result. A brief that fails is rejected, not published — the minimum bar for civic data, where a wrong tax number is worse than no output at all.

Grounding every number is necessary and nowhere near sufficient — this was the finding I did not expect. The first version of verify_brief() did what the phrase "numerically grounded" usually means: it pulled every number out of the generated text and asserted that we had supplied it. On the twelve briefs from run 20260903-125141 it passed nine and rejected three. Both numbers were wrong.

The three rejections were false positives — all of them the year 2001, which the prompt itself mentions and the fact sheet did not contain. Meanwhile the nine that passed included these:

"Property tax rates increased by 44.78%." — West Hartford "Waterbury's property tax rate is 44.98%." "The town's tax rate is 32.0%." — Norwalk

44.78 is West Hartford's mill rate: dollars of tax per $1,000 of assessed value, about 4.5%, and it did not increase by anything. Every figure in those sentences traces back to the fact sheet. Not one of them means what the sentence says it means. Gemma 3 1B was not hallucinating numbers — it was taking real ones and attaching them to the wrong concept, which a traceability check cannot see by construction.

Two changes followed, and they generalise past this project:

1. Verify units, not just values. Fact keys carry their type in the name (_pct, _price, _count, mill_rate), so a figure written with a % that matches only a dollar or mill-rate fact is a unit mismatch and is rejected. Replayed against those same twelve briefs, this catches all four mill-rate-as-percentage errors — three of which the first version had passed — and drops the false positives to zero.
2. Remove the ambiguity upstream. A 1B model asked to relate a tax figure to a price will reach for whichever percentage is nearest. So the fact sheet no longer offers a choice: it supplies a pre-computed effective_tax_pct, labels the mill rate with its unit and states that it is not a percentage, and names the five-year-ago price as the starting point rather than a bare "median price in 2019" — which had produced briefs saying prices "reached" the earlier, lower figure.

python gemma_briefs.py --selftest exercises both layers against the real failing sentences — including the one that matters most, a correctly written mill rate ("$44.98 per $1,000 of assessed value"), because the first draft of the unit rule rejected it. A mill rate really is denominated in dollars; the falsehood is calling it a percentage, not putting a dollar sign on it.

After both changes, the regenerated briefs pass 12 of 12, and the share-of-price figure is right everywhere it appears — 1.9% in Stamford, 4.83% in Hartford, 0.84% in Greenwich. Replayed against the old briefs, the same verifier passes 8 of 12 and names all four mill-rate errors, three of which the first version had waved through, with no false positives.

What neither layer catches is a false claim built from a correctly-united number. In the new set, Danbury's brief says "a significant increase of 73.5% occurred in home sales" — 73.5% is the change in price, not in sales. Every figure is real and correctly united, and the sentence is still wrong. So: these briefs are checked for invented figures and for unit errors. That is two named things, not a guarantee, and I would not call them fact-checked.

Running it

Colab (recommended). Open CT_Property_Pulse.ipynb, set Runtime → Change runtime type → T4 GPU, and run all. The notebook is self-contained — it writes out every module it needs.

Locally, with a CUDA GPU:

pip install --extra-index-url=https://pypi.nvidia.com cudf-cu12
pip install -U transformers accelerate pandas pyarrow matplotlib

python prepare_data.py --data-dir data
python                ct_pipeline.py --data-dir data --out results_cpu.json
python -m cudf.pandas ct_pipeline.py --data-dir data --out results_gpu.json --artifacts artifacts
python gemma_briefs.py --artifacts artifacts --out briefs.json

On Google Cloud, from the terminal. No console clicking. This uses Colab Enterprise notebook executions: an L4 is provisioned, the notebook runs to completion, results are written to GCS, and the runtime tears itself down — so there is no idle clock to forget about.

gcloud auth login && gcloud config set project YOUR_PROJECT
./gcp_run.sh --dry-run      # inspect the plan
./gcp_run.sh                # provision, run, retrieve
python fill_readme.py --results-dir gcp_results/<timestamp>
./gcp_teardown.sh           # remove template + secret

Requires GPUS_ALL_REGIONS and NVIDIA_L4_GPUS quota of at least 1 — both default to 0 on a new project, even a paid one, and must be requested explicitly.

Offline / CI. make_fixture.py generates a synthetic dataset with the identical schema, so the pipeline runs with no network and no portal dependency:

python make_fixture.py --data-dir data_fixture --rows 200000
python ct_pipeline.py --data-dir data_fixture --out results_fixture.json

Gemma is a gated model — accept the licence once at huggingface.co/google/gemma-3-1b-it and supply a read token.

Files

File	What it is
CT_Property_Pulse.ipynb	The notebook. Self-contained; run this.
CT_Property_Pulse_Enterprise.ipynb	Colab Enterprise variant: adds the cudf.pandas profilers, a GCS round-trip, and cleanup.
CT_Property_Pulse_executed_L4.ipynb	The Enterprise notebook as executed on an L4, outputs intact. Read this to see the results without running anything.
gcp_run.sh	Provision an L4, run the notebook headlessly, retrieve results. All from the terminal.
gcp_status.sh	Check, watch and retrieve a running execution. Submitting and collecting are separate on purpose.
gcp_teardown.sh	Remove everything gcp_run.sh created.
fill_readme.py	Writes the results table, the scaling table and the figure captions above from real benchmark JSON.
ct_pipeline.py	The pipeline. Zero GPU code, 12 timed stages.
prepare_data.py	Downloads the two CT datasets, caches as parquet.
gemma_briefs.py	Fact sheets → Gemma → verification of values and units. --selftest exercises both.
make_fixture.py	Synthetic dataset with the real schema, for offline runs.
build_notebook.py	Assembles the notebook from the modules so they can't drift.

Reuse this for your state

Every US state runs a Socrata portal with the same API shape. To point this pipeline at yours, change two URLs and the column map at the top of prepare_data.py:

SALES_URL = "https://data.<your-state>.gov/api/views/<4x4>/rows.csv?accessType=DOWNLOAD"

The benchmark harness, the fingerprint correctness check, the grounding verifier and the charts are all portal-agnostic.

Licence

MIT for the code. The underlying datasets are published by the State of Connecticut under its open-data terms.