Tennis Twin — Build Log & Technical Overview

Prepared for the USTA Innovation Challenge review team.

Tennis Twin is an AI-assisted player-development platform built around one idea: one data spine for the whole player journey — playground to podium. It finds each player's closest competitive "twins," benchmarks them against the players one level above, surfaces rising talent, maps court-access equity against the mission of 35 million players by 2035, and — at the elite end — turns US Open Hawk-Eye tracking into shot-level intelligence. All from the USTA datasets provided for the challenge.

The product is organized as four connected pillars:

1. Find your twin — explainable similarity matching.
2. See your gap — peer-benchmarked path to the next level.
3. Spot rising talent — a junior talent radar plus "hidden gems" (fast risers in the most court-constrained states).
4. Pro Shot IQ — serve maps, court coverage, pressure performance and next-gen stats from Hawk-Eye, alongside a full rally replay.

The narrative and business framing live on the in-app Vision page; this document walks through every step I took, from first opening the raw data to a working prototype, including the decisions (and honest trade-offs) I made along the way.

1. Starting point

I began with the original Tennis Twin proposal: a "player twin" for every athlete, similarity models, progression prediction, personalized development roadmaps, talent identification, and an LLM narrative layer.

Before writing any code, I did a ground-truth assessment of the five datasets USTA provided, because a concept is only as good as the data behind it.

2. Data assessment (what I actually have)

All five files are CSV and share a common person_key join key.

Key findings that shaped the build

1. Ratings are a single snapshot per person, not a time series. 1,045,074 rows for 1,045,071 unique people. ~71% (744,955) have no NTRP level at all. There is no rating history — I cannot watch a player go 3.5 → 4.0 over time.
2. WTN has far better coverage than NTRP. 800,376 people have a WTN singles level vs ~300,000 with NTRP. WTN is also continuous (1–40, lower = stronger) and ships with a confidence score and last-played date. I made WTN the primary rating and NTRP a secondary label.
3. Match history is 2025 only. One season, 464k players. Rich intra-season signal, but no multi-year arc.
4. Junior rankings are weekly but only within 2025 (~50 snapshots). Great for intra-year momentum; not a 12-year-old growing into a college recruit.
5. There is no "success" label — nothing indicates who reached college, pro, or even who advanced a level.
6. No Hawk-Eye or US Open match tracking is present in the five core files. (USTA later provided a separate single-match Hawk-Eye tracking dataset, which I built out as the Match Lab module — see §8.)

NTRP distribution (ratings file)

3. The pivot decision

The original proposal leans on a longitudinal spine: "learn from players who followed the path to success," "predict future rating growth," "juniors who ultimately reached college/pro." That framing needs multi-year per-player trajectories and an outcome label. The data has neither.

Rather than fabricate forecasts a technical judge could catch, I pivoted from time-travel to peer comparison — keeping the Tennis Twin name, dashboard, and spirit, but grounding every insight in defensible, cross-sectional data:

Framing I use with reviewers: "personalized, peer-benchmarked development guidance," not "predicts your future rating." True longitudinal forecasting is noted as future work that unlocks with historical data.

4. Architecture

The raw CSVs total ~4 GB — far too large to touch live in a browser/Next.js app. So the system has two clean stages:

 dataSourceFiles/*.csv  (~4 GB, git-ignored)
        │
        ▼   pipeline/run.sh  (DuckDB, offline, ~7 seconds)
 data/derived/*.parquet + tennis.duckdb  (~42 MB, compact feature tables)
        │
        ▼   Next.js Route Handlers  (@duckdb/node-api, read-only)
 Tennis Twin dashboard  (React / Tailwind)

Tooling: DuckDB 1.5.4 (CLI + @duckdb/node-api 1.5.4), Next.js 16.2.4 (App Router, Turbopack), React 19, Tailwind 4.

I chose DuckDB because it reads the multi-GB CSVs directly, does the heavy feature engineering in SQL in seconds, and the same engine serves the app.

5. The offline pipeline (pipeline/build.sql)

Reads the raw CSVs as views (never copies 4 GB into the DB) and materializes small derived tables. Feature engineering includes:

• Per-player match aggregates (2025): matches played/won, win %, singles vs doubles, tournament vs league mix, distinct events/facilities, first/last match, weeks active.
• Opponent quality (singles self-join on match_key): average opponent WTN (strength of schedule), "quality wins" (beating a stronger/lower-WTN opponent), matches vs stronger opponents, upset win rate.
• Clutch record: 3-set (deciding-set) win rate.
• Junior momentum: each junior's primary ranking list, weeks ranked, best/last rank, and season rank change.
• Master players table: one row per person (1.05M) joining demographics + ratings + all derived features.
• Level benchmarks: metric distributions (median + mean) per gender × NTRP level and per gender × WTN band — the engine behind gap analysis.
• Facility access by section/district: court counts and surface availability.

Outputs (in data/derived/, all git-ignored):

Runtime: ~7 seconds end-to-end. Output size: ~42 MB (from ~4 GB raw).

6. Validation

The pivot is only worth building if the features genuinely separate skill levels. They do. Median WTN and opponent strength track NTRP cleanly (male players):

Twin matching also passes the eye test: a 39-year-old male at WTN 27.2 with 21 matches (38% wins) matches to 37–40-year-olds at WTN ~27 with similar volume and win rates.

Headline dataset stats (from the app summary)

• 1,054,741 total players
• 344,342 active in 2025
• 800,356 with a WTN rating
• 300,104 with an NTRP rating
• 88,692 ranked juniors
• 47,755 facilities
• Median matches for an active player: 8

7. The application

Data-access layer (lib/)

• db.ts — cached, read-only DuckDB connection; normalizes DuckDB BigInt/DATE values so results are JSON-serializable.
• queries.ts — typed query functions (summary, player, featured, search, twins, gap report, talent).
• insights.ts — rule-based coaching narrative. This is the deliberate seam: swap it for an LLM call with the same structured inputs and nothing else changes.
• format.ts, types.ts — client-safe formatting + shared types.

Twin-matching engine

A transparent, weighted k-NN computed in SQL. Distance is a normalized, weighted sum over WTN (highest weight), age, win rate, match volume, tournament mix, and strength of schedule — within the same gender and a soft age window. Every term is human-readable, so matches are explainable (a requirement of the proposal). Distance is mapped to an intuitive 0–100 similarity score.

Gap-to-next-level

Compares a player against the peer group one level above (NTRP + 0.5 where NTRP exists, otherwise the next WTN band) across win rate, matches played, weeks active, opponent strength, clutch win rate, and upset win rate.

Talent radar

Two views over ranked juniors, both normalized to a within-list percentile (0 = bottom of the list, 100 = #1) so lists of very different sizes are comparable: Rising (biggest full-season percentile climb) and Elite (highest season-best percentile). Raw rank and list size are shown for context.

Twin similarity visualization

An SVG scatter plots each twin on WTN (ability) × win rate, sized by similarity, with the selected player highlighted — visualizing the twin cluster. Points are clickable to pivot to that twin.

AI insight generation

lib/insights.ts produces a deterministic rule-based narrative. When OPENAI_API_KEY is set (lib/llm.ts, any OpenAI-compatible endpoint), the app generates the narrative with an LLM instead, using a system prompt that enforces the honest peer-benchmark framing (no fabricated forecasts). It falls back to the rule engine on any error, and the UI badges which source produced each insight.

Court access

A section-level view of the facilities dataset (courts, indoor/outdoor, surface availability, % private) supporting the "democratizing the pipeline" theme.

API (App Router Route Handlers)

• GET /api/summary
• GET /api/players — featured players, or filtered search (gender, ntrp, section, q = player-ID prefix, juniorsOnly)
• GET /api/players/[id] — profile + twins + gap report + insights
• GET /api/talent?sort=rising|elite

Dashboard UI

A single responsive dashboard: headline stats, a player picker with filters, a player-detail view (profile card, AI-insight panel, gap comparison chart, clickable twins grid), and the talent radar.

8. Match Lab — US Open Hawk-Eye rally tracking (added dataset)

Partway through the build, USTA provided an additional dataset: Hawk-Eye "Tennis Rally" tracking feeds for a single US Open 2025 men's singles match (anonymized as Player 11 vs Player 31, Arthur Ashe Stadium, best-of-five). This is an entirely different kind of data from the five core CSVs — granular per-rally 3D tracking rather than population-level records — so I built it as a separate "Match Lab" module rather than forcing it into the twin/benchmark engine (the metrics don't align 1:1, and there are no shared player identities).

What's in it

~310 MB of pretty-printed JSON across five synchronized feeds, all in real-world court coordinates (metres), documented in the accompanying feed spec PDF:

Stats derived from it: 490 tracked rallies, avg rally 2.4 shots (max 20), serve speeds 40–138 mph, 31 aces, ending on a match-point ace.

Offline pipeline (pipeline/build_match.mjs)

Same philosophy as the DuckDB pipeline — precompute offline, ship something small:

• Auto-detects the match folder, dedups revisions (604 raw files → 490 distinct rallies), and filters ball-kids/officials out of the tracking.
• Rounds coordinates to the centimetre and links shots ↔ bounces by event ID.
• Emits static, CDN-friendly files to public/match/: a 0.26 MB index.json (match metadata + one entry per rally) and 394 per-rally trajectory files (the rallies that carry ball tracking). Total ~6 MB, down from ~310 MB raw.

Because these are static JSON, Match Lab needs no database or server function — the page fetches the index up front and lazy-loads a rally's trajectory on demand.

The feature (/match-lab, linked from the dashboard)

• Match header + stat strip — players (handedness/height/nationality), court, tracked points, aces, top serve, average rally.
• Point browser — every tracked rally, filterable by All / Rallies / Aces / Key points and by set, each row showing score context, outcome, serve speed, and badges (BP / SP / MP / GP / TB / ACE).
• Animated court — a top-down court rendering the 50 Hz ball + player tracks: ball sized by height with a flight trail, players colored by team (server ringed), bounce markers colored by the hitting player, plus play/pause, scrub, and a wide range of playback speeds (0.25× up to 3×).
• Auto-play the list — honestly, half the reason this got built was that the tracking data is just fun to watch. So there's an Auto-play toggle: when a rally finishes it automatically loads and plays the next rally in the current filter, so you can lean back and effectively watch the whole match unfold point by point (or filter to "Aces" and watch just those, etc.). The list auto-scrolls to follow along and shows a current / total position counter.
• Shot table — per-shot stroke, placement, speed, spin, and line call.

Caveats (honest framing)

• It is one anonymized match. Match Lab is a shot-intelligence showcase, not a population study — no player identities, no cross-match trends. It intentionally stands apart from the twin/benchmark engine.
• Rendering uses the sampled ball/player positions, not the higher-fidelity polynomial motion arcs — a possible future refinement for perfectly smooth curves.
• Coordinates are reconstructed onto a standard court from the Hawk-Eye origin (net centre); any tiny rendering offset is cosmetic, not analytical.
• If more matches arrive, the pipeline already auto-detects the match folder; multi-match support is a small extension.

9. Data-quality decisions worth flagging

1. "Tournament share" was dropped as a metric. The provided play history is ~99% league/team play — only 7,388 tournament matches exist across 6.85M rows (Team League 4,113,970; Flex League 53,998; Tournament 7,388). A tournament ratio is ~0 for nearly everyone, so I replaced it with weeks active, which genuinely varies by level. Note: this is fundamentally a league dataset, not a tournament dataset.
2. Benchmarks use the mean for engagement metrics. Median tournament ratio and upset rate are 0 for most players, so "next-level target" uses the level mean for those, keeping the target meaningful.
3. WTN over NTRP as the primary rating (see §2) — better coverage, continuous, confidence-scored.
4. Names are anonymized in the source, so the UI shows deterministic handles like "Player 1D7F88" derived from the masked person_key.

10. Known limitations & honest framing

• No forecasting. Insights are peer-benchmarked comparisons using a single-season, single-snapshot dataset — explicitly not predictions of a player's future.
• Talent momentum is now normalized to within-list percentiles, which removes the earlier "entry effect" distortion (juniors joining a ~120k list at a poor rank produced misleadingly huge raw rank deltas). Rising climbs now read as, e.g., "+92.7 percentile points, season best #590 of 29,210."
• Tournament-based insights are limited by the near-absence of tournament matches in the data.
• Hawk-Eye / US Open tracking was provided later — but only for a single anonymized match — and is built out as the separate Match Lab module (§8), a shot-intelligence showcase rather than a population study. No broadcast video was provided.

11. How to run

# 1. Build the derived feature tables from the raw CSVs (requires DuckDB CLI)
brew install duckdb          # one-time
./pipeline/run.sh            # ~7 seconds, writes data/derived/

# 2. (Optional) Rebuild the US Open Match Lab data from the raw Hawk-Eye feeds
node pipeline/build_match.mjs   # writes public/match/ (~6 MB, committed)

# 3. Install and start the app
npm install
npm run dev                  # http://localhost:3000

Optional — enable LLM-generated insights: set OPENAI_API_KEY (and optionally OPENAI_BASE_URL / TENNIS_TWIN_LLM_MODEL, default gpt-4o-mini) before starting the app. Without a key, insights use the deterministic rule engine and everything still works.

The raw datasets live in dataSourceFiles/ and are git-ignored (multi-GB); the DuckDB-derived outputs in data/derived/ are also git-ignored and regenerated by the pipeline. The much smaller Match Lab output in public/match/ (~6 MB) is committed, since it is served as static assets and the raw Hawk-Eye feeds are git-ignored.

12. Roadmap

Recently completed: within-list percentile normalization for talent, an LLM-swappable insight seam, the twin similarity scatter, the Match Lab US Open rally replay (§8), and — most recently — the pillars that connect the whole pathway:

• Shot IQ (Match Lab): serve-placement zones, court-coverage heatmaps, serve+1, and pressure/break-point performance, all precomputed from the Hawk-Eye tracking into a compact public/match/insights.json.
• Access & growth map: a national dot-density map drawn straight from facility coordinates, an "active players per court" equity ranking by state, and hidden gems — the fastest-rising juniors in the most court-constrained states.
• Player-vs-twin radar across the full feature set.
• A Vision page framing the playground-to-podium spine and business model.

Next:

• Match Lab: multi-match support and computer-vision ingestion of broadcast feeds.
• Doubles-specific opponent-quality modeling (currently singles-based).
• With multi-year historical data: true longitudinal trajectory forecasting (a natural fit for an IBM watsonx-backed enterprise data layer).