Skip to content

Commit 8e57a22

Browse files
authored
Merge pull request #130 from defog-ai/catalog-matches-served-data
List every queryable data schema in the schema reference
2 parents 3476f6c + 9e2c158 commit 8e57a22

1 file changed

Lines changed: 27 additions & 8 deletions

File tree

‎references/data/schemas.md‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,24 @@ States lists the tables to use.
1111

1212
| Schema | Source | Coverage |
1313
|---|---|---|
14-
| `bls` | Bureau of Labor Statistics | Employment (CES), unemployment (CPS, e.g. `LNS14000000`), CPI, PPI, JOLTS job openings, wages, productivity |
14+
| `bls` | Bureau of Labor Statistics | Employment (CES), unemployment (CPS, e.g. `LNS14000000`), CPI, PPI, JOLTS job openings, wages, productivity, import/export prices, consumer expenditures, time use, employment projections, workplace injuries |
1515
| `oews` | BLS Occupational Employment & Wage Statistics | Wages and employment by occupation and metro area |
16-
| `census` | Census Bureau | International trade (incl. `us_census_hs` — monthly imports/exports by HS commodity and partner country; quantity `_qty` series exist only at the 10-digit level, 6-digit lines are value-only), retail, housing, demographics, business formation applications (BFS; industry detail may be incomplete) |
17-
| `bea` | Bureau of Economic Analysis | GDP and components, personal income/spending, regional accounts |
16+
| `census` | Census Bureau | International trade (incl. `us_census_hs` — monthly imports/exports by HS commodity and partner country; quantity `_qty` series exist only at the 10-digit level, 6-digit lines are value-only), retail and wholesale trade, housing starts, construction spending, manufacturing and durable goods, quarterly services, quarterly financial reports, tax revenue, business formation applications (BFS; industry detail may be incomplete), and labor-market tables computed from public-use microdata (CPS, ACS PUMS, CPS ASEC, QWI, J2J, SIPP) |
17+
| `bea` | Bureau of Economic Analysis | GDP and components, personal income/spending, PCE inflation, international transactions and investment position, industry value added, input-output tables, fixed assets, regional accounts |
1818
| `eia` | Energy Information Administration | Petroleum, natural gas, electricity, renewables — production, consumption, prices |
1919
| `ers` | USDA Economic Research Service | Agricultural and food economics |
2020
| `bts` | Bureau of Transportation Statistics | Transportation and freight |
21-
| `filings` | Company filings: US SEC filers (10-K, 10-Q, 8-K, 20-F, 40-F, 6-K) and companies listed in Germany, the UK, France, Switzerland, and India (`issuers.source_schema` = `'german_filings'`, `'uk_filings'`, `'fr_filings'`, `'ch_filings'`, `'in_filings'`; Indian tickers end in `.NS`, e.g. `RELIANCE.NS`) | The archive behind `search_company_filings`. Use the tool for one company; use `run_sql` on `filings` for joins and aggregations across companies. Start from `securities` (`ticker`, `issuer_id`, `is_primary`) and `issuers` (`id`, `canonical_name`). `source_filings` is one row per filing (`issuer_id`, `form_type`, `fiscal_year`, `fiscal_period`, `period_end`, `published_at`, `source_url`). `facts` is one row per extracted value (`filing_id`, `source_series_id`, `source_label`, `metric_class`, `period_start`, `period_end`, `period_basis`, `value`, `unit`, `segment_axis`, `segment_member`): always filter `is_serving AND is_active` (the other rows are superseded or never promoted) and by filing, series, or issuer, because the table holds more than 150 million rows. `source_series` (`source_series_id`, `issuer_id`, `dataset_code`, `title`, `measurement_units`, `frequency`, `is_serving`) and `serving_observations` (`source_series_id`, `time`, `value`, `unit`, `filing_id`) are the series and their points; `dataset_code` separates XBRL statements (`sec_10k`, `sec_10q`, `sec_20f`, `sec_40f`, `sec_6k_interim`), management's forward guidance (`sec_guidance`), and company-specific operating KPIs like ARR/RevPAR/subscribers (`sec_kpi`). `filing_claims` holds MD&A and earnings-release commentary and year-over-year risk-factor changes (`claim_kind`, `ticker`, `fiscal_year`, `fiscal_period`, `section`, `claim_family`, `topic`, `canonical_statement`, `verbatim_quote`, `source_url`; quote only `verbatim_quote`). `filing_risks` is one row per risk-factor heading in an annual report (`cik`, `fiscal_year`, `heading`, `summary`). The remaining tables are pipeline bookkeeping. `get_series(schema="filings", series_id=...)` and `describe_dataset("filings", "<dataset_code>")` also work: they return the same result as with `schema="sec"`, and the dataset codes are the `sec_*` codes listed above. Put a row's `source_url` beside any figure you use; `search_company_filings(format="json")` returns the same links as `source_link` on filing and fact nodes. |
21+
| `frb` | Federal Reserve Board | Interest rates (H.15), money supply (H.6), bank balance sheets (H.8), industrial production and capacity utilization (G.17), consumer credit (G.19), exchange rates (H.10), Financial Accounts (Z.1) |
22+
| `treasury` | US Treasury | Debt to the penny, Monthly Treasury Statement (receipts, outlays, deficit), par yield curve, TIC foreign holdings of US securities |
23+
| `ui_claims` | Department of Labor | Weekly unemployment insurance claims (ETA 539): national and per-state initial and continued claims |
24+
| `fhfa` | Federal Housing Finance Agency | House Price Index: national, census division, state and metro |
25+
| `cbo` | Congressional Budget Office | Historical and 10-year projected federal budget and economic data |
26+
| `irs_soi` | IRS Statistics of Income | Individual income tax return totals (income, AGI, tax), with long historical series |
27+
| `cms_nhe` | Centers for Medicare & Medicaid Services | National Health Expenditures in US dollars by source of funds (Medicare, Medicaid, private insurance, out-of-pocket) and type of service, plus per-capita and share of GDP, from 1960. For cross-country health-spending shares use `health_systems` |
28+
| `us_health` | Centers for Disease Control and Prevention | Measles cases by week (from 2022), by year (from 1962) and by state, measles hospitalizations, deaths and outbreaks, and kindergarten vaccination coverage and exemptions by state and school year. For other countries use `who_gho` |
29+
| `us_immigration` | Department of Labor (OFLC disclosure data) | H-1B labor condition applications and PERM labor certifications (the Labor Department step of an employment-based green card) per employer and federal fiscal year, from 2008: certified, denied and withdrawn, plus H-1B positions certified. One series per employer with at least 10 cases in some year, plus all-employer totals |
30+
| `sec` | SEC EDGAR | The company series behind `search_company_filings` as ordinary `series`/`data_points` tables, for about 2,500 US-listed companies and foreign filers: XBRL statements and segment detail (`sec_10k`, `sec_10q`, `sec_20f`, `sec_40f`, `sec_6k_interim`), forward guidance (`sec_guidance`), operating KPIs (`sec_kpi`), and SIC and NAICS classifications. Statements from 2008 |
31+
| `filings` | Company filings: US SEC filers (10-K, 10-Q, 8-K, 20-F, 40-F, 6-K) and companies listed in Germany, the UK, France, Switzerland, India, Japan, and South Korea (`issuers.source_schema` = `'german_filings'`, `'uk_filings'`, `'fr_filings'`, `'ch_filings'`, `'in_filings'`, `'jp_filings'`, `'kr_filings'`; Indian tickers end in `.NS`, e.g. `RELIANCE.NS`; Japanese and Korean tickers carry the exchange suffix, e.g. `7203.T` for Toyota Motor and `005930.KS` for Samsung Electronics) | The archive behind `search_company_filings`. Use the tool for one company; use `run_sql` on `filings` for joins and aggregations across companies. Start from `securities` (`ticker`, `issuer_id`, `is_primary`) and `issuers` (`id`, `canonical_name`). `source_filings` is one row per filing (`issuer_id`, `form_type`, `fiscal_year`, `fiscal_period`, `period_end`, `published_at`, `source_url`). `facts` is one row per extracted value (`filing_id`, `source_series_id`, `source_label`, `metric_class`, `period_start`, `period_end`, `period_basis`, `value`, `unit`, `segment_axis`, `segment_member`): always filter `is_serving AND is_active` (the other rows are superseded or never promoted) and by filing, series, or issuer, because the table holds more than 150 million rows. `source_series` (`source_series_id`, `issuer_id`, `dataset_code`, `title`, `measurement_units`, `frequency`, `is_serving`) and `serving_observations` (`source_series_id`, `time`, `value`, `unit`, `filing_id`) are the series and their points; `dataset_code` separates XBRL statements (`sec_10k`, `sec_10q`, `sec_20f`, `sec_40f`, `sec_6k_interim`), management's forward guidance (`sec_guidance`), and company-specific operating KPIs like ARR/RevPAR/subscribers (`sec_kpi`). `filing_claims` holds MD&A and earnings-release commentary and year-over-year risk-factor changes (`claim_kind`, `ticker`, `fiscal_year`, `fiscal_period`, `section`, `claim_family`, `topic`, `canonical_statement`, `verbatim_quote`, `source_url`; quote only `verbatim_quote`). `filing_risks` is one row per risk-factor heading in an annual report (`cik`, `fiscal_year`, `heading`, `summary`). The remaining tables are pipeline bookkeeping. `get_series(schema="filings", series_id=...)` and `describe_dataset("filings", "<dataset_code>")` also work: they return the same result as with `schema="sec"`, and the dataset codes are the `sec_*` codes listed above. Put a row's `source_url` beside any figure you use; `search_company_filings(format="json")` returns the same links as `source_link` on filing and fact nodes. |
2232
| `screener` | FactIQ, from SEC filings and Nasdaq's public stock list | Finds companies that match conditions (sector, market value, valuation ratios, margins, growth). One view, `screener.companies`, with one row per US-listed company that files financial statements with the SEC (about 2,400 rows): price and `market_cap`, `price_to_sales`, `price_to_earnings`, `price_to_book`, `ev_to_sales`, latest-twelve-month revenue, income, cash flow and margins, and the latest balance sheet. No `series`/`data_points` tables: filter the view with one `run_sql` query. Prices are delayed about fifteen minutes; state `price_as_of` in the answer. **Read "Company screener" in `references/data/sql-guide.md` before writing SQL.** |
2333
| *(not a SQL schema)* | Earnings-call transcripts, decomposed into a claim graph | `search_earnings_transcripts` targets (selected with `search_target`): management statements (`claims`: `query`, `ticker` for tickers or `company_name` for company names (one or the other, both case-insensitive; `company_filter` is the old name of `ticker` and still works), `quarter_filter`, `claim_family`, claims-only `section`, `detail`, `limit`); Q&A pressure (`pressure_points`: query/company/quarter/linked `claim_family`/detail/limit; section ignored); company-level habits (`disclosure_profile`: direct lookup by the first ticker, not text or quarter search; other filters/detail/limit ignored); and corpus inventory (`coverage`: company/limit, not theme or quarter search; returns `calls_covered`, `earliest_period`, and `latest_period`). Text retrieval ranks strict matches above loose partial-term matches and uses trigram only when FTS is empty. For one-call notes, use coverage → exact quarter → bounded empty-query claims + pressure calls. Claim/pressure rows expose `transcript_id`, `source_block_index`, `qa_turn_id`, and `source_link`; `source_link.source_label` names the company or ticker, fiscal period, and earnings call transcript rather than the ingestion vendor. Quote only `verbatim_quote`, preserving any `[…]` omission marker (it stands for skipped transcript sentences between non-adjacent evidence spans); `canonical_statement` is normalized; neither `analyst_hypothesized` nor `mgmt_declined_to_confirm` is management speech. Put a provided source URL beside any used quote using the provided source label as link text; if it is null, report that the link is unavailable rather than searching for or inventing one. Never use `run_sql` on the gated `transcripts` schema or promise full-transcript/paginated access. See `references/report-patterns/earnings-intelligence.md` |
2434
| *(not a SQL schema)* | Executive podcasts, TV interviews, and conference appearances, plus documents published by central banks and monetary authorities (speeches, interviews, statements, press releases, minutes, press-conference transcripts, testimony, reports) | `search_media_appearances` performs deterministic strict lexical → loose any-term → trigram retrieval over precomputed claims and passage cards; no serving-time model. Use `search`, `claims`, `passages`, or `pressure_points` for timestamped paraphrase findings, `appearances` for video rows, and `coverage` for company-level structured-corpus inventory. `institution` (publisher-name substring) and `country` (ISO code, `EU` for the European Central Bank) restrict every target to official documents; `company` matches the subject of a claim, not its publisher. Dates are publication/upload dates; official documents also carry `event_date`. Never quote `canonical_paraphrase`, query `transcripts` with SQL, or infer absence before checking coverage and bounded synonym searches. See `references/report-patterns/media-intelligence.md` |
@@ -34,8 +44,8 @@ States lists the tables to use.
3444

3545
| Schema | Source | Coverage |
3646
|---|---|---|
37-
| `mospi` | Ministry of Statistics (MOSPI) | CPI (national key: `State: All India`; note the 2012→2024 base change — two separate series families), WPI, IIP, GDP |
38-
| `rbi` | Reserve Bank of India | Banking, credit, money supply, rates, forex reserves |
47+
| `mospi` | Ministry of Statistics (MOSPI) | CPI (national key: `State: All India`; note the 2012→2024 base change — two separate series families) and CPI for agricultural and rural labourers, WPI, IIP, national accounts, Annual Survey of Industries (ASI), survey of unincorporated enterprises (ASUSE), PLFS labour force, energy statistics, gender statistics, higher education (AISHE), RBI handbook tables, and monthly domestic air passengers by airline (DGCA) |
48+
| `rbi` | Reserve Bank of India | Banking, credit, money supply, rates, forex reserves, prices and wages, financial markets, external sector, government finance, company finances, the survey of professional forecasters, and NPCI retail payments (UPI, IMPS, NACH, RuPay and more) |
3949
| `india_trade` | DGCI&S | Monthly imports/exports by HS commodity and partner (6- and 8-digit levels — never sum across levels) |
4050
| `traffic` | Live road-traffic API | Bengaluru, India only — no other city, and no city dimension. 28 fixed road points x 3 metrics (84 series): `current_speed` (km/h), `travel_time` (seconds to cross the segment), `congestion_index` (current travel time / free-flow travel time; 1.0 = free-flowing). Dimensions are `metric`, `location` (point name, e.g. "Silk Board Junction"), `area` (central/east/north/south/west) and `road_type` (junction/ring_road/highway/arterial); `traffic.road_segments` lists each point with its area, road type and coordinates. History starts 2026-02-10 and can never start earlier — the source reports present conditions only, so nothing before that date exists or can be backfilled. Readings are irregular sub-daily snapshots (about every 30 minutes during the 07:00-10:00 and 17:00-21:00 local peaks, roughly hourly otherwise), so gaps between points vary — aggregate rather than assuming a fixed interval. `data_points.time` is UTC; Bengaluru local time is UTC+05:30 |
4151

@@ -61,6 +71,7 @@ States lists the tables to use.
6171

6272
| Schema | Source | Coverage |
6373
|---|---|---|
74+
| `eu` | Eurostat, ECB, EEA | Curated EU and euro-area indicators, ECB policy rates and euro reference exchange rates, EEA greenhouse-gas emissions, and EU ETS registry and revenue data |
6475
| `eu_comext_<iso2>` | Eurostat Comext | One schema per EU member-state reporter; monthly imports and exports by partner and detailed CN8 product, with trade value in euros and separate weight or supplementary-quantity series |
6576

6677
Use the reporter's lowercase ISO2 code: `eu_comext_de` for Germany,
@@ -113,10 +124,14 @@ The `uk` schema holds six datasets (filter on `dataset_code`):
113124
|---|---|---|
114125
| `imf` | International Monetary Fund | Cross-country macro indicators, plus the earlier releases of the forecast publications (World Economic Outlook, Fiscal Monitor, the Regional Economic Outlooks, COFER) — see "IMF past releases" below |
115126
| `worldbank` | World Bank | Development and macro indicators by country |
116-
| `singstat` | Singapore Department of Statistics | Singapore national statistics |
127+
| `singstat` | Singapore Department of Statistics | About 1,050 tables across the economy, society, industry, trade and population |
128+
| `sg_health` | Singapore Ministry of Health | Weekly infectious-disease case counts and average daily polyclinic attendances for acute conditions, from 2015 |
129+
| `bis` | Bank for International Settlements | Central-bank policy rates for 49 economies (monthly, some from the 1940s, and daily), debt service ratios of households and companies for 32 economies, and total credit to the non-financial sector by borrower for 44 economies |
130+
| `who_gho` | WHO Global Health Observatory | Headline WHO indicators, one value per country and year for about 240 countries: childhood immunization, maternal and neonatal mortality, tobacco use, life expectancy, hospital beds, UHC service-coverage index |
131+
| `health_systems` | WHO, Eurostat | Cross-country health-system detail: health-financing shares (out-of-pocket, government, private, external), doctors and hospital beds per 10,000 people, UHC and cataract surgical coverage, Eurostat hospital use, and burden of disease 2000–2021 (DALY, YLL, YLD by age, sex and cause) |
117132
| `portwatch` | IMF PortWatch (satellite-AIS) | Daily shipping: transit calls + trade capacity for 28 chokepoints (Suez, Hormuz, Malacca…), port calls + import/export volume estimates for 196 countries and 2,065 ports, 2019→, refreshed weekly (data runs a few days to a week behind) |
118133
| `nasa_fires` | NASA FIRMS (VIIRS 375 m) | Every active-fire detection worldwide since 2012-01-20, one row per satellite pixel, plus pre-computed daily totals per country, per state, and per 0.1° cell. Refreshed daily. **Shaped unlike every other schema — read "The nasa_fires schema" below before writing SQL.** Most fire questions need no SQL at all: use `get_geo_data` (see `references/data/satellite.md`) |
119-
| `satellite` | NASA / CNES satellite-derived | Monthly nighttime lights by country + state (economic-activity proxy, Asia focus, 2019→); lake & reservoir water levels from radar altimetry (650 water bodies incl. 88 Chinese, 15 major Indian reservoirs, 1990s→, per-overpass). Water-level stations come in two grades (`grade` dimension): `operational` updates ~weekly; `research` stations are frozen scientific archives (many end 2020-22) — always check the series `end_time` before presenting a level as current, and filter to operational for live readings |
134+
| `satellite` | NASA / CNES satellite-derived | Monthly nighttime lights by country + state for 16 Asian economies (economic-activity proxy, 2019→); lake & reservoir water levels from radar altimetry (about 730 water bodies in 98 countries, incl. 88 Chinese and 20 Indian, 1992→, per-overpass). Water-level stations come in two grades (`grade` dimension): `operational` updates ~weekly; `research` stations are frozen scientific archives (many end 2020-22) — always check the series `end_time` before presenting a level as current, and filter to operational for live readings |
120135

121136
## IMF past releases
122137

@@ -193,6 +208,10 @@ another dataset, or splitting day passes from night passes.
193208

194209
- US labor/inflation → `bls` (plus `oews` for occupation-level wages)
195210
- US GDP/income → `bea`; US trade → `census`
211+
- US interest rates, money and credit → `frb`; federal debt, receipts and outlays → `treasury` (projections → `cbo`); weekly jobless claims → `ui_claims`; house prices → `fhfa`
212+
- Health → `who_gho` or `health_systems` across countries; `cms_nhe` and `us_health` for the US; `sg_health` for Singapore
213+
- H-1B or green-card sponsorship by employer → `us_immigration`
214+
- Policy rates and credit across many countries → `bis`; euro area → `eu`
196215
- Energy anything → `eia`
197216
- India macro → check BOTH `mospi` and `rbi`
198217
- Trade-war / commodity-flow stories → `census` + `china_customs` +

0 commit comments

Comments
 (0)