[rishsriv]

comext_sql.py now splits a request for more than 5 EU reporter countries into several SQL statements of at most 5 countries each. The FactIQ API is being changed to refuse a single statement that reads trade data from more than 5 Eurostat Comext country schemas, because such statements did not finish inside the 30-second limit. Requests for 5 or fewer countries print exactly the same SQL as before.

The problem: the generator and the guide led clients into time-outs

For a request such as --reporters all, the generator printed one UNION ALL statement over all 27 country schemas. The SQL guide also said that such a union "completes well within the 30-second limit". That is not true: a read of trade data scans every row of each country's data after the start date, even for exact series ids, so the time grows with the number of countries and the date range. Statements over many countries ran past the limit.

The fix: one statement per group of at most 5 countries

• comext_sql.py total, products and trend print one statement per group of at most 5 reporters when a request names more. The groups are as even as possible (27 countries give 6 statements of 5, 5, 5, 4, 4 and 4).
• Each statement has its own line with the schema to run it with.
• The header says to run the statements one after another, not at the same time, and says how to combine the results: add the totals per month, per code or per unit, or put the per-country rows together.
• For products, each statement returns its top 50 codes so that the combined top list can be built from them. The header says that a code missing from some statements has a total that is a lower bound.
• The SQL guide and the bilateral trade report pattern now state the 5-schema limit. The wrong "27 schemas is fast" sentence is gone, and the guide no longer says that an exact-id fetch is fast at any scale.
• Version 0.44.0.

Before and after, same request

Input: comext_sql.py total --reporters de,fr,it,es,nl,pl --partner cn --flow imports --start 2025-01 --end 2025-12 (the SELECT and WHERE lines inside each branch are left out here; they did not change).

Before: one statement over 6 schemas, which the API will refuse.

-- generated by comext_sql.py total: imports vs partner 'cn', 2025-01..2025-12, metric=value
-- run with the run_sql tool, schema="eu_comext_de"
-- all-goods total via the exact cn6_total series; values in EUR
SELECT reporter, SUM(v) AS total_eur
FROM (
  FROM eu_comext_de.data_points
UNION ALL
  FROM eu_comext_fr.data_points
UNION ALL
  FROM eu_comext_it.data_points
UNION ALL
  FROM eu_comext_es.data_points
UNION ALL
  FROM eu_comext_nl.data_points
UNION ALL
  FROM eu_comext_pl.data_points
) u
GROUP BY reporter
ORDER BY 2 DESC;

After: two statements of 3 schemas each.

-- generated by comext_sql.py total: imports vs partner 'cn', 2025-01..2025-12, metric=value
-- all-goods total via the exact cn6_total series; values in EUR
-- the API reads at most 5 reporter schemas in one statement, so this request is 2 statements: run them one after another (not at the same time), each with the schema named above it, then put the reporter rows of all statements together

-- statement 1 of 2: reporters de, fr, it
-- run with the run_sql tool, schema="eu_comext_de"
SELECT reporter, SUM(v) AS total_eur
FROM (
  FROM eu_comext_de.data_points
UNION ALL
  FROM eu_comext_fr.data_points
UNION ALL
  FROM eu_comext_it.data_points
) u
GROUP BY reporter
ORDER BY 2 DESC;

-- statement 2 of 2: reporters es, nl, pl
-- run with the run_sql tool, schema="eu_comext_es"
SELECT reporter, SUM(v) AS total_eur
FROM (
  FROM eu_comext_es.data_points
UNION ALL
  FROM eu_comext_nl.data_points
UNION ALL
  FROM eu_comext_pl.data_points
) u
GROUP BY reporter
ORDER BY 2 DESC;

Details

• Unchanged output: the old and new script print identical output (and identical errors) for 7 requests of 1 to 5 reporters across total, total --monthly, products and trend, including weight and supplementary metrics.
• New tests in tests/test_trade_scripts.py: --reporters all gives 6 statements of at most 5 schemas for each command, each with its own schema line and all 27 countries read once; split products statements return 50 rows each; the domestic-reporter note stays when the partner is itself a member (26 reporters, 6 statements). All 84 tests pass.
• This change can ship before or after the API change. Split statements work with both.

🤖 Generated with Claude Code