ETF constituent betas

etf_constituent_betas.py reads the equity rows from an iShares-style holdings CSV, downloads adjusted daily closes from Yahoo Finance, and computes stock betas against one or more ETFs.

Running configured analyses

run_analyses.py provides a cross-platform replacement for local batch and PowerShell launchers. Its checked-in analyses.toml manifest contains suitable arguments for the project's command-line research programs and an enabled switch for every job. List or preview jobs before starting them with:

python run_analyses.py --list
python run_analyses.py --dry-run
python run_analyses.py --only beta-stability kfas-dynamic-betas --dry-run

Without a selection option, the runner executes manifest entries whose enabled value is true. --only runs named entries even when disabled, --all selects every entry, and --skip removes named entries from either selection. Computationally expensive walk-forward and state-space jobs are disabled by default.

Each subprocess runs from the project directory. Python jobs use the interpreter that launched the runner, while R jobs use Rscript or the executable supplied with --rscript. Output is saved to timestamped files under logs/; add --show-output to stream it simultaneously or --no-logs to suppress logs. The runner stops at the first failure by default. Use --continue-on-error to complete the remaining selection, --timeout SECONDS to override manifest timeouts, and --dry-run to make no changes.

python .\etf_constituent_betas.py .\TOPT_holdings.csv --etfs VOO XLE --days 252 --multiple

Several holdings files can share one merged price download while retaining fund-specific calculation outputs:

python .\etf_constituent_betas.py .\IVV_holdings.csv .\TOPT_holdings.csv `
  --etfs VOO --days 63 126 252

For a multi-file run, merged price files use the joined holdings stems (for example IVV_holdings__TOPT_holdings_adjusted_close.csv). Beta, volatility, and idiosyncratic-volatility result files continue to use each individual holdings file's stem.

--days accepts one or more daily return windows. For example, --days 63 126 252 downloads enough data for the 252-day window and calculates all three windows from that one dataset. Up to one additional price observation is needed for each window. Univariate results are sorted in ascending beta order within each window and ETF. The reported grid uses [-1, 0, 1, 2] as cut points, including underflow and overflow bins: < -1, [-1, 0), [0, 1), [1, 2), and >= 2. Console output reports the actual sample-date coverage for every calculation; constituent result CSVs contain each row's exact start_date and end_date.

Outputs go to output/ by default:

• <holdings>_adjusted_close.csv: all adjusted-close data downloaded
• <holdings>_constituent_adjusted_close.csv: constituent adjusted closes only
• <holdings>_etf_adjusted_close.csv: factor ETF adjusted closes only
• <holdings>_univariate_betas.csv: sorted univariate betas and observation counts
• <holdings>_beta_grid_counts.csv: counts by ETF and beta interval
• <holdings>_multiple_betas.csv: optional joint-regression results

Use --no-save-data to suppress the downloaded-price CSV, --output-dir to choose another directory, and --end-date YYYY-MM-DD for a historical run. The end date is exclusive; by default the script uses today and therefore excludes the possibly incomplete current trading session.

To rerun the calculations without downloading data, use the default combined price file:

python .\etf_constituent_betas.py .\TOPT_holdings.csv --etfs VOO XLE --days 252 --multiple --use-saved-data

An explicit saved CSV can also be supplied:

python .\etf_constituent_betas.py .\TOPT_holdings.csv --etfs VOO XLE --use-saved-data .\archive\prices.csv

Historical and idiosyncratic volatility calculations reuse the same price data:

python .\etf_constituent_betas.py .\TOPT_holdings.csv --etfs VOO XLE `
  --days 63 126 252 --vol --idiovol both --use-saved-data

--vol produces annualized historical constituent volatility. --idiovol by itself calculates both modes; it also accepts an explicit univariate, multiple, or both. The multiple calculation uses all factor ETFs and does not require --multiple. With only one factor ETF, the redundant multiple result is omitted and the calculation is reported as univariate. Console summaries are annualized percentages, while CSV values are annualized decimals. Change the default square-root-of-252 annualization with --annualization-days.

Additional optional outputs are:

• <holdings>_historical_volatilities.csv
• <holdings>_univariate_idiosyncratic_volatilities.csv
• <holdings>_multiple_idiosyncratic_volatilities.csv

Use --sector to add sector classifications to constituent result files and to print beta and requested volatility statistics separately for every sector. It also writes <holdings>_sector_statistics.csv in long format; values in that CSV retain their source units (betas or annualized decimal volatility).

Standalone exponentially weighted betas

ewma_betas.py calculates exponentially weighted betas for an arbitrary set of assets without requiring an ETF holdings file or performing sector pooling:

python ewma_betas.py XLE XLP XLK --benchmark IVV --lookbacks 63 126 252 --history-statistics

Symbols may instead come from repeatable --symbols-file options. If neither positional symbols nor symbol files are supplied, --prices-file supplies all its columns except the benchmark. Missing price columns and history are filled through the normal saved-price cache unless --no-download-missing is used.

Each lookback is an equivalent SMA window: EWMA alpha is 2 / (N + 1), decay is 1 - alpha, and the limiting effective sample size is N. The default 90% coverage threshold delays estimates until their effective sample size reaches at least 90% of N; change it with --coverage.

The normal <name>_ewma_betas.csv contains the latest beta, approximate standard error, effective observations, decay, and half-life for every asset and lookback. --history-statistics additionally writes <name>_ewma_beta_history_statistics.csv, with one row per asset and lookback containing the first and last estimate dates, number of daily beta estimates, last beta, mean, sample standard deviation, minimum, maximum, and daily lag-1 autocorrelation. This autocorrelation describes overlapping, recursively weighted estimates and will normally be high; it is not an independent-sample test of beta stability.

Beta stability and forecasting

etf_beta_stability.py studies whether asset betas to a benchmark are stable and which ordinary rolling-OLS lookback best predicts beta over one or more future horizons. For example, to study sector ETF betas to IVV:

python .\etf_beta_stability.py `
  --symbols-file .\sector_spdr_symbols.txt --benchmark IVV `
  --block-days 126 --lookbacks 21 63 126 252 504 756 1260 `
  --forecast-horizons 63 126 252

The stability analysis divides each symbol's longest paired return history into complete, nonoverlapping blocks, anchored at the latest observation. It reports the dispersion and autocorrelations of the block betas. The constancy statistic is an inverse-variance-weighted chi-square diagnostic based on each OLS beta's standard error; its assumptions are approximate, so it should be read together with beta dispersion, autocorrelations, and the number of blocks. The reliability ratio estimates the share of observed block-beta variance left after subtracting average OLS sampling variance, floored at zero.

For forecasting, each lookback's full rolling beta series is computed once per symbol and reused for every requested future horizon. Within a symbol and horizon, all lookbacks use identical forecast origins and completed future periods do not overlap. This prevents shorter lookbacks from receiving a longer or different evaluation sample. Results include bias, MAE, RMSE, correlation with future realized beta, the rate of beating a fixed beta of 1, and realized volatility after hedging by the forecast beta. Symbol-level comparisons are primary; *AVERAGE* rows are equal-weighted summaries across symbols.

The script uses all available saved history when Yahoo returns less than --history-days. It stores adjusted closes and writes nonoverlapping beta rows, stability summaries, autocorrelations, individual forecasts, forecast-accuracy summaries, and best-lookback tables under output/. Every row includes the relevant estimation or evaluation date range. This first version intentionally uses equal-weighted ordinary OLS; exponentially weighted beta estimators are not mixed into these results.

Sector-shrunk constituent beta forecasts

etf_constituent_beta_forecasts.py uses an ETF holdings file to test whether a stock's future beta is predicted better by its own historical beta, the average beta of other stocks in its sector, or a blend of the two:

python .\etf_constituent_beta_forecasts.py .\IVV_holdings.csv `
  --benchmark IVV --lookbacks 21 63 126 252 504 756 1260 `
  --forecast-horizons 63 126 252

Historical beta defaults to equal-weighted ordinary OLS. Add EWMA beta to the same fair-sample experiment with:

python .\etf_constituent_beta_forecasts.py .\IVV_holdings.csv `
  --benchmark IVV --beta-estimators ols ewma `
  --lookbacks 63 126 252 504 --forecast-horizons 63 126 252

For an EWMA beta, lookback_days is an equivalent SMA window and the decay is 1 - 2 / (lookback_days + 1). This gives the EWMA the same limiting effective sample size and average observation age as that SMA window. The detailed CSV records alpha, decay, half-life, effective estimation observations, and an approximate weighted-regression beta standard error. Future realized beta remains ordinary equal-weighted OLS over the requested forecast horizon, so the test measures whether recency weighting improves prediction of the same target.

Sector means are leave-one-out: the stock being predicted never contributes to its own sector prior. The empirical shrinkage forecast is

stock_weight * stock_beta + (1 - stock_weight) * peer_mean_beta

where stock_weight is estimated as true within-sector beta variance divided by true within-sector variance plus the stock beta's OLS sampling variance. True within-sector variance is estimated by subtracting peers' average sampling variance from their observed beta variance and flooring the result at zero. Consequently, noisy stock betas receive more shrinkage. If fewer than --min-sector-peers other stocks have usable estimates, the script uses the leave-one-out mean of the full constituent universe and records universe in prior_source.

The default out-of-sample comparison includes beta 1, raw stock beta, sector beta, empirical shrinkage, and fixed stock weights of 0.25, 0.50, and 0.75. Change the fixed blends with --fixed-stock-weights; specifying that option without values omits them. Every method, estimator, and lookback is summarized using the same intersection of stocks and forecast origins within each horizon. The detailed forecast file retains all usable rows and marks the comparison subset with common_estimator_lookback_sample; the legacy common_lookback_sample column is retained as an alias. Metrics include beta bias, MAE, RMSE, forecast/future correlation, frequency of beating raw beta, and average future hedge residual volatility. Rolling estimates for every distinct lookback or horizon are calculated once and reused. Because beta 1 is constant, it is reported once per horizon without a meaningless best lookback.

Outputs include the individual constituent forecasts plus overall, sector, and stock accuracy tables and an overall best-lookback table. Adjusted closes are stored in the normal holdings price cache, and all outputs include their estimation and future date ranges. The classifications in a current holdings file are current classifications; rigorous long-history research would require point-in-time sector classifications to avoid classification hindsight. Use --summary-only to skip the large constituent-forecast CSV while retaining the overall, sector, stock, and best-lookback accuracy files. An existing detailed file is left unchanged and reported as not updated.

Dynamic and mean-reverting constituent betas

etf_constituent_dynamic_betas.py compares a trailing constant OLS beta with Kalman-filtered random-walk and stationary AR(1) betas:

python .\etf_constituent_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --parameter-lookback 1260 --backtest-days 1260 `
  --forecast-horizons 63 126 252 --sector Energy --summary-only

Walk-forward estimation displays progress for unique stock/origin/dynamic-model fits, including elapsed time and an estimated time remaining. Use --no-progress to suppress it.

The observation and AR(1) state equations are

stock_return[t] = alpha + beta[t] * market_return[t] + error[t]
beta[t] - long_run_beta = persistence * (beta[t-1] - long_run_beta)
                          + beta_innovation[t]
beta[t+1] - long_run_beta = persistence * (beta[t] - long_run_beta)
                            + leverage * error[t] + beta_innovation[t+1]

kalman-rw fixes persistence at one; kalman-ar1 estimates persistence between zero and one. kalman-ar1-leverage adds the third equation as an alternative transition: a negative leverage coefficient means a negative idiosyncratic return raises the next beta. The observation innovation is handled jointly in the state-space likelihood rather than inserted as a pre-estimated OLS residual. For numerical stability, the coefficient is smoothly constrained to the very broad range -100 to 100; output flags estimates at that bound. The AR(1) forecast is the expected average beta over the future horizon, which matches the future-window OLS-beta target more closely than the terminal state would. For the leverage model, it includes the last observed return innovation in the first future beta and includes uncertainty from unknown future idiosyncratic shocks. Parameter estimation uses only the trailing --parameter-lookback returns available at each forecast origin. Future beta periods are nonoverlapping within each horizon, and accuracy tables use only stock/origin observations available for every requested model. Thus neither the filtered state nor its fitted parameters use future returns in the walk-forward results.

The program reports beta RMSE, MAE, bias, forecast/future correlation, hedge residual volatility, the frequency of beating ordinary OLS, and the AR(1) persistence and implied half-life. The parameter-history CSV also records innovation and observation variances, likelihood, AIC, BIC, convergence, and the exact estimation dates. It also records the leverage coefficient and beta responses to -1% and -1 standard-deviation idiosyncratic returns. A paired model-comparison CSV reports AR(1) minus random-walk AIC and BIC and the log-likelihood improvement. Because the random-walk limit lies on a boundary and long-run beta is weakly identified as persistence approaches one, these likelihood comparisons are descriptive; walk-forward forecast and hedge performance are the primary tests. Numerical maximum-likelihood fitting is expensive; --backtest-days defaults to 1260, while --sector and --max-sym support focused tests. Use --models to omit unwanted models.

Optional --state-history filtered, smoothed, or both fits each dynamic model once to the full available sample and writes daily state paths. These are explicitly labeled as full-sample retrospective diagnostics: even the filtered states use parameters estimated from the full sample. They must not be treated as out-of-sample histories. The smoother additionally uses subsequent returns directly.

To test the leverage hypothesis before adding another state-space parameter, use the residual-return diagnostic:

python .\etf_constituent_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --sector Energy --leverage-diagnostic `
  --leverage-windows 1 5 21 63 --bootstrap-samples 1000 --summary-only

At each forecast origin, the program fits trailing OLS using only the parameter window, constructs recent stock-minus-alpha-minus-beta-times-market residual returns, and standardizes each cumulative residual signal by trailing idiosyncratic volatility and the square root of its signal window. Standardized signals are winsorized at +/-4 by default; change this with --leverage-signal-cap. A signal window must be shorter than the parameter window because residuals sum to zero across the complete OLS calibration sample.

For each beta model and future horizon, the diagnostic regresses future beta minus the model's forecast beta on the signed residual-return signal with stock fixed effects. A negative coefficient supports the hypothesis: a -1 standard deviation idiosyncratic-return signal predicts a beta increase equal to the negative of that coefficient. Confidence intervals resample whole forecast origins, keeping contemporaneous stocks together. This is an exploratory association test rather than a separately trained out-of-sample adjustment. If the negative relationship is stable across windows, horizons, sectors, and historical samples, it motivates a subsequent kalman-ar1-leverage transition model. That model is now available through --models kalman-ar1 kalman-ar1-leverage. The nested comparison writes a separate CSV containing leverage-minus-AR(1) AIC and BIC, the likelihood-ratio statistic and one-degree- of-freedom p-value, the fitted leverage coefficient, and its implied beta responses. These are in-sample model-selection diagnostics; common-sample walk-forward beta RMSE and hedge volatility remain the more important test. The leverage model is opt-in so existing default runs retain their prior run time and common-model sample. For example:

python .\etf_constituent_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --sector Energy `
  --models kalman-ar1 kalman-ar1-leverage `
  --parameter-lookback 1260 --backtest-days 1260 --summary-only

Leverage coefficients can instead be constrained across stocks:

python .\etf_constituent_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --sector Energy `
  --models ordinary-ols kalman-ar1 kalman-ar1-leverage `
  --leverage-pooling sector --leverage-scale standardized `
  --parameter-lookback 1260 --backtest-days 1260 --summary-only

--leverage-pooling none retains separate stock coefficients, sector fits one coefficient per selected sector, and global fits one coefficient across the complete selected universe. With only one sector selected, sector and global pooling are equivalent. --leverage-scale raw makes the coefficient a response per decimal idiosyncratic return. standardized divides the return innovation by its fitted observation volatility, so the negative coefficient is directly the beta increase after a -1 standard-deviation idiosyncratic return. Standardized scaling is generally more comparable across stocks.

Sector and global constraints are estimated by scalar profile likelihood. At each proposed common coefficient, every stock's intercept, long-run beta, persistence, observation variance, and beta-innovation variance are refitted. Use --pooling-profile-evaluations to trade precision for run time and --leverage-pooling-bound to override the default search range of +/-2 for a standardized coefficient or +/-100 for a raw coefficient. The ordinary AR(1) model is also restarted from the leverage solution before comparison, reducing spurious information-criterion gains caused by local optimization.

The pooled objective sums the univariate stock log likelihoods. Because stock observation residuals can be contemporaneously correlated, it is labeled a composite likelihood rather than a full joint likelihood. Its AIC and BIC are useful descriptive comparisons with the identically constructed no-leverage objective, but ordinary likelihood-ratio inference is not reported. Results are written to <holdings>__leverage_<pooling>_<scale>_dynamic_beta_pooled_leverage_diagnostics.csv. The normal forecast and accuracy files use the same distinguishing suffix. Pooled full-sample --state-history output is not currently supported.

To test whether beta changes contain a shared sector component, add:

python .\etf_constituent_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --models ordinary-ols --beta-change-correlation-diagnostic `
  --correlation-block-days 63 --bootstrap-samples 1000 --summary-only

The diagnostic estimates OLS betas in nonoverlapping blocks, differences adjacent block betas, and compares average pairwise beta-change correlations within sectors with those between sectors. Its fast date-block influence-function bootstrap preserves the stock cross-section and approximates re-estimation of those correlations under date resampling. It also reports the leading correlation-matrix eigenvalue share for stocks with a complete beta-change panel. Positive within-minus-between correlation is evidence consistent with a common sector state, but it is not conclusive: finite-window beta estimation error and common omitted return shocks can also generate apparent co-movement. All available benchmark history is used by default; --correlation-days N restricts the diagnostic to the most recent N return days. Selecting only ordinary-ols in this diagnostic run avoids unrelated Kalman refits.

Outputs include:

• <holdings>_dynamic_beta_forecasts.csv
• <holdings>_dynamic_beta_parameter_history.csv
• <holdings>_dynamic_beta_accuracy.csv
• <holdings>_sector_dynamic_beta_accuracy.csv
• <holdings>_stock_dynamic_beta_accuracy.csv
• <holdings>_dynamic_beta_model_comparison.csv
• optional <holdings>_dynamic_beta_leverage_model_comparison.csv
• optional <holdings>_dynamic_beta_leverage_diagnostic.csv
• optional nonoverlapping-beta-change, pair-correlation, and correlation-summary CSVs
• optional state-history and full-sample-parameter CSVs

The dynamic-beta model uses Statsmodels; install the optional dependency with:

python -m pip install statsmodels

KFAS one-step dynamic-beta comparison

kfas_dynamic_betas.R is a deliberately simpler R implementation of the single-stock random-walk and stationary AR(1) beta models. It reads an adjusted- close cache already written by the Python programs and never downloads data. For each stock it estimates model hyperparameters on a training window, fixes those parameters, and then lets the Kalman state update sequentially while measuring genuinely out-of-sample one-day-ahead return innovations:

Rscript -e "install.packages('KFAS')"

Rscript .\kfas_dynamic_betas.R `
  .\output\IVV_holdings_adjusted_close.csv --benchmark IVV `
  --parameter-lookback 1260 --backtest-days 252 --max-sym 50 `
  --fixed-ar1 0.97 0.98 0.99

Use --symbols-file to select symbols explicitly. Without it, every price column other than the benchmark is considered. --fixed-ar1 adds any number of stationary models whose persistence is held fixed while the observation and beta-innovation variances are estimated separately for each stock. The ordinary ar1 model with estimated persistence remains in the comparison. The program writes detailed predictions, fitted parameters, overall and per-stock accuracy, and a random-walk comparison CSV. RMSE, MAE, bias, Gaussian predictive density, matched-observation win rate, MSE improvement versus the random walk, and the fraction of stocks with lower RMSE are reported. All requested models must fit successfully for a stock to enter the comparison, ensuring an identical sample. This test evaluates next-day stock-return prediction, not the accuracy of a beta measured over a future multi-day window, so its conclusions answer a different question from the Python beta-forecast backtests.

Shared sector dynamic betas

etf_sector_dynamic_betas.py is the corresponding parsimonious multivariate experiment. Rather than estimating an unrestricted state covariance across all stocks, it fits each sector as

stock_return[i,t] = alpha[i] + (baseline_beta[i] + sector_shift[t])
                    * market_return[t] + error[i,t]

The shared-only sector shift follows either a random walk or a stationary zero-mean AR(1). Stock-specific intercepts and baseline betas are trailing-panel OLS estimates. The AR(1) forecast uses the expected average sector shift over the future beta-measurement horizon. This gives one dynamic state per sector rather than a dense stock-by-stock state covariance:

python .\etf_sector_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --parameter-lookback 1260 --backtest-days 1260 `
  --forecast-horizons 63 126 252 --summary-only

The model retains a full contemporaneous covariance matrix for stock observation residuals, shrunk 10% toward its diagonal by default. This matters because a common sector return shock should not automatically be interpreted as a beta-state innovation. Change it with --observation-covariance-shrinkage; 0 uses the sample residual covariance and 1 uses only residual variances. If the complete panel does not meet --coverage, the sparsest individually eligible histories are removed until it does, subject to --min-sector-symbols. Parameter output records the exact symbols retained at every forecast origin.

The default walk-forward comparison includes sector-static-ols, sector-kalman-rw, and sector-kalman-ar1, with common-model sample filtering and overall, sector, and stock accuracy CSVs.

Two opt-in univariate dynamic benchmarks estimate each stock from only its own and the index's returns, on the same dates used by its sector panel. This separates the benefit of cross-stock sector pooling from the benefit of allowing beta to vary at all:

python .\etf_sector_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV `
  --models sector-static-ols sector-kalman-rw sector-kalman-ar1 `
           stock-kalman-rw stock-kalman-ar1 `
  --parameter-lookback 1260 --backtest-days 1260 `
  --forecast-horizons 63 126 252 --summary-only

The accuracy output reports how often each sector model has lower squared error than its corresponding single-stock dynamic model. The single-stock fits can make this comparison much slower. Progress is displayed every five percentage points; use --no-progress to suppress it.

The opt-in structured factor models add one stock-specific beta state for every retained constituent:

beta[i,t] = baseline_beta[i] + sector_shift[t] + stock_shift[i,t]
Q_beta = q_sector * 1 * 1' + diag(q_stock[1], ..., q_stock[N])

Thus beta innovations are positively but imperfectly correlated. Each stock has its own estimated idiosyncratic beta-innovation variance, while all stock-specific states share one persistence parameter in the AR(1) version. The common-state loading remains fixed at one to identify the decomposition and avoid another parameter per stock. Run a direct comparison with:

python .\etf_sector_dynamic_betas.py .\IVV_holdings.csv `
  --benchmark IVV --sector Energy `
  --models sector-static-ols sector-kalman-ar1 sector-factor-ar1 `
  --parameter-lookback 1260 --backtest-days 1260 `
  --forecast-horizons 63 126 252 --summary-only

sector-factor-rw provides the corresponding random-walk specification. Parameter output reports the sector and stock innovation variances, common and stock persistence, filtered stock shifts, and the mean implied pairwise beta- innovation correlation. The detailed forecast output includes each stock's sector and stock-specific forecast components and forecast standard error. When a factor model and its shared-only counterpart are both requested, the program writes <holdings>_sector_factor_model_comparison.csv with paired AIC and BIC differences. The additional stock innovation variances are zero under the simpler model and therefore lie on a boundary, so no ordinary chi-square likelihood-ratio p-value is reported.

The factor models are opt-in because their state dimension is the number of stocks plus one and their numerical optimization is substantially slower. Start with one --sector and, if needed, --max-sym before attempting the full holdings universe. These models do not yet combine the sector factor structure with the separate univariate leverage-effect transition.

Sector-shrunk constituent volatility forecasts

etf_constituent_volatility_forecasts.py performs the analogous experiment for individual-stock variance forecasts:

python .\etf_constituent_volatility_forecasts.py .\IVV_holdings.csv `
  --variance-estimators sma ewma `
  --lookbacks 21 63 126 252 504 756 1260 `
  --forecast-horizons 63 126 252

The script shrinks daily variance—not volatility—toward the leave-one-out sector mean variance. It compares raw, sector-only, empirical, and fixed-weight forecasts. Use --variance-estimators sma ewma to compare simple moving variances and exponentially weighted variances on the same observations. For an EWMA equivalent window N, alpha = 2 / (N + 1) and decay = (N - 1) / (N + 1). With a complete long history, this asymptotically matches both the SMA's effective sample size and average observation age. The output records alpha, decay, half-life, and effective observation count. The future target remains an ordinary sample variance over the completed forward horizon.

Use --variance-models market-residual to decompose each forecast into a market-related component and an idiosyncratic component:

python .\etf_constituent_volatility_forecasts.py .\IVV_holdings.csv `
  --variance-models direct market-residual --market-symbol IVV `
  --variance-estimators sma ewma --lookbacks 63 126 252 `
  --beta-lookbacks 126 252 --beta-shrinkage none empirical `
  --forecast-horizons 63 126 252

For the market-residual model, total forecast variance is beta^2 times the forecast market variance plus forecast residual variance. Market and residual variance use the requested SMA or EWMA estimator and variance lookback. A stock's historical residual on date t uses an intercept and beta estimated only through t-1, avoiding look-ahead. Residual variance is then compared in raw, sector-only, empirical-shrinkage, and fixed-weight forms. Beta can remain unshrunk (none) or be empirically shrunk toward its leave-one-out sector mean. --beta-uncertainty-adjustment replaces beta^2 with beta^2 + beta_standard_error^2 in the market component.

Detailed market-residual output includes the raw and forecast beta, beta standard error, market variance, market-related variance, and residual-variance components. Best-lookback reports select both a variance lookback and a beta lookback. Requesting multiple models, estimators, variance lookbacks, beta lookbacks, and beta methods forms their full configuration grid and can substantially increase run time and output size.

Forecast evaluation uses the array-oriented engine by default. It performs leave-one-out sector shrinkage, prediction, and loss calculations with NumPy arrays and constructs each detailed result table once. Use --engine legacy to run the original pandas implementation when auditing results. The two engines produce the same output schema and are intended to be numerically equivalent; console output identifies the selected engine. Detailed timing separates rolling estimation, shrinkage/evaluation/row assembly, common-sample summaries, and CSV writing.

The empirical stock weight uses estimated true cross-sectional variance within the sector and the stock variance estimate's sampling variance. SMA uncertainty uses the standard iid fourth-moment formula. EWMA uncertainty uses weighted fourth moments and squared normalized weights. Both reflect observed kurtosis but not serial dependence in squared returns or volatility clustering. The estimator and that limitation are recorded explicitly rather than treating the resulting weight as exact.

QLIKE is the primary forecast loss. The reports also include annualized variance bias and RMSE, volatility bias, MAE and RMSE, forecast/future volatility correlation, and the frequency with which each method beats raw variance by QLIKE. Best lookbacks are reported separately under QLIKE and volatility RMSE.

As in the beta forecast program, detailed rows retain every usable forecast and accuracy summaries use only the common intersection across all requested model, estimator, variance-lookback, beta-lookback, and beta-shrinkage configurations within each horizon. Future periods are completed and nonoverlapping. The ETF inferred from an <ETF>_holdings.csv filename defines the trading-day calendar; override it with --calendar-symbol. Outputs include detailed, overall, sector, stock, and best-lookback CSVs, together with calculation and overall elapsed times. Use --summary-only to skip the large constituent-volatility-forecast CSV while retaining all four accuracy summaries. An existing detailed file is left unchanged and reported as not updated.

Use repeatable, case-insensitive --sector options to restrict a run before prices are downloaded and rolling estimates are calculated:

python .\etf_constituent_volatility_forecasts.py .\IVV_holdings.csv `
  --sector Energy --sector Utilities --variance-estimators sma ewma

Quote sector names containing spaces, such as --sector "Information Technology". --list-sectors prints the available names, constituent counts, and holdings weights without loading prices. Sector filters are applied before --max-sym. Results use a distinct stem such as IVV_holdings__energy__utilities, while the normal IVV_holdings_adjusted_close.csv cache is reused so sector studies do not redownload data or overwrite full-universe result files.

Signed-return volatility asymmetry

etf_constituent_volatility_asymmetry.py performs a walk-forward test of whether recent return direction improves forecasts beyond historical variance:

python .\etf_constituent_volatility_asymmetry.py .\IVV_holdings.csv `
  --lookbacks 21 63 126 252 --forecast-horizons 21 63 126 252 `
  --signals cumulative-return ma-deviation

For every lookback, cumulative return is standardized by sqrt(lookback * historical_variance). Moving-average deviation is the latest adjusted log price minus its rolling mean log price, standardized by the historical volatility and the exact sum of squared return weights implicit in that deviation. Standardized signals are winsorized at +/-4 by default; change this with --signal-cap. Detailed output also retains cumulative return and price/geometric-moving-average deviation in percentage units.

The program compares raw historical variance, a calibrated historical-variance regression, and symmetric and downside-asymmetric versions of each requested signal. A symmetric model adds absolute signal magnitude. Its asymmetric counterpart also adds abs(signal) * I(signal < 0). A positive downside-extra coefficient means an equally large negative signal predicts greater future variance than a positive signal.

Regressions predict log future variance and use a fitted smearing factor when returning to variance units. At every forecast origin, coefficients use only earlier forecast periods whose future end date is already known. Defaults require eight completed periods and 100 pooled stock-period observations; control these with --min-training-periods and --min-training-observations. Forecast horizons are nonoverlapping, and accuracy comparisons use the common intersection of every requested model and lookback.

Universe-pooled coefficients remain the default. To compare them with separate sector fits and empirically partially pooled sector coefficients, use:

python .\etf_constituent_volatility_asymmetry.py .\IVV_holdings.csv `
  --coefficient-structures pooled sector partial `
  --min-sector-training-observations 30 --summary-only

sector estimates every coefficient separately within each sector. partial shrinks each fitted sector coefficient toward the corresponding pooled coefficient. Its data-estimated sector weight is the nonnegative estimated between-sector coefficient variance divided by that variance plus the sector coefficient's estimated sampling variance. A sector with insufficient training data falls back to pooled coefficients in partial mode and is omitted in sector mode. The coefficient CSV records the source, standard error, and sector weight for every coefficient. This is an empirical partial-pooling approximation rather than a full hierarchical model, and the conventional OLS sampling variances do not account for common shocks across stocks.

QLIKE comparisons report both performance against raw historical variance and, for asymmetric models, the paired improvement and win rate against the matching symmetric model. Conventional OLS coefficient standard errors are saved as descriptive diagnostics; they do not adjust for common shocks across stocks. For uncertainty that preserves each forecast date's cross section, request a forecast-origin block bootstrap:

python .\etf_constituent_volatility_asymmetry.py .\IVV_holdings.csv `
  --bootstrap-samples 1000 --confidence-level 0.95 --summary-only

It reports percentile confidence intervals for mean paired QLIKE improvement, the rate at which the asymmetric model beats its symmetric counterpart, and the mean downside-extra coefficient. Entire forecast-origin blocks are resampled; stocks within a date stay together. The random seed defaults to 12345 and can be changed with --bootstrap-seed.

--sector, --list-sectors, and --max-sym support focused and test runs.

Outputs include detailed forecasts, walk-forward coefficient histories, overall/sector/stock accuracy tables, and best-lookback results. This first version forecasts total stock variance; market and residual signed signals are reserved for a subsequent decomposed-model extension. Use --summary-only to skip the potentially very large detailed forecast CSV while retaining the coefficient history and every accuracy summary. If a detailed forecast file already exists at the normal path, summary-only mode leaves it unchanged and prints that fact explicitly.

When --bootstrap-samples is positive, the standalone <holdings>_volatility_asymmetry_bootstrap.csv contains the interval estimates; they are also merged into the overall accuracy and selected best-lookback rows.

Current-holdings-implied ETF risk

etf_holdings_risk.py estimates the volatility, covariance, and correlation of one or more ETFs from their current constituent weights. Supply one ETF symbol for each holdings file, in the same order:

python .\etf_holdings_risk.py `
  .\IVV_holdings.csv .\TOPT_holdings.csv `
  --etfs IVV TOPT --days 63 126 252 `
  --covariance sample ledoit-wolf --sector

For each window and constituent covariance estimator, the program forms one covariance matrix over the union of eligible stocks and calculates W' * stock_covariance * W. This produces a coherent current-holdings-implied ETF covariance matrix. It compares that matrix with the ordinary sample covariance of historical ETF returns over exactly the same common dates. The historical estimator remains sample even when ledoit-wolf is selected for the much larger constituent matrix.

Sample covariance is the default and is valid for this calculation even when there are more stocks than observations because no matrix inverse is required. Ledoit-Wolf provides an alternative shrinkage-based risk forecast. Console covariances are annualized and displayed in percent squared; covariance CSVs contain annualized decimal-squared values. Every difference is defined as implied minus historical.

By default, eligible positive equity holdings are normalized to 100%. Use --weight-treatment as-reported to preserve their reported weights and implicitly treat omitted weight as zero-volatility cash. The summary always reports total source equity weight, included source weight, and the fraction of source weight retained, making missing securities and non-equity exposures visible.

The combined adjusted-close cache includes the union of constituents and the ETF symbols. Separate constituent and ETF price files are also saved. Outputs include:

• <merged-holdings>_holdings_risk_summary.csv
• <holdings>_holding_risk_contributions.csv
• <holdings>_sector_risk_contributions.csv with --sector
• <merged-holdings>_<window>d_<method>_implied_annualized_covariance.csv
• corresponding historical and covariance-difference matrices
• corresponding implied, historical, and correlation-difference matrices

Constituent risk files contain calculation weights, marginal and component annualized volatility, and fractional variance contributions. Fractional contributions sum to one but individual values can be negative when a holding provides diversification. The calculation is a risk estimate for today's weights under a recent covariance regime, not a reconstruction of the ETF's historical holdings. Summary rows and console headings report the exact common return dates used for each window and method.

Minimum-variance portfolios

minimum_variance_portfolio.py reads the saved adjusted-close data and constructs long-only minimum-variance portfolios independently for every holdings file and window. It uses Ledoit-Wolf covariance shrinkage by default:

python .\minimum_variance_portfolio.py .\IVV_holdings.csv .\TOPT_holdings.csv `
  --prices-file .\output\IVV_holdings__TOPT_holdings_adjusted_close.csv `
  --days 126 252 --max-weight 0.05

Omit --prices-file when the price file uses the normal single- or multi-holdings prefix. Use --covariance sample for the ordinary sample covariance estimator. --coverage defaults to 0.90; securities below that fraction of available window returns are excluded, after which covariance is estimated on common dates. SciPy is required for optimization and scikit-learn is required for Ledoit-Wolf:

python -m pip install scipy scikit-learn

Matplotlib is optional and is required only for plot output:

python -m pip install matplotlib

Portfolio outputs are:

• <holdings>_minimum_variance_weights.csv
• <holdings>_minimum_variance_summary.csv

Each optimization heading and summary row includes the exact first and last common return dates used to estimate its covariance matrix.

For a portfolio specified directly as Yahoo symbols, with no holdings file, use minimum_variance_symbols.py:

python .\minimum_variance_symbols.py IVV XLE XLP --days 63 126 252 `
  --max-weight 0.60

The script downloads enough adjusted-close history for the longest window, saves it as output/IVV__XLE__XLP_adjusted_close.csv, and constructs a separate long-only minimum-variance portfolio for every requested window. Subsequent runs reuse that cache; --no-download-missing enforces offline use. Use --prices-file to select another cache, --covariance sample for sample covariance, and --coverage to control the minimum usable history per symbol. Results are written to:

• <symbols>_minimum_variance_weights.csv
• <symbols>_minimum_variance_summary.csv

The console heading and summary CSV record the exact common return date range used for each window.

Symbols can instead be read from a CSV containing a symbol or ticker column, or from a text file containing one symbol per line:

python .\minimum_variance_symbols.py --symbols-file .\symbols.csv --days 126 252

--symbols-file may be repeated and combined with positional symbols. Symbols from all sources are normalized and deduplicated in their original order. For a quick test on a large universe, --max-symbols N (or --max-sym N) keeps only the first N unique symbols after that processing:

python .\minimum_variance_symbols.py --symbols-file .\symbols.csv --max-symbols 10

With one symbols file and no positional symbols, its filename stem becomes the default output prefix. For example, sector_spdr_symbols.txt produces sector_spdr_symbols_adjusted_close.csv and corresponding short result names. Use --name to select an explicit prefix, especially for mixed or multiple symbol sources:

python .\minimum_variance_symbols.py --symbols-file .\symbols.csv `
  --name sector_etfs --days 126 252

When --prices-file is provided without positional symbols or --symbols-file, the program infers the universe from the price column headers. This makes a saved cache directly reusable, with --max-symbols applied to the inferred column order. The price filename without _adjusted_close becomes the result prefix unless --name is supplied:

python .\minimum_variance_symbols.py --days 126 252 --max-sym 4 `
  --prices-file .\output\SPY__XLB__XLC__XLE_adjusted_close.csv

The symbol table also reports each asset's annualized covariance-model volatility, correlation to the optimized portfolio, cumulative window return, and Sharpe ratio. The portfolio summary reports its cumulative return and Sharpe ratio as well. Sharpe ratios use arithmetic mean daily excess returns and the selected covariance model's annualized volatility. The annual effective risk-free rate defaults to 4%; set it as a decimal with, for example, --risk-free-rate 0.03. Portfolio returns assume daily rebalancing. All reported returns, correlations, and Sharpe ratios are in-sample statistics rather than an out-of-sample backtest.

Use --corrmat to print an estimator-consistent correlation matrix for every window. Each matrix contains all eligible assets plus a final *MIN_VOL* row and column, and is also saved as <name>_<days>d_correlation_matrix.csv. The console asset table always ends with the same marked portfolio row for comparison; this synthetic row is not added to the weights CSV:

python .\minimum_variance_symbols.py --symbols-file .\sector_spdr_symbols.txt `
  --days 126 252 --corrmat

The direct-symbol optimizer is long-only by default. Set a negative per-symbol lower bound with --min-weight (also accepted as --min-wgt or --min_wgt) to permit shorts. Because short positions can create substantial leverage, an optional gross-exposure cap is available:

python .\minimum_variance_symbols.py --symbols-file .\sector_spdr_symbols.txt `
  --days 126 --min-weight -1.0 --max-weight 1.0 --max-gross 2.0

Weights still sum to 100%; --max-gross 2.0 limits absolute weights to 200% in total. Console and summary CSV output report long, short, and gross exposure plus the largest long and short positions. If negative weights are enabled without --max-gross, the program prints a leverage warning.

To minimize absolute portfolio volatility while limiting annualized tracking error relative to an ETF, provide a benchmark and a decimal tracking-error cap:

python .\minimum_variance_symbols.py --symbols-file .\sector_spdr_symbols.txt `
  --days 126 252 --benchmark SPY --max-tracking-error 0.03

The benchmark is added to the saved price cache automatically when necessary; --no-download-missing retains its strict offline meaning. The benchmark need not be an investable portfolio symbol. Output reports benchmark volatility and return, portfolio tracking error, active period return, information ratio, and whether the constraint binds. Tracking error uses the selected covariance estimator and --annualization-days. If the cap cannot be achieved under the weight and gross-exposure constraints, the error reports the minimum achievable tracking error. --benchmark without a cap reports the same comparison metrics without constraining the minimum-volatility portfolio.

An idiosyncratic-volatility constraint lets benchmark beta vary instead of fixing it at one as tracking error does. It limits the annualized residual volatility after an optimal single-benchmark beta hedge:

python .\minimum_variance_symbols.py --symbols-file .\sector_spdr_symbols.txt `
  --days 126 --benchmark SPY --min-weight -1 --max-gross 2 `
  --max-idio-vol 0.03

The investable weights are optimized with their requested leverage first; the benchmark hedge is diagnostic and is not included in the weight-sum or --max-gross constraint. Output reports portfolio beta, idiosyncratic volatility, the implied benchmark hedge, and gross exposure after adding that hedge. If the benchmark is also investable, the hedge is netted against its existing portfolio weight when calculating hedged gross exposure. Hedged net exposure is reported as well. The summary also reports whether the idiosyncratic-volatility cap binds. If it is infeasible—including when combined with a tracking-error cap—the error reports the minimum achievable residual volatility under the other constraints.

Walk-forward minimum-variance backtests

minimum_variance_backtest.py applies the same covariance estimators, weight bounds, gross limit, and benchmark-risk constraints without look-ahead. This example estimates weights from the preceding 126 returns and rebalances every 63 trading days over a requested five-year evaluation period. --backtest-days is an upper bound; if less history exists, the program uses every available return after reserving the initial lookback window:

python .\minimum_variance_backtest.py `
  --symbols-file .\sector_spdr_symbols.txt `
  --lookback 126 --rebalance-days 63 --backtest-days 1260 `
  --benchmark SPY --transaction-cost-bps 5

Use --min-symbols N (or --min-sym N) to move the starting date forward until at least N investable symbols meet --coverage over a complete lookback window:

python .\minimum_variance_backtest.py `
  --symbols-file .\sector_spdr_symbols.txt `
  --lookback 63 --rebalance-days 63 --backtest-days 10000 `
  --min-sym 10 --benchmark SPY

This is a starting-universe condition only. Later temporary gaps continue to use the ordinary coverage and missing-return policies.

With a benchmark, --vol-ratio-analysis tests whether the strategy subsequently does better when its forecast volatility is especially low relative to the benchmark:

python .\minimum_variance_backtest.py `
  --symbols-file .\sector_spdr_symbols.txt `
  --lookback 63 --rebalance-days 63 --backtest-days 10000 `
  --min-sym 10 --benchmark SPY --vol-ratio-analysis

Each completed, non-overlapping holding period is assigned using the volatility ratio forecast at its rebalance date, avoiding look-ahead bias. The fixed buckets are <0.60, 0.60-<0.80, and >=0.80. Console and CSV output include average portfolio, benchmark, and active period returns; win rate; realized volatilities; Sharpe ratio; maximum drawdown; daily up/down capture; and an unadjusted t-statistic for mean active period return. The program additionally writes <name>_backtest_volatility_ratio_periods.csv and <name>_backtest_volatility_ratio_summary.csv. The last, unfinished holding period appears in the period file but is excluded from grouped statistics.

Use repeatable --plot options to save backtest figures:

python .\minimum_variance_backtest.py `
  --symbols-file .\sector_spdr_symbols.txt --benchmark SPY `
  --plot performance --plot volatility --plot vol-ratio --plot weights

The available plot names are performance, volatility, vol-ratio, and weights; --plot all selects all four. Figures are saved without opening a window by default. Add --show-plot to display all requested figures after they have been saved. --show-plot without --plot is an error. Use --plot-format png|pdf|svg to select the file type and --plot-top-weights N to control how many individual positions appear in the weights chart; remaining positions are combined as Other.

Weights estimated through a given close first earn the following trading day's return. Between rebalances, positions drift with asset returns instead of being implicitly rebalanced every day. One-way turnover is the sum of absolute trades; the first portfolio purchase is included. Transaction costs are deducted on each rebalance date.

The backtest writes:

• <name>_backtest_daily.csv: daily net/gross returns, wealth, benchmark, costs, and rebalance indicators
• <name>_backtest_weights.csv: pre-trade, target, and trade weights by rebalance
• <name>_backtest_rebalances.csv: covariance and constraint diagnostics
• <name>_backtest_summary.csv: portfolio and benchmark return, volatility, Sharpe ratio, maximum drawdown, compounded relative return, cumulative-return difference in decimal units, realized correlation, turnover, and cost statistics

--benchmark, --max-tracking-error, --max-idio-vol, --min-weight, --max-weight, and --max-gross have the same meanings as in the single-period program and are re-evaluated using only each rebalance's training window. Price history is expanded automatically when the existing cache is too short; use --no-download-missing for strict offline operation.

Yahoo data occasionally contains isolated missing observations. The default --missing-return zero uses a stale-price assumption and reports the number of affected held-asset and benchmark returns. Use --missing-return error for a strict run.

Results based on a current constituent or symbol list have survivorship bias and are not a historical constituent backtest. Adjusted-close returns also omit taxes, bid/ask effects beyond the specified transaction cost, short borrow fees, and financing costs. These outputs are research estimates, not executable performance records.

Walk-forward mean-variance backtests

mean_variance_backtest.py uses the same walk-forward accounting, constraints, benchmark analysis, plots, price cache, and output structure, but estimates expected returns at every rebalance and solves either

minimize (risk_aversion / 2) * variance - expected_return

or minimum variance subject to an expected-return target. For example:

python .\mean_variance_backtest.py `
  --symbols-file .\sector_spdr_symbols.txt `
  --lookback 126 --return-lookback 252 `
  --return-estimator shrinkage --mean-shrinkage 0.50 `
  --risk-aversion 3 --max-weight 0.25 --benchmark SPY

--return-estimator accepts historical, ewma, or shrinkage. Shrinkage, the default, pulls each arithmetic historical mean toward the cross-sectional mean; --mean-shrinkage 0 is the raw historical estimator and 1 gives every asset the cross-sectional mean. For EWMA, configure --ewma-halflife. --return-lookback defaults to the covariance --lookback.

If neither formulation is supplied, --risk-aversion 3 is used. Alternatively, replace it with an annualized decimal target such as --target-return 0.10. The two options are mutually exclusive, and an infeasible target reports the maximum expected return allowed by the weight and gross-exposure constraints.

Mean-variance result names include _mean_variance by default, while saved price files retain the ordinary symbol-list name and are shared with the minimum- variance backtest. The weights CSV includes each eligible asset's estimated annualized return. Expected-return estimates are especially noisy, so compare results across estimators and lookbacks and include realistic transaction costs. The shared implementation is in portfolio_backtest.py; the two strategy files are small command-line entry points.

Sector-sleeve optimization reduces the covariance dimension while keeping stock weights proportional to the source ETF weights within each sector:

python .\minimum_variance_portfolio.py .\IVV_holdings.csv `
  --prices-file .\output\IVV_holdings__TOPT_holdings_adjusted_close.csv `
  --days 126 252 --portfolio-level sectors --max-sector-weight 0.25

Use --portfolio-level both to run stock- and sector-level optimizations. Stocks that fail --coverage or lack a positive source holdings weight are excluded; their weight is redistributed proportionally among eligible stocks in the same sector. Console and CSV output compare optimized weights with normalized ETF holdings weights; active_weight is minimum-variance weight minus ETF weight. Both ETF and active weights are also included for derived constituents. Sector mode writes:

• <holdings>_sector_minimum_variance_weights.csv
• <holdings>_sector_minimum_variance_constituent_weights.csv
• <holdings>_sector_minimum_variance_summary.csv

Fixed legacy position and new-money completion

Use --legacy-symbol to keep an existing ETF position fixed and optimize only the new-money sleeve. For example, this keeps 90% in IVV and allocates the remaining 10% across sector sleeves:

python .\minimum_variance_portfolio.py .\IVV_holdings.csv --days 126 `
  --portfolio-level sectors --max-sector-weight 0.30 `
  --legacy-symbol IVV --legacy-weight 0.90

The primary saved-price file must include the legacy symbol. If its prices are stored in a separate CSV, provide --legacy-prices-file as well. --legacy-symbol alone uses a 0.90 legacy weight; specifying --legacy-weight without --legacy-symbol is an error.

If a required symbol column is absent, the portfolio script downloads only the missing symbol through the shared Yahoo helper, aligns it to the saved file's last date, and updates that CSV without removing existing columns. Existing unavailable constituents remain subject to coverage exclusion. Use --no-download-missing for strict offline operation.

Completion output distinguishes sleeve weights (summing to 100% of new money), portfolio weights (summing to the new-money fraction), and total look-through weights after adding the legacy ETF exposure. Summaries report legacy volatility, standalone sleeve volatility, completed-portfolio volatility, and estimated volatility reduction versus investing everything in the legacy ETF.

For several fixed positions, use a CSV with signed final-portfolio weights:

ticker,weight
IVV,0.90
XLE,-0.10

python .\minimum_variance_portfolio.py .\IVV_holdings.csv --days 126 `
  --portfolio-level sectors --legacy-positions .\legacy_positions.csv

Negative fixed weights represent shorts or hedges. Their net sum determines the new-money fraction, while gross, long, and short exposures are reported separately. The net fixed weight must be below 1; net-zero signed portfolios are supported. The optimized new-money sleeve remains long-only. For multiple legacy assets, direct weights are reported but sector look-through is omitted unless holdings mappings become available for every legacy asset.

The shared etf_data.py module owns holdings and symbol-list parsing, Yahoo-symbol normalization, multi-fund merging, price downloading/loading, history-aware cache completion, validation, and return calculation. portfolio_optimization.py owns coherent candidate/benchmark covariance estimation and the bounded optimizer used by the single-period, holdings, and backtest programs. The command-line scripts keep their simulation and reporting logic separate.