Skip to content

README_extended.md

Latest commit

 

History

History
903 lines (642 loc) · 33.7 KB

README_extended.md

File metadata and controls

903 lines (642 loc) · 33.7 KB
Raw

Full Real-Data Workflow

This document describes the full Sharadar / FRED / Ken French workflow behind the project write-up at https://danielpellatt.github.io/project.html. It is intended for technical reviewers who have their own licensed data and want to understand or reproduce the full analysis path.

For a quick run that does not require proprietary data, start with the root README.md and run:

python scripts/run_synthetic_demo.py

The synthetic demo starts after Layer B. This extended README assumes you want the full real-data workflow:

Layer A -> Layer B -> Layer C -> D0 / cost tuning / Layer D -> Layer E

1. Data and repository safety

Sharadar / Nasdaq Data Link files are proprietary and must remain local. This repository does not include raw Sharadar files, derived real-data panels, local .env files, virtual environments, caches, or real-data outputs.

Local real-data inputs go under:

data/external/nasdaq_bulk/
data/external/fred/
data/external/ken_french/

Local generated outputs go under:

data/raw/
data/interim/
data/processed/

The root data/ directory is gitignored except for placeholder files and the short data/README.md.

Real-data run provenance

The root README and synthetic demo provide the public no-proprietary-data execution path. This extended README describes the full Sharadar / FRED / Ken French workflow for technical reviewers with their own licensed data.

The empirical analysis summarized in the project write-up was run in a separate local licensed-data environment using:

  • Sharadar / Nasdaq Data Link bulk files,
  • a local FRED DGS3MO.csv file, and
  • local Ken French factor CSVs.

Those proprietary inputs and derived real-data outputs are not included in this repository.

The snapshot below records the Python/package environment observed for that real-data run. It is provided for provenance and debugging context only. It is not required for the public synthetic demo, and it should not be treated as a lock file. For running this repository, use the dependency metadata and setup instructions in this repo.

Item Value
Environment type Local .venv
Operating system / platform macOS-26.3.1-arm64-arm-64bit-Mach-O
Machine arm64
Python Python 3.14.4 (main, Apr 7 2026, 13:13:20) [Clang 21.0.0 (clang-2100.0.123.102)]

Core package versions observed in that environment:

Package Version
numpy 2.4.0
pandas 2.3.3
pyarrow 23.0.0
scipy 1.17.1
scikit-learn 1.8.0
xgboost 3.2.0
matplotlib 3.10.8
PyYAML 6.0.3

This snapshot records the software environment only. It does not pin the exact vintages of the Sharadar bulk files, the FRED CSV, or the Ken French factor CSVs. Future real-data reruns should record vendor download dates or file vintages in the relevant run manifest or results folder.

2. Python setup

From the repository root:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

On Windows PowerShell:

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

Create a local .env file and add your Nasdaq Data Link API key. The .env file is gitignored and should not be committed.

cp .env.example .env

Then edit .env:

NASDAQ_DATA_LINK_API_KEY="paste_your_key_here"

3. Required real-data inputs

Sharadar bulk ZIPs

Download the following Sharadar bulk tables from the Nasdaq Data Link UI and place the ZIPs under data/external/nasdaq_bulk/:

SHARADAR/SEP
SHARADAR/DAILY
SHARADAR/ACTIONS
SHARADAR/SF1
SHARADAR/TICKERS

Layer A converts these vendor ZIPs into local Parquet files under data/raw/sharadar/.

FRED input

Layer B needs the 3-month Treasury yield series by default:

data/external/fred/DGS3MO.csv

Use the standard FRED CSV format:

observation_date,DGS3MO

The pipeline converts the annualized 3-month yield into interval risk-free returns using actual day gaps:

rf = (1 + y / 100)^(delta_days / risk_free_day_count) - 1

The default risk_free_day_count is 360.

Other FRED files may also be present. They are outer-joined into the macro output files, but DGS3MO.csv is the required series for the current main workflow.

Ken French factor inputs

Layer E expects monthly Ken French CSV files under:

data/external/ken_french/

Required files:

F-F_Research_Data_Factors.csv
F-F_Research_Data_5_Factors_2x3.csv
F-F_Momentum_Factor.csv

These are used for CAPM, FF3, FF3+UMD, FF5, and FF5+UMD time-series regressions.

4. Configuration map

The repository uses standalone YAML configs rather than a complicated overlay system.

Config Main use
configs/demo_synthetic.yaml Public synthetic C/D/E demo only.
configs/bulk_full_template.yaml Main real-data A/B/C workflow and a general full-pipeline working config.
configs/d0_tuning_template.yaml D00/D01 signal-selection candidate catalog.
configs/cost_tuning_template.yaml Post-D0 cost-aware implementation sweep.
configs/run_frictionless_replication_kim.yaml Paper-comparable Kim-only reporting run using the broad reproduction universe.
configs/run_frictionless_replication_kim_10pct.yaml Kim-only reporting run with NYSE-referenced 10th-percentile size breakpoint.
configs/run_frictionless_final_comparison.yaml Frictionless 15-year family comparison for selected model representatives; supplemental to the main project write-up.
configs/run_cost_aware_final_comparison.yaml Cost-aware 15-year holdout comparison for selected model representatives.

A practical note: bulk_full_template.yaml includes Layer D and Layer E options even when you are mainly using it for Layers A-C. That is okay. ProjectConfig is a single shared config object, and keeping the file complete makes it runnable as a general pipeline template. For the final reporting runs summarized in the project write-up, use the standalone run_* configs because those intentionally tailor the model family, universe screen, evaluation windows, and cost settings.

5. Layer A — vendor bulk ingestion

Layer A converts manually downloaded Sharadar bulk ZIPs into local partitioned Parquet files.

Run:

python scripts/run_layer_a_download.py --config configs/bulk_full_template.yaml

Expected local outputs:

data/raw/sharadar/TICKERS.parquet
data/raw/sharadar/SEP/year=YYYY/*.parquet
data/raw/sharadar/DAILY/year=YYYY/*.parquet
data/raw/sharadar/ACTIONS/year=YYYY/*.parquet
data/raw/sharadar/SF1/dimension=XXX/year=YYYY/*.parquet

Notes:

  • The current bulk template ingests TICKERS, SEP, DAILY, SF1, and ACTIONS.
  • The default SF1 dimensions are ART as primary and ARY as fallback.
  • Universe restrictions are not imposed in Layer A; they are applied later using the configured universe filters.

6. Layer B — point-in-time panel inputs

Layer B builds the panel inputs used by feature construction. It combines price data, daily data, fundamentals, risk-free returns, market returns, delisting assumptions, and eligibility flags.

Run:

python scripts/run_layer_b_build_panel.py --config configs/bulk_full_template.yaml

Expected local outputs under data/interim/:

daily_prices.parquet/
base_panel_M.parquet/
panel_inputs_M.parquet/
universe_permatickers.parquet
tickers_ref_u_full.parquet
tickers_ref_u_slim.parquet
market_return_1d.parquet
market_return_M.parquet
macro_daily.parquet
macro_monthly.parquet
delisting_adjustments.parquet
risk_free_M.parquet

Layer B does several important timing and accounting jobs:

  • defines the rebalance schedule through the shared calendar utilities;
  • builds a permaticker-keyed monthly base panel;
  • computes ret_current over the current panel period and ret_fwd over the next holding period;
  • converts DGS3MO into daily and panel-frequency risk-free returns;
  • computes daily and monthly excess-return fields;
  • builds daily and monthly market-return files;
  • attaches Sharadar DAILY valuation fields at panel dates;
  • attaches SF1 fundamentals using point-in-time availability dates;
  • computes price-history and fundamentals-history eligibility flags;
  • creates delisting-adjusted return variants for long/short P&L accounting.

Point-in-time SF1 policy

Sharadar SF1 uses datekey as the filing / information-availability date. The pipeline defines:

SF1 rows are assigned an internal availability date equal to the first observed weekday trading date on or after datekey + max(1, pit_safety_shift_days). With the default one-day shift, a filing dated on a Friday is not eligible for a Friday-close or weekend-labeled month-end rebalance; it first becomes eligible on the next observed trading session.

The default safety shift is one (trading) day. Layer B then merges fundamentals as of the panel date using this availability timestamp. The output panel keeps datekey and sf1_age_days for diagnostics.

This differs from the Fama-French June/July accounting convention used in many CRSP/Compustat papers. The goal here is a point-in-time Sharadar implementation rather than a literal CRSP/Compustat reproduction.

Delisting and exit-payoff assumptions

Sharadar does not provide CRSP-style delisting returns. Layer B therefore applies an explicit approximation at true exits, defined as the end of a permaticker price series rather than a ticker-label change.

The pipeline checks SHARADAR/ACTIONS near the final trade date:

  • distressed exits, such as bankruptcy or regulatory delisting, receive dlret_assumed;
  • non-distressed corporate actions, such as mergers or acquisitions, receive no penalty;
  • ambiguous exits receive dlret_assumed_ambiguous.

The default distressed assumption is:

P_exit = P_last_trade * (1 + dlret_assumed)
dlret_assumed = -0.55

This is also the default for ambiguous exits.

Layer B also creates *_short_pnl return fields with separate short-book assumptions. Layer E uses ret_fwd for long positions and ret_fwd_short_pnl for short positions. The default setting undoes the downward adjusting for distressed and ambiguous exits when evaluating short book performance (the delisting adjustment is 0 when the portfolio is short a stock that exits - i.e., the last traded price is recognized if the portfolio is short a stock that was delisted for distressed/ambiguous reasons). This enforces a conservative evaluation of portfolio performance.

Price conventions carried forward

Layer B keeps multiple price conventions because later layers need them for different purposes:

px             # fully adjusted close; used for portfolio returns
split_close    # split-adjusted close; not dividend-adjusted
actual_close   # unadjusted close
px_short_pnl   # alternate series for short-side P&L treatment

7. Layer C — feature construction

Layer C computes stock characteristics from Layer B inputs and supporting interim datasets. It does not download data and does not define a new schedule.

Run the full implemented feature set:

python scripts/run_layer_c_features.py --config configs/bulk_full_template.yaml

Optional explicit subset override:

python scripts/run_layer_c_features.py --config configs/bulk_full_template.yaml --features LME,BEME,r12_2

Expected outputs under data/processed/:

panel_combined_M.parquet/
panel_combined_M_manifest.json
panel_combined_M_feature_diagnostics.parquet
panel_combined_M_batch_diagnostics.parquet
panel_combined_M_yearly_feature_coverage.csv

The feature registry is based on the Freyberger et al. / Kim et al. characteristic set, with Sharadar-compatible proxies where the available fields differ from CRSP / Compustat. Layer C can compute a broader implemented registry; later Layer D configs exclude a small number of sparse or auxiliary fields.

In the analysis described on the website, operating accruals (OA) and absolute operating accruals (AOA) are excluded because their Sharadar proxy coverage is materially weaker than the rest of the feature set. The default Layer D exclusions also remove two sales-zero indicator helper fields and two extra profitability variants that were not part of the target paper feature set.

8. Layer D — rolling portfolio weights

Layer D estimates portfolio weights on a rolling window. The main implemented estimator is kim_etal, the Kim et al. arbitrage-portfolio estimator. The project also includes two XGBoost-based extensions:

kim_ml_alpha_mean
kim_ml_alpha_panel

Run the configured Layer D models:

python scripts/run_layer_d_model_est.py --config configs/bulk_full_template.yaml

Expected outputs under data/processed/:

weights_M.parquet
rebalance_diagnostics_M.parquet
layer_d_config_snapshot_M.json
layer_d_manifest_M.json

Optional CLI overrides:

# Run only a subset of estimators
python scripts/run_layer_d_model_est.py --config configs/bulk_full_template.yaml --estimators kim_etal

# Override the feature list used in Layer D
python scripts/run_layer_d_model_est.py --config configs/bulk_full_template.yaml --features LME,BEME,r12_2

# Send Layer D outputs to a different directory
python scripts/run_layer_d_model_est.py --config configs/bulk_full_template.yaml --output-dir data/processed/layer_d_candidate_example

# Continue even if an estimator errors; errors are recorded in diagnostics
python scripts/run_layer_d_model_est.py --config configs/bulk_full_template.yaml --no-fail

Rolling-window construction

For each rebalance date t, Layer D constructs:

R  = trailing excess-return matrix from ret_current_xs
X0 = characteristics at the start of the estimation window
Xt = characteristics at the rebalance date

With monthly data, the baseline uses layer_d_lookback_periods: 12, so the model estimates from the previous 12 months and forms the portfolio for the next holding period. The six-eigenvector setting is controlled by:

layer_d_n_eigenvectors: 6

Eligibility and universe filters

At each rebalance date, Layer D requires assets to have:

  • presence on the rebalance date;
  • complete trailing return history over the lookback window;
  • complete selected features at both X0 and Xt;
  • the relevant Layer B price and fundamentals history flags;
  • positive end-of-month volume;
  • non-missing forward return.

If layer_d_marketcap_percentile is set, Layer D also applies a live-date market-cap screen using marketcap_from_daily. If layer_d_marketcap_reference_exchanges is empty, the percentile breakpoint is computed from the full live universe. If it contains NYSE, the breakpoint is computed using NYSE names and then applied to all names.

The website-style workflow uses both variants:

  • broad reproduction universe: whole-universe 10th-percentile screen;
  • stricter implementation-aware universe: NYSE-referenced 10th-percentile screen.

The NYSE-referenced screen is useful for implementation-aware comparisons, but Sharadar exchange metadata should be treated as an approximate static vendor classification rather than a perfect historical CRSP venue reconstruction.

ML extensions

The ML extensions replace the linear alpha-function step with XGBoost while keeping the Kim-style risk-hedging structure. After prediction, the signal is projected against the estimated beta-loading space so the portfolio formation remains consistent with the hedge-risk-first logic.

The two extension families are:

  • kim_ml_alpha_mean: XGBoost on the residualized mean-return target;
  • kim_ml_alpha_panel: XGBoost on a month-level panel target.

The final project write-up treats these as empirical extensions, not as new asymptotic theory.

See supplementary_model_note.pdf for additional modeling details and motivation for the ML extensions.

9. Optional Layer D0 — D00 / D01 signal tuning

Layer D0 is a research orchestration layer. It does not implement a new estimator. Instead, it repeatedly runs the existing Layer D → Layer E pipeline for a catalog of candidate settings and writes each candidate to its own archive.

The recommended workflow is:

  1. Run d00 for the broad frictionless screen.
  2. Review data/processed/d0/d00/candidate_leaderboard.csv.
  3. Edit data/processed/d0/d00/shortlist_template.yaml by filling in the candidate_ids field.
  4. Run d01 with that shortlist file.

Run D00:

python scripts/run_layer_d0_tuning.py --config configs/bulk_full_template.yaml --tuning-spec configs/d0_tuning_template.yaml --stage d00

Then run D01 after editing the shortlist file:

python scripts/run_layer_d0_tuning.py \
  --config configs/bulk_full_template.yaml \
  --tuning-spec configs/d0_tuning_template.yaml \
  --stage d01 \
  --shortlist data/processed/d0/d00/shortlist_template.yaml

Expected D0 outputs under data/processed/d0/<stage>/:

candidate_catalog.csv / .parquet
candidate_leaderboard.csv / .parquet
stage_manifest.json
shortlist_template.yaml                  # D00 only
<candidate_id>/layer_d/weights_M.parquet
<candidate_id>/layer_d/rebalance_diagnostics_M.parquet
<candidate_id>/layer_d/layer_d_manifest_M.json
<candidate_id>/layer_e/summary_by_window.csv
<candidate_id>/layer_e/factor_regression_summary.csv
<candidate_id>/layer_e/factor_regression_coefficients.csv

Important notes:

  • D00 / D01 are intentionally frictionless: layer_e_half_spread_multiplier = 0, layer_e_no_trade_band = 0, and layer_e_gross_exposure_cap = null.
  • D0 is for signal selection only.
  • Practical implementation levers such as volatility target, no-trade band, and gross-exposure cap are tuned later.

10. Post-D0 cost-aware tuning sweep

After D01 finishes, use the separate cost-tuning orchestration layer to hold one selected representative from each model family fixed and study practical implementation levers.

Files:

src/alpha_portfolio_project/pipelines/layer_d_cost_tuning.py
scripts/run_layer_d_cost_tuning.py
configs/cost_tuning_template.yaml

Recommended run:

python scripts/run_layer_d_cost_tuning.py \
  --config configs/bulk_full_template.yaml \
  --tuning-spec configs/cost_tuning_template.yaml

Default workflow encoded in the template:

  • choose one fixed representative for each family:
    • kim_etal
    • kim_ml_alpha_mean
    • kim_ml_alpha_panel
  • run Layer D once for each shared annual_vol_target;
  • reuse each Layer D weights archive across the Layer E grid:
    • layer_e_no_trade_band
    • layer_e_gross_exposure_cap
  • evaluate the three family models plus an ensemble portfolio;
  • rank primarily on the combined pre-final validation window prefinal_full, while retaining the separate d00 and d01 windows as diagnostics.

Expected outputs under data/processed/cost_tuning/:

run_catalog.csv / .parquet
model_results_master.csv / .parquet
leaderboard_net_prefinal_full.csv / .parquet
leaderboard_gross_prefinal_full.csv / .parquet
leaderboard_net_prefinal_full__kim_etal.csv
leaderboard_net_prefinal_full__kim_ml_alpha_mean.csv
leaderboard_net_prefinal_full__kim_ml_alpha_panel.csv
leaderboard_net_prefinal_full__ensemble_average.csv
stage_manifest.json
vol_<target>/layer_d/
vol_<target>/band_<band>__cap_<cap>/layer_e/

The cost-tuning spec is where family representatives can be updated after a D0 manual review. The included template contains representative choices consistent with the final reporting configs, while remaining reusable for a fresh research pass.

11. Frictionless reporting runs

After Layer C has produced the full feature panel, the standalone reporting configs can be used to generate paper-comparable and family-comparison output bundles. These configs are standalone rather than overlays on bulk_full_template.yaml.

11.1 Kim replication / paper-comparison run

This run evaluates only the original kim_etal model with beta_projection_mode = "kim" and reports:

full_sample
paper_overlap
post_paper

Run Layer D:

python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_replication_kim.yaml \
  --output-dir data/processed/frictionless_replication_kim/layer_d

Then run Layer E:

python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_replication_kim.yaml \
  --weights data/processed/frictionless_replication_kim/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_replication_kim/layer_e \
  --models kim_etal

This config keeps the spread-cost path visible, but it does not apply no-trade bands or gross-exposure caps:

layer_e_half_spread_multiplier = 0.5
layer_e_no_trade_band = 0.0
layer_e_gross_exposure_cap = null
layer_e_suppress_ensemble = true

11.2 Kim replication with NYSE-referenced 10th-percentile filter

This run evaluates the same Kim baseline but uses the stricter NYSE-referenced 10th-percentile size breakpoint.

Run Layer D:

python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_replication_kim_10pct.yaml \
  --output-dir data/processed/frictionless_replication_kim_10pct/layer_d

Then run Layer E:

python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_replication_kim_10pct.yaml \
  --weights data/processed/frictionless_replication_kim_10pct/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_replication_kim_10pct/layer_e \
  --models kim_etal

11.3 Final 15-year frictionless family comparison

This run compares the selected representative of each model family over the final holdout window:

2011-03-31 through 2026-02-28

The current selected representatives encoded in the config are:

kim_etal with beta_projection_mode = kim
kim_ml_alpha_mean with profile p1 and beta_projection_mode = updated
kim_ml_alpha_panel with profile p13, beta_projection_mode = kim, iterate_to_final = false

Run Layer D:

python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_final_comparison.yaml \
  --output-dir data/processed/frictionless_final_family_comparison/layer_d

Then run Layer E:

python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_final_comparison.yaml \
  --weights data/processed/frictionless_final_family_comparison/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_final_family_comparison/layer_e

If you omit --models, Layer E evaluates all persisted base models plus the ensemble unless the config suppresses the ensemble. If you pass --models explicitly, include ensemble_average if you want the ensemble in the run.

These outputs are supplemental to the project write-up. The website focuses on the cost-aware final comparison, while this frictionless bundle is useful for inspecting pre-cost model behavior and performance without no-trade-band implementation choices.

12. Final 15-year cost-aware family comparison

After D0 and cost tuning, use the standalone final cost-aware config:

configs/run_cost_aware_final_comparison.yaml

This config is meant for the true final holdout window only:

final_holdout: 2011-03-31 through 2026-02-28

It evaluates one selected representative from each family:

kim_etal
kim_ml_alpha_mean
kim_ml_alpha_panel

The currently encoded cost-aware settings are:

Kim et al. baseline:       annual_vol_target = 0.125, gross cap = 2.5, no-trade band = 0.0025
ML extension 1 mean:       annual_vol_target = 0.100, gross cap = 2.5, no-trade band = 0.0025
ML extension 2 panel:      annual_vol_target = 0.125, gross cap = 4.5, no-trade band = 0.0050

If you rerun D0 and cost tuning and choose different finalists, edit these blocks in the config:

  • the three family representative blocks under layer_d_estimator_params;
  • the explicit xgb_params blocks for the ML finalists;
  • each model’s annual_vol_target;
  • layer_e_no_trade_band and layer_e_no_trade_band_by_model;
  • layer_e_gross_exposure_cap and layer_e_gross_exposure_cap_by_model.

Run Layer D:

python scripts/run_layer_d_model_est.py \
  --config configs/run_cost_aware_final_comparison.yaml \
  --output-dir data/processed/cost_aware_final_family_comparison/layer_d

Then run Layer E:

python scripts/run_layer_e_backtest.py \
  --config configs/run_cost_aware_final_comparison.yaml \
  --weights data/processed/cost_aware_final_family_comparison/layer_d/weights_M.parquet \
  --output-dir data/processed/cost_aware_final_family_comparison/layer_e

As with any Layer E run, the output bundle includes both the frictionless target-book path and the cost-aware executed-book path.

13. Layer E — backtest, costs, plots, and factor regressions

Layer E reads:

data/processed/weights_M.parquet
data/processed/panel_combined_M.parquet/
data/external/ken_french/*.csv

or the explicit paths passed by CLI.

Default run:

python scripts/run_layer_e_backtest.py --config configs/bulk_full_template.yaml

Optional CLI overrides:

# Evaluate only a subset of models
python scripts/run_layer_e_backtest.py --config configs/bulk_full_template.yaml --models kim_etal,ensemble_average

# Override paths if needed
python scripts/run_layer_e_backtest.py   --config configs/bulk_full_template.yaml   --weights data/processed/weights_M.parquet   --panel data/processed/panel_combined_M.parquet   --factors-dir data/external/ken_french   --output-dir data/processed/layer_e_custom

Expected outputs under the selected Layer E output directory:

date_level_metrics.parquet / .csv
summary_by_window.parquet / .csv
summary_by_window_raw.parquet / .csv
exposure_summary_by_window_frictionless.parquet / .csv
exposure_summary_by_window_cost_aware.parquet / .csv
gross_exposure_distribution_by_window_frictionless.parquet / .csv
gross_exposure_distribution_by_window_cost_aware.parquet / .csv
monthly_turnover_distribution_by_window.parquet / .csv
monthly_active_holdings_distribution_by_window.parquet / .csv
annual_returns.parquet / .csv
factor_regression_summary.parquet / .csv
factor_regression_coefficients.parquet / .csv
manifest.json
plots/<window>/...

Return and cost accounting

Layer E maps each formation date to the next panel date and labels realized portfolio returns by the end-of-holding-period date.

The headline series are:

gross_return
net_return
turnover
trading_cost
long_exposure
short_exposure
gross_exposure
net_exposure

Gross returns use the target-weight book:

  • long weights use ret_fwd;
  • short weights use ret_fwd_short_pnl.

The cost-aware path uses a drift-aware executed book. Before each rebalance, previously executed weights drift with realized asset returns. The engine then applies the optional no-trade band and charges costs only on trades that are executed:

trading_cost_i,t = layer_e_half_spread_multiplier * cs_spread_i,t * abs(delta_weight_i,t)

The cost-aware path is useful for sensitivity analysis, but it is not a complete trading simulator. It does not include borrow fees, financing, locate risk, forced buy-ins, market impact, or institutional execution constraints.

Factor regressions

Layer E runs monthly time-series regressions with Newey-West standard errors for:

CAPM
FF3
FF3_UMD
FF5
FF5_UMD

The intercept is reported as the factor-adjusted return, or alpha. layer_e_nw_lags controls the Newey-West lag length.

14. Workflow used for the website-style analysis

A practical reproduction sequence is:

# 1. Build real-data inputs
python scripts/run_layer_a_download.py --config configs/bulk_full_template.yaml
python scripts/run_layer_b_build_panel.py --config configs/bulk_full_template.yaml
python scripts/run_layer_c_features.py --config configs/bulk_full_template.yaml

# 2. Optional D0 signal screening.
#    D0 is an orchestration wrapper: for each candidate it runs Layer D,
#    then runs Layer E, and writes candidate leaderboards under data/processed/d0/<stage>/.
python scripts/run_layer_d0_tuning.py \
  --config configs/bulk_full_template.yaml \
  --tuning-spec configs/d0_tuning_template.yaml \
  --stage d00

# Edit data/processed/d0/d00/shortlist_template.yaml by filling in candidate_ids.

python scripts/run_layer_d0_tuning.py \
  --config configs/bulk_full_template.yaml \
  --tuning-spec configs/d0_tuning_template.yaml \
  --stage d01 \
  --shortlist data/processed/d0/d00/shortlist_template.yaml

# 3. Optional post-D0 cost tuning on selected representatives.
#    Cost tuning runs Layer D once per volatility target, then reuses those
#    weights while sweeping Layer E no-trade-band and gross-exposure-cap settings.
python scripts/run_layer_d_cost_tuning.py \
  --config configs/bulk_full_template.yaml \
  --tuning-spec configs/cost_tuning_template.yaml

# 4. Paper-comparable reporting bundles
python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_replication_kim.yaml \
  --output-dir data/processed/frictionless_replication_kim/layer_d
python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_replication_kim.yaml \
  --weights data/processed/frictionless_replication_kim/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_replication_kim/layer_e \
  --models kim_etal

python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_replication_kim_10pct.yaml \
  --output-dir data/processed/frictionless_replication_kim_10pct/layer_d
python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_replication_kim_10pct.yaml \
  --weights data/processed/frictionless_replication_kim_10pct/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_replication_kim_10pct/layer_e \
  --models kim_etal

# 5. Final family comparisons
python scripts/run_layer_d_model_est.py \
  --config configs/run_frictionless_final_comparison.yaml \
  --output-dir data/processed/frictionless_final_family_comparison/layer_d
python scripts/run_layer_e_backtest.py \
  --config configs/run_frictionless_final_comparison.yaml \
  --weights data/processed/frictionless_final_family_comparison/layer_d/weights_M.parquet \
  --output-dir data/processed/frictionless_final_family_comparison/layer_e

python scripts/run_layer_d_model_est.py \
  --config configs/run_cost_aware_final_comparison.yaml \
  --output-dir data/processed/cost_aware_final_family_comparison/layer_d
python scripts/run_layer_e_backtest.py \
  --config configs/run_cost_aware_final_comparison.yaml \
  --weights data/processed/cost_aware_final_family_comparison/layer_d/weights_M.parquet \
  --output-dir data/processed/cost_aware_final_family_comparison/layer_e

The D0 and cost-tuning stages are optional if you only want to run the final configs as currently shipped. They are included because they document how the model representatives and implementation settings were selected.

The D0 and cost-tuning commands above are higher-level loops around the same Layer D and Layer E pipeline code. The explicit Layer D/E commands in the reporting sections are for fixed, paper-comparable and final comparison bundles after the screening/tuning choices have been selected.

15. Interpreting the website-oriented results

The website reports two main kinds of findings:

  1. Reproduction / persistence: a Sharadar-based point-in-time implementation recovers the central Kim et al. frictionless result and shows that the signal remains visible after the original paper period.
  2. Implementation caveat: the paper-comparable portfolio has high gross exposure and turnover, so implementation-aware constraints and spread costs matter. The XGBoost extensions show some gross-return promise but do not deliver a robust net-of-cost improvement on the final holdout.

The repository is structured to make those conclusions auditable:

  • Layer B controls the point-in-time panel and return construction.
  • Layer C reports feature coverage and writes a manifest.
  • Layer D writes rebalance diagnostics and portfolio weights.
  • D0 separates signal screening from final evaluation.
  • Cost tuning separates implementation-lever selection from the final holdout.
  • Layer E saves return, exposure, turnover, cost, regression, and plot outputs for every run.

16. Identifiers and share-class handling

The pipeline uses Sharadar permaticker as the operational security identifier. Ticker changes do not create new securities if the permaticker remains the same.

Layer B chooses one ticker per permaticker-date when multiple ticker rows are available, prioritizing same-date market capitalization when available. However, economically related share classes can have different permatickers, so the pipeline does not automatically collapse issuers such as dual-class structures. If issuer-level consolidation is desired, it should be added as a separate documented step.

17. Common gotchas

  • Run commands from the repository root.
  • Do not commit .env or real data/ outputs.
  • DGS3MO.csv must be present before Layer B if you want risk-free and excess-return fields to be populated.
  • Ken French files must be present before Layer E.
  • D0 and cost tuning can be expensive because they repeatedly run Layer D and Layer E over many candidates.
  • The final reporting configs assume Layer C has already built data/processed/panel_combined_M.parquet/.
  • If you change the feature set or exclusions, rerun Layer C before rerunning Layer D.
  • If you change Layer B timing assumptions, risk-free inputs, or delisting assumptions, rerun Layer B and then rerun C/D/E.

18. References

  • Kim, Soohun, Robert A. Korajczyk, and Andreas Neuhierl. 2021. “Arbitrage Portfolios.” Review of Financial Studies.
  • Freyberger, Joachim, Andreas Neuhierl, and Michael Weber. 2020. “Dissecting Characteristics Nonparametrically.” Review of Financial Studies.
  • Corwin, Shane A., and Paul Schultz. 2012. “A Simple Way to Estimate Bid-Ask Spreads from Daily High and Low Prices.” Journal of Finance.