Skip to content

Add TW3Cast results - #209

Open
nattanioCalor wants to merge 1 commit into
SalesforceAIResearch:mainfrom
nattanioCalor:add-tw3cast
Open

nattanioCalor wants to merge 1 commit into
SalesforceAIResearch:mainfrom
nattanioCalor:add-tw3cast

Conversation

@nattanioCalor

Copy link
Copy Markdown

Adds results for TW3Cast (TW3 Partners): a frozen per-configuration router over lightly fine-tuned experts (Chronos-2, TiRex, Toto 2.5B LoRA), no agent and no LLM at inference, no test-data leakage in any component. Scores computed with the official harness (gluonts evaluate_forecasts, leaderboard parameters). Code and router table: https://github.com/TW3-Partners-OS/TW3-Cast

@salesforce-cla

Copy link
Copy Markdown

Thanks for the contribution! Unfortunately we can't verify the commit author(s): Nathan <n***@t***.com>. One possible solution is to add that email to your GitHub account. Alternatively you can change your commits to another email and force push the change. After getting your commits associated with your GitHub account, sign the Salesforce Inc. Contributor License Agreement and this Pull Request will be revalidated.

@cuthalionn

Copy link
Copy Markdown
Contributor

Hi @nattanioCalor,

Thanks for the submission! Frankly this is the first time we get a mixture of routing + finetuning submission so I am not sure about what to label it, but agentic seems like a better fit to me since it is in the end routing multiple models and seems like a fairer comparison to other models in the same family in the leaderboard. Would you be okay with making that change?

FYI: I started working on replicating the results.

@cuthalionn

Copy link
Copy Markdown
Contributor

Hi @nattanioCalor,

Separately on replication I tried to give it a go on a smaller subset and realised that certain parts of the code is not released? For bizitobs_application/10S/{medium,long}, the public router specifies tournament, but the repo does not provide the tournament code and winners are not published (please correct me if I am wrong). Any suggestions how to proceed?

Moreover I tried m4 weekly which does not require a tournament:

Model Dataset/frequency/term Rep MASE Submitted MASE Δ MASE Rep WSQL Submitted WSQL Δ WSQL
TW3Cast m4_weekly/W/short 1.906935 1.869772 +0.037163 0.035549 0.035317 +0.000232

I followed the public TW3-Cast README and used the published expert checkpoints and router configuration, I used the published frozen Toto/TimesFM route and got ~0.03 diff.

Would appreciate your help to address these problems. It would make life much easier for me if the repo had all code available and a single entrypoint that lets me replicate any dataset. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants