chore(assessment): Enhance cost tracking features

[vprashrex]

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

• Before: Cost estimation and tracking were not implemented for assessment processing.
• Now: Introduced functions for estimating stage costs and tracking costs in pre-filter and assessment stages.
• Added RunCost and RunPreFilterCost models for cost management.
• Refactored assessment configuration resolution for better model name retrieval.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

Original PR description

• Introduced functions to estimate stage costs based on input/output tokens and model pricing.
• Added cost tracking for pre-filter and assessment stages, ensuring costs are recorded in the execution bag.
• Updated models to include RunCost and RunPreFilterCost for better cost management.
• Refactored assessment configuration resolution to streamline model name retrieval for L2 assessments.
• Enhanced validation for input schema to support strict column requirements.
• Updated tests to cover new cost tracking functionality and ensure accurate cost calculations across stages.
• Improved documentation to reflect changes in cost handling and assessment processing.

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

Explain the motivation for making this change. What existing problem does the pull request solve?

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Please add here if any other information is required for the reviewer.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9d4fcb7e-a267-4ac3-a79c-6b657a815ce3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   🔴 8 breaking changes

Caution

Downstream consumers may need an update before merging.

Breaking changes  ·  8

	Method	Path	Change
🔴	GET	/api/v1/assessment/assessments	removed the optional property data/anyOf[subschema #1]/items/executions from the response with the 200 status
🔴	GET	/api/v1/assessment/assessments/{assessment_id}	removed the optional property data/anyOf[subschema #1: AssessmentPublic]/executions from the response with the 200 status
🔴	GET	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1]/items/batch_job_id from the response with the 200 status
🔴	GET	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1]/items/execution from the response with the 200 status
🔴	GET	/api/v1/assessment/runs/{run_id}	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/batch_job_id from the response with the 200 status
🔴	GET	/api/v1/assessment/runs/{run_id}	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/execution from the response with the 200 status
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/batch_job_id from the response with the 200 status
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/execution from the response with the 200 status

Full changelog  ·  53

	Method	Path	Change
🔴	GET	/api/v1/assessment/assessments	removed the optional property data/anyOf[subschema #1]/items/executions from the response with the 200 status
🔴	GET	/api/v1/assessment/assessments/{assessment_id}	removed the optional property data/anyOf[subschema #1: AssessmentPublic]/executions from the response with the 200 status
🔴	GET	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1]/items/batch_job_id from the response with the 200 status
🔴	GET	/api/v1/assessment/runs	removed the optional property data/anyOf[subschema #1]/items/execution from the response with the 200 status
🔴	GET	/api/v1/assessment/runs/{run_id}	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/batch_job_id from the response with the 200 status
🔴	GET	/api/v1/assessment/runs/{run_id}	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/execution from the response with the 200 status
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/batch_job_id from the response with the 200 status
🔴	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	removed the optional property data/anyOf[subschema #1: AssessmentRunPublic]/execution from the response with the 200 status
🟢	—	—	removed the schema RunExecution
🟢	GET	/api/v1/assessment/assessments	added the optional property data/anyOf[subschema #1]/items/dataset_id to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments	added the optional property data/anyOf[subschema #1]/items/dataset_name to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments	added the optional property data/anyOf[subschema #1]/items/error_message to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments	added the optional property data/anyOf[subschema #1]/items/run_stats to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments	added the required property data/anyOf[subschema #1]/items/counts to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments/{assessment_id}	added the optional property data/anyOf[subschema #1: AssessmentPublic]/dataset_id to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments/{assessment_id}	added the optional property data/anyOf[subschema #1: AssessmentPublic]/dataset_name to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments/{assessment_id}	added the optional property data/anyOf[subschema #1: AssessmentPublic]/error_message to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments/{assessment_id}	added the optional property data/anyOf[subschema #1: AssessmentPublic]/run_stats to the response with the 200 status
🟢	GET	/api/v1/assessment/assessments/{assessment_id}	added the required property data/anyOf[subschema #1: AssessmentPublic]/counts to the response with the 200 status
🟢	GET	/api/v1/assessment/datasets/{dataset_id}/rows	endpoint added
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/cost to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/dataset_id to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/dataset_name to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/experiment_name to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/input to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/pipeline to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/prefilter_total_passed to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/prefilter_total_rejected to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/prefilter_total_rows to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/stage to the response with the 200 status
🟢	GET	/api/v1/assessment/runs	added the optional property data/anyOf[subschema #1]/items/stage_status to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/cost to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/dataset_id to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/dataset_name to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/experiment_name to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/input to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/pipeline to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_passed to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_rejected to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_rows to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/stage to the response with the 200 status
🟢	GET	/api/v1/assessment/runs/{run_id}	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/stage_status to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/cost to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/dataset_id to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/dataset_name to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/experiment_name to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/input to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/pipeline to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_passed to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_rejected to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/prefilter_total_rows to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/stage to the response with the 200 status
🟢	PATCH	/api/v1/assessment/runs/{run_id}/post-processing	added the optional property data/anyOf[subschema #1: AssessmentRunPublic]/stage_status to the response with the 200 status

_{main ↔ 354f1672 · generated by oasdiff}