Upgrade training interactions - #279
Merged
Merged
Conversation
added 3 commits
July 30, 2026 10:26
Adds first-class support for string predictions/labels (e.g. an LLM's generated reply as "prediction", a reference/target string as "label") throughout the framework, additive alongside the existing numeric classification/segmentation/detection path so nothing there changes. - src.py: save_signals' to_numpy/normalize no longer force strings through the classification-label uint16 cast (which raised ValueError on non-numeric text) -- native str values pass through unwrapped. - trainer_tools.py: get_data_set_representation routes string label/prediction values to new RecordMetadata.sample_label_text / sample_prediction_text proto fields instead of crashing on int(target)/int(pred). Numeric tasks are unaffected. - data_service.py: _process_sample_row's classification/tabular label/prediction branches emit a string DataStat (type="string", value_string=...) for text values instead of silently dropping them after a failed float() cast. Also fixes a latent bug in the "is this label empty" file-path heuristic that misclassified ordinary multi-sentence text (any string with a non-trailing '.') as an empty label to be reloaded -- extracted to looks_like_file_path_label() and narrowed to require a short alphanumeric extension, so it still detects real file paths (e.g. "mask.png") without misfiring on text. - experiment_service.proto: additive repeated string sample_label_text/sample_prediction_text fields on RecordMetadata; regenerated pb2 stubs with the pinned grpcio-tools~=1.68 toolchain (matches this repo's documented Colab-compatibility constraint). Full test suite passes (1018 passed, 133 skipped, 0 failures).
process_sample() (used by the GetSamples/BatchSampleRequest endpoint for
image thumbnails) unconditionally ran torch.tensor(tensor) on whatever a
dataset's __getitem__/_getitem_raw returned as its first element. For an
image dataset that's the image tensor; for a generative/RLHF prompt
dataset (see wl-llm-rlhf) it's the prompt STRING, and torch.tensor() on
a string raises ("new(): invalid data type 'str'") -- caught internally,
so it didn't crash the server, but logged an ERROR for every sample of
a text dataset and returned nothing useful.
There's genuinely no image to preview for a text-only sample (its
content shows via the pred/target DataStat string fields added earlier
on this branch, not this thumbnail path), so this is an expected
"nothing to render here" case, not a failure -- return the same empty
result the exception handler already returned, without the error log.
Confirmed against a live wl-llm-rlhf training run: a manual GetSamples
call for in-flight sample_ids logged this exact error before the fix.
…11448")
sample_id is always stored as a string column (save_signals builds
batch_ids_np via str(i) for i in batch_ids), but a user naturally types
an unquoted integer in the UI's quick filter/query box. df.query("sample_id
== 11448") does NOT raise -- pandas happily evaluates the string "11448"
== int 11448 as False for every row -- so it "succeeds" with a silently
empty result instead of ever reaching the existing _mask_from_coerced_query
fallback, which only triggers on an exception. The UI reported "0 of N
samples" for an id that demonstrably exists.
Retry once against the coerced view when a query succeeds but returns zero
rows on a non-empty dataframe. Scoped to numeric-looking object columns
only (via the existing _mask_from_coerced_query helper), so genuine text
columns (e.g. the prediction/target strings from this branch's earlier
commit) and genuinely-empty-on-purpose queries are unaffected.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends WeightsLab’s training/data interaction pipeline to support text labels and text predictions (e.g., prompt/ground-truth/generated replies for LLM-style generative tasks) without breaking existing numeric-label workflows.
Changes:
- Add new
RecordMetadataproto fields for additive text labels/predictions (sample_label_text,sample_prediction_text) while keeping existing int32 fields unchanged. - Update signal normalization and sample-stat/data-stat construction paths to preserve and render text labels/predictions end-to-end (instead of crashing on numeric coercion).
- Improve
df.querybehavior for numeric-lookingsample_idfilters against string-typedsample_idcolumns, and add targeted regression tests.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| weightslab/trainer/trainer_tools.py | Routes string labels/preds into new RecordMetadata text fields; avoids crashing on text-only dataset samples in process_sample. |
| weightslab/trainer/services/data_service.py | Adds heuristic for string label “emptiness” detection, emits string DataStats for text label/pred, and improves df.query fallback for sample_id. |
| weightslab/src.py | Preserves Python str for text preds/targets during normalization so downstream isinstance(x, str) checks work correctly. |
| weightslab/proto/experiment_service.proto | Adds sample_label_text / sample_prediction_text fields to RecordMetadata. |
| weightslab/proto/experiment_service_pb2.py | Regenerated protobuf stubs to include new fields. |
| weightslab/proto/experiment_service_pb2_grpc.py | Regenerated grpc stubs/version guards. |
| tests/trainer/test_trainer_tools.py | Adds coverage for _numeric_or_text_list, text label/pred routing, and text-only dataset behavior in process_sample. |
| tests/trainer/services/test_data_service_text_predictions.py | New unit tests for text DataStats emission and the file-path-vs-free-text heuristic. |
| tests/trainer/services/test_data_service_sample_id_query.py | New regression tests for sample_id == 11448 query behavior. |
| tests/general/test_signals.py | Adds regression test ensuring save_signals forwards native str preds/targets without crashing. |
Files not reviewed (1)
- weightslab/proto/experiment_service_pb2.py: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consider preds or labels text also (e.g., prompts, ground truth, and preds for LLMs)