[segger-io] Output a more "complete" transcripts.parquet

[Tobiaspk]

Currently, segger outputs a segger_segmentation.parquet file, which contains only the row_index and segger derived similarity and thresholds. If users want to use this data for downstream analyses, they need to be confident in handling this data:

1. Add row index to the input transcripts
2. Merge input transcripts and output segger values
3. Filter by qv and segger_similarity thresholds

Example:

import polars as pl
from segger.io import get_preprocessor, StandardTranscriptFields

std = StandardTranscriptFields()
tx = get_preprocessor(source_path).transcripts       # polars DataFrame; qv >= 20 already applied
seg = pl.read_parquet("outputs/segger_segmentation.parquet")

merged = tx.join(seg, on=std.row_index, how="left")  # row_index aligns rows across both tables
assigned = merged.filter(
    pl.col("segger_cell_id").is_not_null()
    & (pl.col("segger_similarity") >= pl.col("similarity_threshold"))  # per-gene threshold

If the order is shuffled, or a step skipped, the resulting dataset will be noisy or plainly wrong. See #4 which accidentally skipped an important step during anndata creation.

I'd like to propose to store a "complete" anndata file, which contains the columns [x, y, feature_name, cell_id, filtered, segger_similarity, segger_threshold] (maybe more?), and has the same shape and order as the input transcripts. filtered should be True if a transcript is assigned (qv > 20 per default, similarty > threshold, converged = True, cell_id != None).

Some benefits:

• For less experienced segger users and bots, this will significantly simplify and reduce errors when working with segger outputs
• Segger outputs can be shared, without necessarily requiring the input dataset

Open for discussion. @EliHei2 @andrewmoorman

[MeyerBender]

Hi,

I think this is a really good idea, I was just wondering if this is also already done in the spatialdata objects that segger outputs? Currently, I am manually assembling a spatialdata object from the anndata and the transcripts.parqut files, however this is quite error-prone, especially due to the filtering that differs between the two. Having one consolidated spatialdata object would be super helpful for a lot of downstream applications.