Proposal: Stop using `AnnData.raw`

[ivirshup]

I would like to see data distributed with cellxgene stop using .raw.X for counts and to instead put this matrix in adata.layers["counts"]. I would ideally like to get this schema change in for 6.0.

For some background, raw was initially put in anndata so the user could have a "normalized" and "raw" copy of the matrix. The workflow back in the day also assumed you may only be interested in normalized values for a subset of selected features – largely due to memory constraints and the use of densifying normalization methods. However, people generally need all of their features normalized for downstream methods that are used across all features (e.g. differential expression/ plotting/ enrichment) and densifying normalization methods are less popular now. In addition, Anndata has since added the .layers attribute which allows storing multiple matrices.

Within scverse, we would like to eventually get rid of the .raw entry as a whole. It's confusing to users, has difficult semantics (e.g. it's assumed to be read-only, but we can't actually enforce that), and is easily replaced by existing functionality/ just using a separate object. In addition, we have stopped developing features for .raw (including improved out-of-core compute support) a while ago and will not be adding more features.

Because cellxgene stores a matrix in .raw.X with the same shape as .X, I don't see any barrier to moving this over. What is gained is better support within scanpy api, out-of-core support, and better usability.

[brianraymor]

Because cellxgene stores a matrix in .raw.X with the same shape as .X

Small clarification per the table in X (Matrix Layers), the raw matrix may be stored in .X when no normalized matrix is submitted:

Assay	"raw" required?	"raw" location	"normalized" required?	"normalized" location
scRNA-seq (UMI, e.g. 10x multiome, 10x v3, Slide-seqV2)	REQUIRED. Values MUST be de-duplicated molecule counts. Each cell MUST contain at least one non-zero value. All non-zero values MUST be positive integers stored as numpy.float32.	AnnData.raw.X unless no "normalized" is provided, then AnnData.X	STRONGLY RECOMMENDED	AnnData.X
Visium Spatial (e.g. V1, CytAssist)	REQUIRED. Values MUST be de-duplicated molecule counts. All non-zero values MUST be positive integers stored as numpy.float32. If uns['spatial']['is_single'] is False then each cell MUST contain at least one non-zero value. If uns['spatial']['is_single'] is True then the unfiltered feature-barcode matrix (raw_feature_bc_matrix) MUST be used. See Space Ranger Feature-Barcode Matrices. if assay_ontology_term_id is "EFO:0022860" for Visium CytAssist Spatial Gene Expression, 11mm, this matrix MUST contain 14336 rows; otherwise, this matrix MUST contain 4992 rows. If the obs['in_tissue'] value is 1, then the cell MUST contain at least one non-zero value. If any obs['in_tissue'] values are 0, then at least one cell corresponding to a obs['in_tissue'] with a value of 0 MUST contain a non-zero value.	AnnData.raw.X unless no "normalized" is provided, then AnnData.X	STRONGLY RECOMMENDED	AnnData.X
scRNA-seq (non-UMI, e.g. SS2)	REQUIRED. Values MUST be one of read counts (e.g. FeatureCounts) or estimated fragments (e.g. output of RSEM). Each cell MUST contain at least one non-zero value. All non-zero values MUST be positive integers stored as numpy.float32.	AnnData.raw.X unless no "normalized" is provided, then AnnData.X	STRONGLY RECOMMENDED	AnnData.X
unpaired Accessibility (e.g. ATAC-seq, mCT-seq)	NOT REQUIRED		REQUIRED	AnnData.X
AnnData.raw.X unless no "normalized" is provided, then AnnData.X

[ejmolinelli]

I like this. I think that it is semantically clearer to call it "counts", which has a more specific interpretation than does "raw".

From a validator perspective, I think it would be a fairly straightforward, albeit high-touch change (there are several hundred instances of 'raw' in the codebase, excluding markdown).

[brian-mott]

I really like this as well.

From a curator perspective, this would greatly improve our resource use. Wrangling of .raw is the most memory-intensive step of curation since like Isaac mentioned above, it is essentially read-only. For every dataset, we need to filter the raw.var index to approved Ensembl IDs and/or backfill the normalized matrix with 0s so the matrices are the same shape. We have to completely copy the AnnData object to do this. We would also get rid of the an almost duplicate dataframe by dropping .raw.var

Once curation/data downside would be the potential loss of some raw data. Currently, it is fairly common to get an AnnData object with a normalized matrix with say 5-10K most-expressed genes, and then the raw.X matrix with 2-4X as many genes. We can easily wrangle both matrices to the same shape to retain all the raw data. But with no .raw.X, we may not initially get that larger raw matrix. It's not a huge ask to request this, but some submitters may not readily have it available, it could add more time to the process, or in the worst case, we never get it.

Since a raw count matrix is always required for submission, would it make sense to have .X always contain the count matrix and for a normalized matrix to be a layer? Within CELLxGENE this seems like it could simplify lots of checks for where the raw matrix resides, but I don't know how helpful this is for users. If the research community expects a normalized matrix in .X and counts to be in layers['counts'], then this could be too disruptive to scverse and the surrounding ecosystem.

[Jchaffer787]

For the benefits to resource use I also like this, assuming we can minimize the potential raw data loss that Brian M mentioned.

I also think its important to note that this will require making changes to many of the curation and wrangling tools we use. For most of these, such as map_filter_gene_ids and evaluate_dup_counts from cellxgene_mods this should be a straight forward update. For other, such as the flattener, the changes required could be more extensive.

[ivirshup]

@brian-mott

We have to completely copy the AnnData object to do this. We would also get rid of the an almost duplicate dataframe by dropping .raw.var

Yes completely, in my mind my proposal is to remove all of .raw.

Since a raw count matrix is always required for submission, would it make sense to have .X always contain the count matrix and for a normalized matrix to be a layer? Within CELLxGENE this seems like it could simplify lots of checks for where the raw matrix resides, but I don't know how helpful this is for users. If the research community expects a normalized matrix in .X and counts to be in layers['counts'], then this could be too disruptive to scverse and the surrounding ecosystem.

Most methods default to using .X and I think most methods expect normalized data, so I think it makes the most sense to keep normalized values here. I think it could also be reasonable to put both entries only in layers (e.g. .layers["counts"], .layers["normalized"] and to have no .X. This just forces people to be explicit.

[Zethson]

Most methods default to using .X and I think most methods expect normalized data, so I think it makes the most sense to keep normalized values here. I think it could also be reasonable to put both entries only in layers (e.g. .layers["counts"], .layers["normalized"] and to have no .X. This just forces people to be explicit.

Phil wants to pick up scverse/anndata#1707 again soon.

[pablo-gar]

In support of the change but there is one gotcha. This from @ivirshup is not guaranteed

Because cellxgene stores a matrix in .raw.X with the same shape as .X, I don't see any barrier to moving this over. What is gained is better support within scanpy api, out-of-core support, and better usability.

Currently cellxgene schema allows differences in number of genes in .X vs .raw.X, hence feature_is_filtered.

I don't know if any data in CxG actually has different gene sets for .X vs .raw.X but the schema does allow it so in the migration this need to be checked and resolved for.

And one preference of mine, I think it would be better to always store counts in .X and normalized layer in .layers["normalized"]. Since the counts are required but no the normalized values, this way we guarantee that .X will always be consistent in its content, instead of flipping between having normalized values vs count values