various fixes and additions for IO #19

giovp · 2023-02-27T12:55:30Z

IO fixes for xenium
IO fixes for visium
add mcmicro
add steinbock

should close respective PRs once this is merged

codecov-commenter · 2023-02-27T13:05:03Z

Codecov Report

Merging #19 (3907d68) into main (e061d47) will decrease coverage by 2.36%.
The diff coverage is 32.38%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #19      +/-   ##
==========================================
- Coverage   44.90%   42.54%   -2.36%     
==========================================
  Files          11       13       +2     
  Lines         432      604     +172     
==========================================
+ Hits          194      257      +63     
- Misses        238      347     +109

Impacted Files	Coverage Δ
src/spatialdata_io/readers/_utils/_read_10x_h5.py	`19.14% <ø> (ø)`
src/spatialdata_io/readers/_utils/_utils.py	`28.26% <0.00%> (-0.63%)`	⬇️
src/spatialdata_io/readers/cosmx.py	`19.08% <7.59%> (-11.51%)`	⬇️
src/spatialdata_io/readers/visium.py	`35.00% <13.63%> (+0.15%)`	⬆️
src/spatialdata_io/readers/xenium.py	`34.06% <17.85%> (-2.93%)`	⬇️
src/spatialdata_io/readers/steinbock.py	`40.90% <40.90%> (ø)`
src/spatialdata_io/readers/mcmicro.py	`43.18% <43.18%> (ø)`
src/spatialdata_io/__init__.py	`100.00% <100.00%> (ø)`
src/spatialdata_io/_constants/_constants.py	`100.00% <100.00%> (ø)`

src/spatialdata_io/readers/cosmx.py

LucaMarconato · 2023-02-28T11:06:43Z

src/spatialdata_io/readers/mcmicro.py

+        imread_kwargs,
+        image_models_kwargs,
+    )
+    labels[f"{dataset_id}_nuclei"] = _get_labels(


Just one note (nothing to do): in the example the labels of the nuclei coincide with the other cell labels.

LucaMarconato · 2023-02-28T11:52:09Z

src/spatialdata_io/readers/visium.py

-        shape_size=scalefactors["spot_diameter_fullres"],
-        index=adata.obs_names,
+        geometry=0,
+        radius=np.repeat(scalefactors["spot_diameter_fullres"], len(adata)),


I would modify the parser signature to allow also a single float here. This actually already works because when we assign a float to the column in the dataframe the value is copied for each row.

true it works already, clarified there

LucaMarconato · 2023-02-28T14:18:40Z

src/spatialdata_io/readers/xenium.py

    adata.obs = metadata
-    transform = Scale([1.0 / specs["pixel_size"], 1.0 / specs["pixel_size"]], axes=("x", "y"))
-    diameters = 2 * np.sqrt(adata.obs[XeniumKeys.CELL_AREA].to_numpy() / np.pi) / specs["pixel_size"]
-    circles = ShapesModel.parse(


I am also not a fan of this but I have restored it because this is the only way that we have at the moment to view the data with napari. The polygons are too slow, takes minutes to render the napari view. One option could be to rasterize the polygons into labels. Shall I do it? (I started working on rasterization already so I could code it today)

I load the circles for the nuclei (not the cell boundaries anymore) otherwise the circles are overlapping in napari.

There was a bug in the radius size, fixed.

I mean I understand the napari thing but can't it be done temporarily there in case? I don't think it's any useful and it's not really an element provided by the tech but some type of processing, would keep it removed

I can think of a few alternative options:

we keep this code in the io until a better solution

we make the user manually create circles every time he/she is working with Xenium data (in particular the has to manually pass the radius column to the parser of circles)

I implement the rasterization of polygons to labels and this is called by napari when too many polygons are detected. This function is probably slow, so better if the user calls it beforehand (or even if we call it in cosmx(): if I can implement it lazily that would be the best, so cosmx() returns fast and the time is used only when saving to zarr.)

napari shows only the centroids (it can't show the shape sizes since it doesn't know that the radii are stored in a column of the table).

We make a function to compute the area of polygons and we create a function to create circles with radius from the polygons. As above, we either call this function automatically in napari, either we call it in cosmx(), either the user will call it manually if needed.

wdyt?

ok fine for leaving it in, would add an argument like centroids as shapes then to have it at least optional

Yes I would add the arguments. I will implement gradually the rasterization of labels and the "circularization" of polygons and labels, then we can test what works best.

LucaMarconato · 2023-02-28T15:39:58Z

src/spatialdata_io/readers/xenium.py


    transform = Scale([1.0 / specs["pixel_size"], 1.0 / specs["pixel_size"]], axes=("x", "y"))
-    # points = PointsModel.parse(coords=arr, annotations=annotations, transformations={"global": transform})
    points = PointsModel.parse(


Here we get

INFO Instance key `cell_id` could be of type `pd.Categorical`. Consider casting it.

this warning would keep for now and then maybe remove

LucaMarconato · 2023-02-28T15:51:43Z

I made some changes in spatialdata-sandbox (in this small PR: https://github.com/giovp/spatialdata-sandbox/pull/17/files), in napari_spatialdata (simply pushed to the spatialdata branch, and to this PR. Now I push my edits of this PR.

for more information, see https://pre-commit.ci

LucaMarconato · 2023-02-28T15:57:07Z

In my edits I in particular added the coordinate systems to steinbock_io and changed all the values of region, and all the rows in the region_key column to have the full path, for instance instead of cells, we need /labels/cells, etc. Probably we should go for unique names soon (scverse/spatialdata#124) and simplify that.

giovp · 2023-03-03T10:24:27Z

ok great, gonna check that all files are consistent mirroring spatialdata sandbox and then will merge.

giovp · 2023-03-04T20:32:37Z

one thing is that I added spatial coordinates in obsm in every tech, mostly because then we can use squidpy out of the box.

will merge this even without re-review hope it's fine

Co-authored-by: Luca Marconato <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

giovp added 15 commits January 14, 2023 18:29

add steinbock

a485be6

add steinbock

382dec0

updates

e260952

fixes

5631648

Merge branch 'io/nanostring' into io/steinbock

62af10e

update constants and init

4cd565b

fix init

6c60a02

update

742b7df

update visium and xenium

9f16a34

readd images in xenium

3447c9e

Merge branch 'io/steinbock' into fixes/io

15ad9d1

fix steinbock

da512d6

Merge branch 'io/mcmicro' into fixes/io

4dacd1f

add mcmicro

aac84f0

pre-commit fixes

e761e85

giovp requested a review from LucaMarconato February 27, 2023 13:04

giovp added 3 commits February 27, 2023 15:47

minor fixes cosmx

2615da1

remove temp dir

cfd12f2

io fixes

31b1d2a

giovp mentioned this pull request Feb 27, 2023

minor fix for points model parser scverse/spatialdata#157

Merged

LucaMarconato reviewed Feb 28, 2023

View reviewed changes

src/spatialdata_io/readers/cosmx.py Show resolved Hide resolved

LucaMarconato requested changes Feb 28, 2023

View reviewed changes

LucaMarconato and others added 2 commits February 28, 2023 16:52

tested all technologies with napari and added fixes

40ca0b8

[pre-commit.ci] auto fixes from pre-commit.com hooks

0560498

for more information, see https://pre-commit.ci

giovp added 2 commits March 3, 2023 14:25

minor fixes

b05fd8d

fixes

f66f522

add spatial key in obsm

3907d68

giovp merged commit 61177e7 into main Mar 4, 2023

giovp deleted the fixes/io branch March 4, 2023 20:32

This was referenced Mar 4, 2023

fixes and enhancements for IO #17

Closed

add steinbock #13

Closed

add mcmicro #14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

various fixes and additions for IO #19

various fixes and additions for IO #19

giovp commented Feb 27, 2023 •

edited

Loading

codecov-commenter commented Feb 27, 2023 •

edited

Loading

LucaMarconato Feb 28, 2023

LucaMarconato Feb 28, 2023

giovp Mar 3, 2023

LucaMarconato Feb 28, 2023

LucaMarconato Feb 28, 2023

LucaMarconato Feb 28, 2023

giovp Mar 3, 2023

LucaMarconato Mar 3, 2023

giovp Mar 3, 2023

LucaMarconato Mar 3, 2023

LucaMarconato Feb 28, 2023

giovp Mar 4, 2023

LucaMarconato commented Feb 28, 2023 •

edited

Loading

LucaMarconato commented Feb 28, 2023

giovp commented Mar 3, 2023

giovp commented Mar 4, 2023

various fixes and additions for IO #19

various fixes and additions for IO #19

Conversation

giovp commented Feb 27, 2023 • edited Loading

codecov-commenter commented Feb 27, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LucaMarconato commented Feb 28, 2023 • edited Loading

LucaMarconato commented Feb 28, 2023

giovp commented Mar 3, 2023

giovp commented Mar 4, 2023

giovp commented Feb 27, 2023 •

edited

Loading

codecov-commenter commented Feb 27, 2023 •

edited

Loading

LucaMarconato commented Feb 28, 2023 •

edited

Loading