Segmentation Masks

[BryonLewis]

resolves #234

• Basic mask endpoint for creation/editing masks
• Client API functions for uploading blob data to create a mask
• Client side button that generates an initial generic star mask right now

TODO:

• RLE Support
  • Back-end updating/conversion from images into RLE data and storage
  • Retrieving RLE data for rendering
  • Client Side tool to conver RLE data to Images
  • Look into webworker or other tool to speed this up.
• Mask Displaying
  • Media endpoint updated to return masks in the mask folders
  • Client side store for storing mask images and pre-caching them
  • I believe I should add into the track/feature type an indication that a mask exists
  • The LayerManager needs to display these images overlaid on top of each other based on track visibility
  • Color scales needs to be applied to the black/white image maybe using SVG filters to align with the track color
  • Conversion to utilize RLE for mask storage: https://github.com/facebookresearch/sam2/blob/main/demo/frontend/src/jscocotools/mask.ts#L26
    • This in testing seemed to be 1/20th of the size
    • Would require generating each image on each frame for rendering
    • Maybe doing some pre-rendering of frames.
• GeoJS Updating Frame Texture
  • GeoJS update to allow webGL texture access for faster rendering
• Client Side Mask Editor
  • Using the Same canvas as geoJS to draw on top of
  • Allows for erasing or drawing
  • Creation of brand new mask for initial layer
  • Brush size changes
  • Adjustment of the layer opacity
  • Check if dynamic panning/zooming and swapping the Canvas to Image work
  • middle mouse click pan
  • scroll wheel zoom in/out
  • keyboard shortcuts (1 pointer, 2, brush, 3, eraser)
  • useMask add to editorFunctions a saveMask. this needs to take the image reference from the maskEditorLayer and calculate the bounding box, update the bounding box and then add the image to the back-end and update the RLE for that track/frame.
  • After drawing a mask it automatically updates the bounding box for the track
  • Save Button for saving a mask after editing it. It should update the image and RLE version on the back-end
  • Delete Button for deletion of a mask.
  • Eventually integration with SAM for magic picking of areas
  • The bounding box should adjust to the extent of the mask automatically.
  • Fix issue when jumping to out of cache image.
  • Segmentation masks as a whole should probably have an opacity indicator as well initial should be 30-40% or so
  • When masks exists the bounding box doesn't need to be displayed. Make it similar to the polygon where it is small less opacity bounding box
  • Visual representation of the brush size
  • Brush size limits corresponding to image size. I.E a 480x480 image size shouldn't have the same max brush size as a 4k image.
• Back-end Mask management
  • Deletion of masks when tracks are deleted from the system, should write a function to do this when processing the save button data
  • Exporting annotations (how to export masks?)
    • Will Export utilizing a Zip file that will export the whole 'masks' folder for dataset including the RLE and the images
    • Update the Export endpoint to export 'masks' as a zip file
    • Update the Client side exporting tool to export masks as well as viame_csv/trackJSON
  • Importing Masks?
    • Zip upload to the base DatasetFolder Id
    • Zip Processor endpoint that will kick of celery job
    • Celery job will will extract Zip and create a new mask folder and extract it to the TrackId->FrameId structure with the proper metadata being set for each folder and file.
• Documentation
  • Mask Editing interface documentation
  • Mask Dataset Structure documentation
    • Include a description of the metadata
  • Importing and exporting the Masks
• Mouse Scroll + CTRL for brush size changes

Issues:

• Masks will cause the initial loading and centering of the image to be off
• Existing mask editing mode and being able to restore the previous interactorOptions
• RLE Masks seem to be rotated so the conversion isn't right from RLE back to an image
• Button to Exit Mask Editing Mode
• Automaticall save when saving/deleting masks in the editor interface
• update Worker RLE version to display image once it is ready
• Do some performance testing on the RLE worker version

[BryonLewis]

I have implemented two methods so far for Image Generation on the Client:

• Girder File Image URLS
  • Using girder files as png images and having them loaded directly from the server
  • This requires using similar caching to the image-sequence tool where it will grab +5s and -1s of frames around the current frame and cache them
  • These image files are then directly used for an image Quad per trackId in the renderer
• COCO RLE Compressed Masks
  • Masks are compressed into a json file using pycocotolls rle encoding
  • Additional endpoint will return data to the client
  • Client then needs to convert the RLE encoding into images that can be used
  • Currently this is done in the main thread and takes significantly long.

[BryonLewis]

The performance of the RLE version isn't fast enough when creating HTMLImageElements. I will focus on the girder image item reference and am currently preloading them.

Playback seems to be okay when doing it using this format.

Export and Import for now will be done using the Masks folder with the trackId -> FrameId.png file structure for the references. We need to make clear in documents the structure for users.

Mask Folder in the root of the DIVE Dataset with a metadata field "mask: true"
folders are the TrackId Numbers and have a metadata field of "mask_track: true"
Finally under each folder the images are in GirderItems and have a fileName of the FrameId Number.
They images also have the following metadata set:
mask_frame_parent_track: {trackId number}
mask_frame_value: {A reference to the Frame number (should be same as the item Name)
mask_track_frame: true

This is the structure.

Exporting masks will export the entire masks folder as a Zip file. This should include everything that represents the masks.

Importing will take in a Zip file with the masks (folder) -> trackIds (folders) -> frames (images) and will process and import them into the correct location.

• Process will use the current import within the DIVE Annotation Viewer
• It will upload a zip file to the DatasetFolder.
• I don't think postprocess will be run but it will be a ZipProcessor function that will process Zip files found in the root of a DIVE Dataset Folder
• This ZipProcessor will be a celery task that will check the Zip file for a 'mask' folder and then will import the masks based on the TrackId (folder) -> FrameId (images) structure by creating the MaskFolder in the DIVE Dataset and uploading the files.