Skip to content

WSClean: support full image cubes #624

@mpluess

Description

@mpluess

We don't currently support creating full image cubes in either WscleanDirtyImager (combine_across_frequencies=False) or WscleanImageCleaner, even though this is a quite commonly used feature (already came up as a requirement from Devin / ETH and in SRCNet). We should therefore implement this. The relevant WSClean parameter is this one:
-channels-out <count> Splits the bandwidth and makes count nr. of images. Default: 1.

While in principle, users can use this parameter through karabo.imaging.imager_wsclean.create_image_custom_command, the function currently fails when using the parameter because this changes the naming of the output files, see https://wsclean.readthedocs.io/en/latest/making_image_cubes.html.

Another blocker that needs to be addressed to solve this issue is that currently, imagers return Image objects which load the full image data into memory. This quickly leads to OOM problems with full image cubes. For example, the full image cube of the PI24 SRCNet Karabo run with 2717 channels and an image size of 20000x20000 would be around 4 TB.

  • Find a smart solution to deal with image cubes that don't fit into memory. Maybe make type of output configurable with a reasonable default: return full cube, return MFS image (weighted sum over channels), return specific channel, return nothing. Or find a way to stream image into memory on demand when trying to access it downstream.
  • Properly support -channels-out in create_image_custom_command. Current problems: output_filenames parameter, return type / value.
  • Support -channels-out in WscleanDirtyImager, possibly change combine_across_frequencies config parameter, change output_fits_path parameter. Specifics depend on task 1.
  • Support -channels-out in WscleanImageCleaner as well: change dirty_fits_path and output_fits_path parameters, support generating full image cube for both scenarios: when creating a cleaned image from visibilities and from an existing dirty image cube.
  • output_fits_path @ WscleanDirtyImager + WscleanImageCleaner: if set: move file(s), don't copy
  • Apply changes in imaging interface to other imager implementations, i.e. OSKAR + RASCIL

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions