Skip to content

IMPLEMENTATION_PLAN.md

Latest commit

 

History

History
186 lines (125 loc) · 6.24 KB

IMPLEMENTATION_PLAN.md

File metadata and controls

186 lines (125 loc) · 6.24 KB

Implementation Plan

This plan tracks the staged implementation for asdf-firefly, focused on Roman Space Telescope ASDF products. Update it whenever scope or status changes.

Status Legend

  • Not Started: no implementation yet.
  • Planned: design is documented and ready to build.
  • In Progress: code is being implemented.
  • Blocked: waiting on a decision, dependency, or external behavior.
  • Done: implemented, tested, and documented.

Current Status

The project currently has documentation, an initial SPEC.md, and a completed package scaffold. The documented scope is Roman WFI ASDF inspection, efficient local and remote source access, and Firefly handoff.

Project spec:

  • Status: Planned
  • SPEC.md captures goals, constraints, CLI interfaces, invariants, tasks, and bug table in caveman format.
  • FORMAT.md is not present; spec formatting currently follows the installed caveman skill grammar.

Phase 1: Project Scaffold

Status: Done

Deliverables:

  • pyproject.toml with package metadata, console script, dependencies, and optional extras.
  • src/asdf_firefly/ package layout.
  • tests/ layout with minimal generated fixtures.
  • Basic CI-ready commands for pytest, ruff check, and ruff format --check.

Acceptance criteria:

  • asdf-firefly --help runs.
  • pytest discovers the test suite.
  • Documentation commands reflect the actual package name, CLI entry point, and Roman product examples.

Phase 2: ASDF Inspection

Status: Done

Deliverables:

  • Shared ASDF source resolver for local paths, HTTP(S) URLs, and S3 URIs.
  • info command.
  • tree command with --depth, --types, --arrays, and --find.
  • get command with slash-separated path resolution.
  • validate command using ASDF validation APIs.

Relevant docs:

  • README.md: command examples.
  • DEVELOPER.md: ASDF navigation and core rules.

Acceptance criteria:

  • Commands open ASDF files through context managers.
  • S3 sources are opened through fsspec/s3fs file-like objects when the remote extra is installed.
  • Missing remote dependencies produce a clear install hint.
  • Large arrays are summarized, not printed.
  • Metadata-only remote commands avoid whole-object downloads where a seekable stream is available.
  • Missing paths return the documented path-not-found exit code.

Phase 3: Object Discovery

Status: Planned

Deliverables:

  • Tree walker for scalar, mapping, array, table, and unsupported object classification.
  • tables command.
  • images command.
  • Conservative WCS metadata detection.

Acceptance criteria:

  • Astropy tables and structured NumPy arrays are listed as table-like.
  • 2D arrays are listed as image-like.
  • Unsupported candidates include a reason.

Phase 4: Table Commands

Status: Planned

Deliverables:

  • table show.
  • table export.
  • table open.
  • Output formats: ECSV, CSV, FITS table, VOTable, and Parquet where optional dependencies are installed.

Relevant docs:

  • Astropy Table unified I/O.
  • Firefly table documentation.

Acceptance criteria:

  • Terminal previews are row-limited by default.
  • ECSV export preserves table metadata where Astropy supports it.
  • Unsupported format and optional dependency errors are clear.
  • Firefly handoff uses an exported temporary table file.

Phase 5: Image Commands

Status: Planned

Deliverables:

  • image export.
  • image open.
  • FITS export for 2D arrays.
  • Initial --plane support for 3D cubes.
  • Best-effort FITS WCS/header preservation.

Relevant docs:

  • Astropy FITS I/O.
  • Firefly FITS visualization documentation.

Acceptance criteria:

  • Exported FITS files can be opened by Astropy.
  • Firefly handoff uses a readable FITS file.
  • Warnings are emitted for metadata that cannot be represented in FITS.

Phase 6: Firefly Integration

Status: Planned

Deliverables:

  • Config file loading from ~/.config/asdf-firefly/config.toml.
  • ASDF_FIREFLY_URL and ASDF_FIREFLY_KEEP_TEMP environment variable support.
  • Local and remote Firefly URL handling.
  • Temporary file retention with --keep-temp.

Acceptance criteria:

  • A configured Firefly URL is used consistently by table open and image open.
  • Temporary files are cleaned up by default.
  • --keep-temp prints the retained file path.

Phase 7: Packaging And Release

Status: Planned

Deliverables:

  • Version command.
  • Build metadata.
  • Optional remote extra for S3 URI dependencies.
  • Release notes template.
  • Contributor workflow documented.

Acceptance criteria:

  • Package builds locally.
  • Installation exposes the console script.
  • README installation instructions match the package metadata.

Key Risks

  • Roman ASDF files can contain mission-specific tagged objects that require optional extension packages.
  • WCS conversion from ASDF/GWCS to FITS headers may be partial or impossible for some data products.
  • Firefly handoff behavior depends on deployment configuration and may require server-side file access or URL-accessible files.
  • Large arrays and tables can cause accidental memory pressure if previews or exports are not bounded.
  • Remote ASDF performance depends on ASDF block layout, seek behavior, cache configuration, object-store latency, and whether the command needs to materialize selected data.

Open Decisions

  • asdf-astropy, gwcs, and roman-datamodels are optional Roman extras for now; revisit if core commands require typed Roman objects instead of raw fallback metadata.
  • Decide the exact Firefly open mechanism for local files: Python wrapper, URL launch, upload endpoint, or deployment-specific plugin.
  • Decide whether fits-header should inspect only embedded FITS metadata or also synthesized FITS headers from ASDF metadata.
  • Decide which remote authentication profiles, beyond the default S3/AWS provider chain, should be documented.

Verification Sources

  • Context7 lookup for /asdf-format/asdf, /astropy/astropy, and /textualize/rich.
  • Web search and upstream docs for ASDF overview and standard.
  • ASDF Python asdf.open(...) API docs confirming file-like object and HTTP URI support.
  • fsspec and s3fs docs confirming protocol-prefixed file opening and seekable S3 file-like objects.
  • Upstream Roman ASDF data model and extension package documentation.
  • GitHub and upstream docs for Caltech-IPAC/firefly, including FITS visualization, table display, and Python wrapper references.