[Collector] Update DB to support eCAPRIS funding sync

[mddilley]

Associated issues

Closes cityofaustin/atd-data-tech#24593

This PR updates the moped_proj_funding schema to enable syncing eCAPRIS funding information on projects linked to a eCAPRIS subproject. It also adds columns to the project funding table that will help us differentiate records created in Moped and records created from eCAPRIS data which we can't currently do. There is also a new column that will be important for tracking records that were created prior to the new funding enhancements.

Testing

URL to test:

Local please 🙏

Steps to test:
For this test, we will be updating the following project that is linked to the following eCAPRIS id:

• https://localhost:3000/moped/projects/697
• eCAPRIS subproject ID: 13305.009

1. Start your local stack and then start the Hasura console to test some queries and mutations
2. Use the mutations and query variables in the next code block to insert three records that simulate a manual entry, an import from eCAPRIS, and a sync from eCAPRIS. You should see three project funding record ids returned.

-- mutation
mutation AddProjectFunding($objects: [moped_proj_funding_insert_input!]!) {
  insert_moped_proj_funding(objects: $objects) {
    returning {
      proj_funding_id
    }
  }
}

-- query variables to simulate syncing from eCAPRIS, importing from eCAPRIS, and manually creating funding records
{
  "objects": [
    {
      "project_id": 697,
      "fdu": "4730 2507 M121",
      "ecapris_funding_id": 130286,
      "unit_long_name": "Ter Project #4 - Cameron Rd - Cmta",
      "funding_amount": 777000,
      "funding_status_id": 2,
      "funding_description": "Added through syncing from eCAPRIS"
    },
    {
      "project_id": 697,
      "fdu": "820B 2507 4A09",
      "ecapris_funding_id": 127642,
      "unit_long_name": "2022 SS4A Systemic Cameron/Dessau 2020 Bond Match",
      "funding_amount": 250000,
      "funding_status_id": 2,
      "funding_description": "Added through importing from eCAPRIS"
    },
    {
      "project_id": 697,
      "fdu": null,
      "ecapris_funding_id": null,
      "unit_long_name": null,
      "funding_amount": 200000,
      "funding_status_id": 2,
      "funding_description": "Manually entered because FDU doesn't exist"
    }
  ]
}

3. Now, let's query for all the funding records for project 697 with the new results first to show the 3 freshly-inserted records. You will notice that:
   • the first three records have is_legacy_funding_record set to false while the rest that were created using our current (legacy) UI have it set to true
   • is_editable should also be false for any record that has ecapris_funding_id since synced and imported records will be updated by the ETL. The MVP will not enable overrides.
   • every record except the one with the description of "Manually entered because FDU doesn't exist" has both new fdu and unit_long_name columns populated

-- select all funding rows for project 697 from moped_proj_funding
query GetProjectFunding {
  moped_proj_funding(where: {project_id: {_eq: 697}}, order_by: {created_at: desc}) {
    fdu
    ecapris_funding_id
    project_id
    unit_long_name
    funding_amount
    funding_status_id
    funding_description
    is_editable
    is_legacy_funding_record
  }
}

5. Next, let's query the project_funding_view database view which is available in the Power BI data flow and selects from moped_proj_funding and joins in lookup values.

-- select all funding rows for project 697 from project_funding_view
query GetPowerBIProjectFunding {
  project_funding_view(where: {project_id: {_eq: 697}}, order_by: {created_at: desc}) {
    fund_dept_unit
    funding_amount
    funding_description
    proj_funding_id
    project_id
  }
}

6. Last, let's enable and then disable fund syncing on project 697. You can toggle the boolean in the mutation to see the column switching. The ETL will look at this column to determine which projects to sync funding from eCAPRIS (if there is also an eCAPRIS subproject id on the project).

-- turn sync on or off
mutation ToggleFundSyncing {
  update_moped_project_by_pk(pk_columns: {project_id: 697}, _set: {should_sync_ecapris_funding: true}) {
    project_id
    should_sync_ecapris_funding
    ecapris_subproject_id
  }
}

---

Ship list

• Code reviewed
• Product manager approved
• Product manager checked DB view dependencies
• Product manager added to QA test script if applicable

[netlify]

✅ Deploy Preview for atd-moped-main ready!

Name	Link
🔨 Latest commit	7ff21e9
🔍 Latest deploy log	https://app.netlify.com/projects/atd-moped-main/deploys/691f470aa5cbb60008a30328
😎 Deploy Preview	https://deploy-preview-1694--atd-moped-main.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[mddilley]

This is cleanup from the eCAPRIS statuses work. We don't select directly from this table. eCAPRIS statuses are selected through the combined_project_notes_view so we don't use these permissions.

[mddilley]

These two columns intend to:

• is_legacy_funding_record will be used to track before/after state of funding records since we are changing the upstream data source for eCAPRIS information when this new feature launches
• is_editable will be used in frontend code to determine when to show the edit UI or not (eCAPRIS synced or imported records should use eCAPRIS numbers as the source of truth and any need to modify the funding in a project should be covered by manually entering data in the funding DataGrid table)

But, I am looking for feedback on them. Using a generated column to provide the UI code with information about whether or not to allow editing made sense to me, but I am open to other ideas if this doesn't seem like it will be clear in the future.

[frankhereford]

I'm following the logic in your design here, and I think your choice with the generated column is better than the alternative of putting the logic controlling editing in the front-end application. This feels self-documenting and explicit, and has zero performance cost, so 👍 in my book.

[roseeichelmann]

yeah i agree w frank, i think the generated column is good!

[johnclary]

I agree — this looks very good. One alternative would be to store the ecapris funding records in a separate but similar table, following the pattern of the ecapris project statuses. That might be a cleaner way to do it if the ecapris records are truly intended to be read-only, because you can clamp the edit permissions down on them. Again, what you have looks very good and I don't know the full picture of where this feature is headed.

[mddilley]

thanks, y'all! I'm taking another look at the schema today. On Friday, I made progress on an ETL that populates synced records in the moped_proj_funding, and I am now concerned that having the ETL and users operating on the same table could create issues down the line - particularly the is_deleted status of rows becoming de-synced.

Today, I am going to test a separate table for the synced records so we handle the three sources of funding info like:

1. Manually created funding records stored in moped_proj_funding (just like today)
2. Imported records from eCAPRIS data source stored in moped_proj_funding (just like today but we also have the unique fao_id to tie back to eCAPRIS if ever needed) This also covers the need to override eCAPRIS amounts if an FDU is split between multiple Moped projects which was a new need that came up in the last MUG meeting.
3. Synced records stored in separate table (like eCAPRIS subproject statuses). This is the only table that the ETL would operate on and would provide un-editable eCAPRIS data.

[mddilley]

We currently have a generated column called fund_dept_unit that pulls the F and DU parts out of fund and dept_unit jsonb columns, but this new fdu column will store the FDU string that we can tie back to eCAPRIS by the unique ecapris_funding_id.

[mddilley]

This last part is important to carry over existing data captured from the Socrata jega-nqf6 and bgrt-2m2z datasets into the new schema that can be tied back to eCAPRIS by an id.

Last in this migration, we need to update the database view that powers some Power BI dashboards to use our new FDU column instead of the generated one that stitched data from the fund and dept_unit columns together.