[eCAPRIS status collector] Create view to combine Moped internal notes, Moped status updates, and eCapris status updates by mddilley · Pull Request #1607 · cityofaustin/atd-moped

mddilley · 2025-06-03T16:35:52Z

Associated issues

Closes cityofaustin/atd-data-tech#22700, cityofaustin/atd-data-tech#22217

This PR creates a new view to collect all types of notes that can be displayed in the Notes tab into one single timeline. The view handles the mapping of eCapris columns that we want to show in the UI to conform to the existing schema of Moped notes and status updates. It also joins in note type and note phase data since these details are needed for the current note features.

It also updates the project_list_view to consider whether a project has an eCapris subproject ID and if syncing is enabled. This will prevent eCapris statuses from showing on projects where the PMs have opted out of eCapris statuses (toggle is off in Notes tab) without any additional business logic in the React code or AGOL ETL.

These are the places that we are concerned about seeing status updates:

Project notes tab: query from new view and filter by 3 note types
Project summary view status update: query from new view filter to 2 status update types
Dashboard (My Projects and Following tabs): query from new view filter to 2 status update types
Project list: query from updated project_list_view which handles whether to include eCapris statuses or not
AGOL dataset: selects from updated project_list_view which handles whether to include eCapris statuses or not

This feels like a good way to wrangle the concept of notes in a single place - I'm open to other ideas too!

Testing

URL to test:

Local

Steps to test:

Start your local stack and follow this SQL:

-- Query notes from project 331 that has eCapris subproject ID 13240.027 set on it
-- and has syncing enabled
SELECT project_id, ecapris_subproject_id, project_status_update FROM project_list_view WHERE project_id = 331;

-- Now, soft-delete the latest Moped note so that we can see that the view will use
-- the latest eCapris status
UPDATE moped_proj_notes SET is_deleted = TRUE WHERE project_note_id = 11307;

-- Repeat the query to see that the view is now showing the latest status update
-- that is sourced from eCapris
SELECT project_id, ecapris_subproject_id, project_status_update FROM project_list_view WHERE project_id = 331;

-- Check the AGOL view to see the latest project status rolled up into the component
-- rows
SELECT project_id, project_component_id, project_status_update FROM component_arcgis_online_view WHERE project_id = 331;

-- Now, disable syncing so that our view will not used eCapris statuses when
-- finding the latest status update
UPDATE moped_project SET should_sync_ecapris_statuses = FALSE WHERE project_id = 331;

-- Repeat the query to see that the view will show the next newest Moped
-- status update with eCapris syncing now disabled
SELECT project_id, ecapris_subproject_id, project_status_update FROM project_list_view WHERE project_id = 331;
SELECT project_id, project_component_id, project_status_update FROM component_arcgis_online_view WHERE project_id = 331;

Now, open your Hasura console, and test some queries with data needed by the frontend:

# Get all notes associated with a project or its eCapris subproject
query GetCombinedProjectNotes {
  combined_project_notes_view(where: {_or: [{ecapris_subproject_id: {_eq: "13240.027"}}, {project_id: {_eq: 331}}]}, order_by: {created_at: desc}) {
    author
    created_at
    ecapris_subproject_id
    id
    is_editable
    note_type_name
    phase_id
    project_id
    project_note
    created_by_user_id
    original_id
    note_type_slug
    phase_key
    phase_name
  }
}

# Get only Moped-type notes
query GetCombinedProjectNotes {
  combined_project_notes_view(where: {project_id: {_eq: 331}}, order_by: {created_at: desc}) {
    author
    created_at
    ecapris_subproject_id
    id
    is_editable
    note_type_name
    phase_id
    project_id
    project_note
    created_by_user_id
    original_id
    note_type_slug
    phase_key
    phase_name
  }
}

# Get the latest status from the project_list_view
query GetProjectListLatestStatus{
  project_list_view(where: {project_id: {_eq: 331}}){
    project_id
    project_status_update
  }
}

Ship list

Code reviewed
Product manager approved
Product manager checked DB view dependencies
Product manager added to QA test script if applicable
- Mike will write new tests to cover:
  - Adding & removing eCAPRIS subproject IDs and the expected changes to sync status
  - Toggling sync on and off in the Notes tab and seeing the expected changes in notes displayed
  - Filtering between note types
  - Disabled filtering and sync switch if no eCAPRIS subproject id
  - Seeing the expected events in the activity tab

…table joins

…ecapris author emails

netlify · 2025-06-03T16:35:57Z

✅ Deploy Preview for atd-moped-main ready!

Name	Link
🔨 Latest commit	`d807f4d`
🔍 Latest deploy log	https://app.netlify.com/projects/atd-moped-main/deploys/686689d6a7856a0008a6e21a
😎 Deploy Preview	https://deploy-preview-1607--atd-moped-main.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

mddilley · 2025-06-03T20:28:34Z

+-- Update moped_note_types to have source column
+ALTER TABLE moped_note_types
+ADD COLUMN source text NOT NULL DEFAULT 'moped';
+
+-- Insert eCapris status note type that will be used in notes view
+INSERT INTO moped_note_types (name, slug, source)
+VALUES
+('eCapris Status Update', 'ecapris_status_update', 'ecapris');
+
+COMMENT ON COLUMN moped_note_types.source IS 'Source of the note type, e.g., Moped or eCapris applications';


The reasoning behind adding the eCapris note type to our lookup table is that our React code renders radio buttons and filter buttons based off values in this table. The new source column helps us control when these elements render - for example, radios to choose new note type could be limited to Moped-sourced types only.

I realize that changes the concept of this table from "Moped note types" to "types of notes that can be displayed in Moped." It is unlikely that we will see other types, and I'm hoping the source column is enough to help us remember how this works in the future.

mddilley · 2025-06-03T20:31:03Z

+    moped_phases.phase_key AS phase_key,
+    moped_proj_notes.is_deleted,
+    moped_proj_notes.phase_id,
+    TRUE AS is_editable,


The intent of is_editable in this view is to help us know when to edit components like the pencil and trash icons since eCapris statuses will not be editable in Moped. This would work along with the existing business logic in the React code that note authors can only edit their own notes and statuses.

mddilley · 2025-06-03T20:33:47Z

+    COALESCE(
+        (moped_users.first_name || ' ' || moped_users.last_name),
+        CASE
+            WHEN ecapris_subproject_statuses.reviewed_by_name LIKE '%,%' 
+                THEN TRIM(SPLIT_PART(ecapris_subproject_statuses.reviewed_by_name, ',', 2)) || ' ' || TRIM(SPLIT_PART(ecapris_subproject_statuses.reviewed_by_name, ',', 1))
+            ELSE LOWER(ecapris_subproject_statuses.reviewed_by_email)
+        END
+    ) AS author,


I was originally going to use reviewed_by_email as a fallback to eCapris statuses that didn't match an existing Moped user, but this column is not always populated in the eCapris database.

I could not find any statuses without reviewed_by_name or statuses where the format was not Last, First but I added a check just in case we see something different in the future.

mddilley · 2025-06-03T20:36:43Z

+    ('M' || moped_proj_notes.project_note_id) AS id,
+    moped_proj_notes.project_note_id AS original_id,


The prefixed ids serve as unique keys for rendering lists of notes. The original_id is needed to know which editable notes and statuses to mutate.

mddilley · 2025-06-03T20:46:40Z

+    NULL AS phase_name,
+    NULL AS phase_key,


eCapris statuses do have "phases" with their own lookup values. No stakeholders have asked for us to attempt to map them to Moped phases, and it doesn't seem like a great idea to me to put effort into this.

Either way, our React code doesn't render a status badge when these are null which is what we want to happen.

Sounds good!

johnclary

Mike, I'm blocked on replicating the DB at the moment, so I'm going to submit these handful of comments in case they're helpful. I didn't see your comments until I had reviewed. This looks great! Will come back to test it soon.

johnclary · 2025-06-04T18:26:57Z

+-- Most recent migration: moped-database/migrations/default/1748534889272_create_notes_view/up.sql
+
+CREATE OR REPLACE VIEW combined_project_notes AS SELECT
+    'M'::text || moped_proj_notes.project_note_id AS id,


is the intention to render this ID in the UI? or merely have unique ID for every row in the view? my first reaction is is to suggest you use moped_ and ecapris_ as prefixes to make it self-documenting.

yes, the intent is to have a unique id that we can depend on for React keys, and I think that adding the full names is a great call. I'll update 🙏

johnclary · 2025-06-04T18:31:08Z

+            WHEN ecapris_subproject_statuses.reviewed_by_name ~~ '%,%'::text THEN (TRIM(BOTH FROM SPLIT_PART(ecapris_subproject_statuses.reviewed_by_name, ','::text, 2)) || ' '::text) || TRIM(BOTH FROM SPLIT_PART(ecapris_subproject_statuses.reviewed_by_name, ','::text, 1))
+            ELSE LOWER(ecapris_subproject_statuses.reviewed_by_email)
+        END
+    ) AS author,


If it's possible to include comments in these files it'd be nice to have some sample strings of what you're parsing here.

I can't remember if comments persist inside of these views after the bot handles them or not so I'll give it a go, and we'll find out. 🚀

Nope, the comments didn't make it through. Maybe an improvement that we could look into down the road with a future VZ SQL bot. 🤖

johnclary · 2025-06-04T18:33:17Z

+    moped_note_types.slug AS note_type_slug,
+    null::text AS phase_name,
+    null::text AS phase_key,
+    false AS is_deleted,


I can't recall if you mentioned whether or not ecapris statuses may ever be deleted upstream?

Not sure if I mentioned or not but they are not deleted upstream. We should be covered handling edits only.

johnclary · 2025-06-04T18:33:45Z

+    null::text AS phase_key,
+    false AS is_deleted,
+    null::integer AS phase_id,
+    false AS is_editable,


Nice! this view is looking great—I can see how this is going to feed nicely into the UI code 🚀

johnclary · 2025-06-05T13:35:48Z

+    NULL AS phase_name,
+    NULL AS phase_key,


Sounds good!

…_filter Disable eCAPRIS Notes Filter

…pris_false Update should_sync_ecapris_statuses to false for projects without eCAPRIS IDs

… text when switched on

Handle eCAPRIS sync activity in log when there is no eCAPRIS subproject id

…ty_log Show eCapris status syncing updates in project activity log

…atuses Show eCapris statuses in Notes tab

mddilley and others added 11 commits May 29, 2025 11:08

Add initial view

a4cd402

Add todo

35af230

Update down migration and update view with table references and user …

e993458

…table joins

Add to project note types

9779ffa

Add comment and todo about AGOL view

157b803

Test updated proj_status_update join

a3ec554

Replace proj_status_update lateral join

d451c35

Only use ecapris statuses if project sync is enabled; handle missing …

b95b67c

…ecapris author emails

Update order and add comment on source column

99b4212

Handle malformed author names

42dc367

🤖 Export view for 22270_notes_view

93d9e58

mddilley added the WIP Work in progress label Jun 3, 2025

mddilley and others added 4 commits June 3, 2025 11:58

Update source to be not null; add original id

8b162bf

🤖 Export view for 22270_notes_view

8535700

Update view with phase info and note type slug

e1a9e18

🤖 Export view for 22270_notes_view

b337f94

mddilley commented Jun 3, 2025

View reviewed changes

mddilley removed the WIP Work in progress label Jun 3, 2025

mddilley requested review from chiaberry, frankhereford, johnclary, mateoclarke, roseeichelmann and tillyw June 3, 2025 20:51

johnclary reviewed Jun 5, 2025

View reviewed changes

mddilley and others added 28 commits June 23, 2025 16:32

Fix delete by using correct id reference

259ed1d

Add sync status field to project part of dashboard query

48dab0e

Add style back to restore dashboard status update pointer cursor

17be147

Update ecapris note type name

8c178ec

🤖 Export view for 21794_show_ecapris_statuses

dd3ea42

Update var that I somehow named worse

2a780f8

wip updates to activity log presentation

edc976b

simplify activity log to expect an array of changes

6a6db25

clean up logic for if statements to assume fields is array

6429b06

slight name change

0826501

text update

90db565

pass disable ecapris filter prop

78df2da

disable when there is no syncing, render tooltip

5c0567e

Update project activity formatter to ignore sync change w/ no id

4616fe7

update var name

6fa3e6b

add to docstring

f0004df

Add comment explaining tooltip span

30c8a34

Merge pull request #1620 from cityofaustin/rose/22702_disable_ecapris…

5cf0b7b

…_filter Disable eCAPRIS Notes Filter

Update comment to mention edge case

7dcb106

Update project summary query to use plv object relationship

c2f0794

disable triggers

233d0a5

Merge pull request #1613 from cityofaustin/rose/23041_should_sync_eca…

b797d95

…pris_false Update should_sync_ecapris_statuses to false for projects without eCAPRIS IDs

Memo userSessionData; update tooltip to show when disabled and helper…

013bf1b

… text when switched on

Remove TODOs

265d315

Merge pull request #1621 from cityofaustin/mike/23350_activity_no_id

6a3c8ca

Handle eCAPRIS sync activity in log when there is no eCAPRIS subproject id

Merge pull request #1618 from cityofaustin/mateo/22431_ecapric_activi…

6b723e7

…ty_log Show eCapris status syncing updates in project activity log

Merge pull request #1615 from cityofaustin/mike/21794_show_ecapris_st…

da7137d

…atuses Show eCapris statuses in Notes tab

Merge branch 'main' into mike/22270_notes_view

d807f4d

mddilley merged commit e9e2de1 into main Jul 3, 2025
5 checks passed

mddilley deleted the mike/22270_notes_view branch July 3, 2025 14:01

		('M' \|\| moped_proj_notes.project_note_id) AS id,
		moped_proj_notes.project_note_id AS original_id,

Conversation

mddilley commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Associated issues

Testing

Ship list

Uh oh!

netlify Bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for atd-moped-main ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnclary left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mddilley Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mddilley commented Jun 3, 2025 •

edited

Loading

netlify Bot commented Jun 3, 2025 •

edited

Loading

mddilley Jun 3, 2025 •

edited

Loading

mddilley Jun 3, 2025 •

edited

Loading

mddilley Jun 3, 2025 •

edited

Loading

mddilley Jun 10, 2025 •

edited

Loading

mddilley Jun 10, 2025 •

edited

Loading