Special groups aggregate metrics

[dirkdk]

Is your feature request related to a problem? Please describe.

Cities often request data on how many lower income users we reach with our vehicle service, and how many trips such users take. MDS does not currently support user segmentation. We would oppose attaching any user data to the Trips endpoint as it would involve information close to Personally Identifiable Information (PII) and would make it fairly easy to identify individuals by trip route and user segment. Instead, we propose the solution of providing aggregate trip data by user segment.

Describe the solution you'd like

We propose that Providers provide aggregated data on trips by special user segments, using the new Metrics API. By using aggregation, it should be impossible to trace back this data to individuals. This does mean we need to set meaningful minimals for certain metrics so that the aggregated data has k-anonymity.

The new Metrics API specifies parameters for name, since, interval, dimensions that we will assume these metrics support.

Proposed metrics for special groups:

Metric	Description
special_groups	array of names of special groups as served by Provider. This list will be fairly static. Example values are low_income or students
active_users[special_group]	total number of users for given group with at least 1 trip in interval
trips[special_group]	count of trips by users of given special group during interval
trips.average_duration[special_group]	average duration in seconds of trips by users of given special group during interval
trips.median_duration[special_group]	median duration in seconds of trips by users of given special group during interval
trips.std_duration[special_group]	standard deviation of duration  in seconds of trips by users of given special group during interval
trips.average_distance[special_group]	average distance in meters of trips by users of given special group during interval
trips.median_distance[special_group]	median distance in meters of trips by users of given special group during interval
trips.std_distance[special_group]	standard deviation of distance in meters of trips by users of given special group during interval

Overall aggregate statistics

For overall usage we can do what is listed below. Please note that this data should be derivable from MDS trip data, minus active users.

Metric	Description
active_users	total number of all users with at least 1 trip in interval
trips	count of trips by all users during interval
trips.average_duration	average duration in seconds of trips by all users  group during interval
trips.median_duration	median duration in seconds of trips by all users  during interval
trips.std_duration	standard deviation of duration in seconds of trips by all users  during interval
trips.average_distance	average distance in meters of trips by all users  during interval
trips.median_distance	median distance in meters of trips by all users  during interval
trips.std_distance	standard deviation of distance in meters of trips by all users  during interval

Is this a breaking change

A breaking change would require consumers or implementors of the API to modify their code for it to continue to function (ex: renaming of a required field or the change in data type of an existing field). A non-breaking change would allow existing code to continue to function (ex: addition of an optional field or the creation of a new optional endpoint).

• No, not breaking

Impacted Spec

For which spec is this feature being requested?

• metrics but only served by Providers. Only Providers will have the raw data

Describe alternatives you've considered

Alternatives would be to add user segments to individual trips in the Trips API. We oppose this method as it would make user identification extremely easy. We currently send this data to cities via manually compiled Excel sheets, and it would be better to have an official API.

[schnuerle]

Some notes from WG meeting on Aug 27:

• This would allow reporting on data beyond what MDS provides, such as interest on usage of low-income/student plans. This would be data that only the Provider would be able to calculate but avoids privacy issues by not sharing the underlying data to calculate these metrics through MDS.
• These special groups could be defined in the spec, with the option of other special groups definable between a city and a provider.
• Timeframe and geography would need to be further defined. These should not be so granular that individuals could be identified. Could use Geography API to define areas and start, end, traversal. Time could be day, week, month.
• k-anonymity values would be key (eg, 5, 10), where a minimum value would have to be returned or else no data is shown.
• There is likely a common list of these "special group" metrics across cities, but there would need to be room for flexibility to support additional metrics. Having a standard would provide that flexibility.

[johnclary]

@schnuerle can we have a Privacy label on this one, please?

R.e:

We would oppose attaching any user data to the Trips endpoint as it would involve information close to Personally Identifiable Information (PII) and would make it fairly easy to identify individuals by trip route and user segment. Instead, we propose the solution of providing aggregate trip data by user segment.

Timeframe and geography would need to be further defined. These should not be so granular that individuals could be identified. Could use Geography API to define areas and start, end, traversal. Time could be day, week, month.

Yes—there are significant privacy concerns with adding any additional PII to MDS. In my opinion, this is something that feels beyond the scope of what MDS should ever support. Can we not leave it up to agencies to use their own data—presumably collected responsibly (e.g. through surveys or the US Census)—to conduct this kind of analysis?

If providers are already holding this kind of information: why? can you stop?

[joshuaandrewjohnson1]

@johnclary the original intention with this idea was to ensure that no personally identifiable information of specific user groups would flow through MDS, nor would their individual trip data, but rather aggregated information (such as overall number of low-income program users or overall number of trips), which would be calculated by the provider before sharing with a city.

This is data that almost every city already asks for, and is mostly facilitated via custom reports, so there would be benefit to both providers and cities in standardizing.

[schnuerle]

Maybe a clarification from the call that could be added to the main description here: the 'low income' group from providers means any riders using an equity plan, eg Bird Access, Spin Access, Bolt Forward, etc. So this is known by the providers per rider as part of operations and billing for discounts.

Note the k-anonymity part which says if a query returns too few results based on filters/time/geography, no data is returned for that slice.

[johnclary]

If I'm reading these materials correctly, it is being proposed that this API make it possible to report on
the average duration and distance of trips taken in an arbitrary geography by riders who are unbanked.

What's missing is a specific explanation of why an agency would want to collect this data and what meaningful action they would expect to take as a result of it.

Assuming we all agree that the "special groups" being discussed here are people who have have a disproportionately high risk of harm if their PII were to be leaked, who generally have fewer mobility choices, and who would almost definitely prefer that service providers did not hold this kind of data about them—this warrants an extremely well-defined use-case.

Regardless of whether or not providers are currently collecting this data, do we as agencies really want to reinforce that practice based on the case that there might be some useful information for an as-yet unproven technocratic purpose?

[johnclary]

What really bothers me about this proposal is that the development of MDS continues to default to collecting as much data as possible on the premise that it will be theoretically useful at some point. There remain very few stories to tell of cities doing right by there publics as a result of this casting of an extremely wide collection net.

[marie-x]

What really bothers me about this proposal is that the development of MDS continues to default to collecting as much data as possible on the premise that it will be theoretically useful at some point. There remain very few stories to tell of cities doing right by there publics as a result of this casting of an extremely wide collection net.

That's now how I read Dirk's proposal. It is from an employee of a mobility provider. Spin's assertion is that many cities have specific purposes for this data, they are already making these requests of mobility providers, and that this would be an (optional!) way to standardize the requests that are already being made.

Also I am not sure I agree with the characterization that MDS "defaults to collecting as much data as possible". The Metrics API is clearly less data, explicitly de-identified. The OMF privacy committee is also publishing standards for privacy protection including data deletion principles.

I agree that some city representatives should join Dirk/Spin to describe the use cases and public-policy aims for this information.

[schnuerle]

For the sake of documenting what cities are currently asking for in terms of monthly reporting, I went through every city permit/policy application I could find from our Cities Using MDS list and summarized very broadly the relevant, documented asks.

These are mostly in the terms of monthly reports, and can include MDS derived data (most cities are asking for aggregated MDS data in addition to the API feeds).

I'm sure there are more examples, but I could not find them online. If anyone has more cities please add them in the comments. The mobility service providers like Spin and Bird may have a more comprehensive list of the things cities are requiring, and could provide an aggregated (not broken down by city) list.

List Notes

I put special user groups in bold, and equity/zone area info in italics.

Note some cities are asking for aggregated demographics.

Almost all cities also ask for complaints, safety, collision, injury, and vandalism numbers, and these seem like something missing in the proposals so far.

Outside of the scope of this are system alerts, pricing plans, and hours of operation, which gets back to a brief conversation we had in the last Provider WG call around a kind of 'Policy for Providers' API idea.

City List

Austin

1. Complaints: number, nature, time to remedy
2. Collision: number, severity, location, time

Denver

1. App: downloads, active users, repeat users
2. Trips: total, day of week, time, distance
3. Origin/Destination
4. Opportunity areas: in and out
5. Parking: transit/bus stop compliance
6. Theft/Vandalism
7. Maintenance Reports
8. Complaints
9. Special groups: low income, students
10. Collision
11. Payment Method

Detroit

1. Utilization rates
2. Special groups: Membership volumes
3. Trips: volumes, day of week, time of day, origins, destinations, distance
4. Parking: compliance rates
5. Theft/vandalism
6. Maintenance reports
7. Complaints
8. Collision

Kelowna, Canada

1. Trip
2. Parking
3. Incidents
4. Maintenance
5. Collision
6. Survey: 3-20 questions about riders

Long Beach

1. Riders: total unique number
2. Parking zone: counts, percent, incentives
3. Trip: distance
4. Devices: serviced, lost, stolen, missing, impounded, make/model/count
5. Revenue: membership, penalty, per minute, per ride
6. Injuries
7. Legal actions
8. Complaints: resolutions
9. Community Outreach
10. Special Events
11. Maintenance
12. Geographic distribution

Louisville

1. Rides
2. Vehicles
3. Parking: performance, preferred, designated
4. Operator Staff Levels
5. Complaints
6. Vandalism
7. Collision: injury, fatality
8. Gender
9. Age: 8 buckets
10. Equity Distribution

San Francisco

1. System alerts
2. Pricing plans
3. Hours of operations
4. Complaints: response times, wait times, nature
5. Demographics
6. Injury
7. Low income participation (communities of concert low income, cash options)
8. Distribution stats
9. Parking and rebalancing
10. Energy sources
11. Outreach activities
12. Counts, trips, length, revenue
13. Training

San Jose

1. Complaints
2. Parking
3. Collisions: location, # involved, severity, response
4. Maintenance

Santa Monica

1. Collisions
2. Injuries
3. Operations
4. Complaints: responses
5. Maintenance
6. Education/outreach
7. Deployment zones: percent targets

Washington DC

1. Trips: total, per vehicle, average, distance
2. Equity areas: start/end
3. Violations: response time
4. Equity programs
5. Safety
6. Education
7. Parking: locations, violations, incentives
8. Idle Time: per vehicle
9. Vehicles: days in service, distance (mean, median, StdDev), decommissioned
10. Repairs: lights, seats, brakes, gears, locks, frame, other
11. Complaints: time, vehicle, location, severity, collision
12. Special users: total, low income (signups, rides, count, minutes, miles)

Cities with low income participation count requirements:
Denver, Detroit, DC, San Francisco. Also Portland, Chicago but I don't see it explicitly in their policy.

Cities with equity distribution count requirements:
DC, Santa Monica, Louisville, Long Beach, Denver. Many more, just not called out in monthly reports area of their policy

[johnclary]

to be clear, I support the metrics API and really appreciate @dirkdk's work on it. but are there member agencies pushing for these "special groups" metrics as they're currently being proposed? would like to hear from them.

[dirkdk]

Cities generally have equity as a key goal and basic mandate, working to ensure that all its residents have the opportunity to be on equal footing. We as an operator have this same goal, which is the reason we established our Access program, providing low-income pricing and a means to use the service without a smartphone or credit card. We track these metrics internally to gauge the effectiveness of our Access program, and inform decisions on any needed adjustments to the program. Cities essentially do the same thing, working to ensure any pilots or programs they deliver are working to achieve equitable outcomes, and evaluating operators to that end.

Pretty much all cities where we operate require the metrics included in this proposal as part of our data sharing terms, although it is generally shared in aggregated form via a custom report via spreadsheet or powerpoint. As stated in the original issue and on the working group call when this was discussed, Spin feels strongly about the need to protect the privacy of users, and that this proposal strikes a good balance of sharing data with cities while maintaining privacy.

Use cases should be driving changes to MDS, and we do see a clear use case here for sharing aggregated data on equity programs. Maybe a solution could be that this issue be solely focused on that equity program data, and the other groups or categories of data mentioned in the replies to this issue should be tabled until a clear and meaningful use case is presented. It would also be good to hear another operator's perspective such as @bhandzo from Bird.

@dirkdk and @joshuaandrewjohnson1, Spin