Merge pull request #1 from cmu-delphi/main

Sync repository

Author: benjaminysmith

dev/docker/database/epidata/Dockerfile

@@ -10,3 +10,6 @@ ENV MYSQL_PASSWORD pass
 
 # provide DDL which will create empty tables at container startup
 COPY repos/delphi/delphi-epidata/src/ddl/*.sql /docker-entrypoint-initdb.d/
+
+# grant access to SQL scripts
+RUN chmod o+r /docker-entrypoint-initdb.d/*.sql

dev/docker/web/epidata/Dockerfile

@@ -7,3 +7,5 @@ COPY repos/delphi/delphi-epidata/src/server/*.php /var/www/html/epidata/
 
 # point to the development database (overwrites the production config)
 COPY repos/delphi/delphi-epidata/dev/docker/web/epidata/assets/database_config.php /var/www/html/epidata/
+
+RUN chmod o+r /var/www/html/epidata/*

docs/api/covidcast-signals/jhu-csse.md

@@ -9,7 +9,7 @@ grand_parent: COVIDcast API
 
 * **Source name:** `jhu-csse`
 * **Number of data revisions since 19 May 2020:** 1
-* **Date of last change:** [3 June 2020](../covidcast_changelog.md#jhu-csse)
+* **Date of last change:** [7 October 2020](../covidcast_changelog.md#jhu-csse)
 * **Available for:** county, hrr, msa, state (see [geography coding docs](../covidcast_geography.md))
 
 This data source of confirmed COVID-19 cases and deaths is based on reports made
@@ -59,41 +59,32 @@ should be interpreted purely as an artifact of data reporting and correction.
 Due to differences in state reporting standards, certain counties are not
 reported in the JHU data or are reported combined with other counties.
 
-At the County (FIPS) level, we report the data _exactly_ as JHU reports their
-data, to prevent confusing public consumers of the data. JHU FIPS reporting
-matches that used in the other signals, except for the following exceptions.
+### Rhode Island
 
-### New York City
+As of June 2020, the JHU data [does not report any cases or deaths](https://github.com/CSSEGISandData/COVID-19/issues/3165) for individual counties in Rhode Island. These cases and deaths are reported unassigned to any individual county, and are attributed to the megaFIPS 44000 as well as contributing to the state-level estimates.
 
-New York City comprises five boroughs, each of which is a county:
+### Puerto Rico
 
-|Borough Name       |County Name        |FIPS Code      |
-|-------------------|-------------------|---------------|
-|Manhattan          |New York County    |36061          |
-|The Bronx          |Bronx County       |36005          |
-|Brooklyn           |Kings County       |36047          |
-|Queens             |Queens County      |36081          |
-|Staten Island      |Richmond County    |36085          |
+JHU does not report Puerto Rico deaths at the municipal level, but instead reports the numbers for the whole Commonwealth in the "Unassigned" category. We map this JHU UID to megaFIPS 72000 and use this information when reporting deaths at the state level.
 
-**Data from all five boroughs are reported under New York County, FIPS Code
-36061.** The other four boroughs are included in the dataset and show up in our
-API, but always report zero cases and deaths.
+### Utah
 
-All NYC counts are mapped to the MSA with CBSA ID 35620, which encompasses all
-five boroughs. All NYC counts are mapped to HRR 303, which intersects all five
-boroughs (297 also intersects the Bronx and 301 also intersects Brooklyn and
-Queens, but absent additional information, we chose to leave all counts in 303).
+JHU reports some counts for Utah in special geocodes. These geocodes correspond to clusters of counties overseen by the same health department.
 
-### Rhode Island
+|County Name        |JHU UID Code   |
+|-------------------|---------------|
+|Bear River         |84070015       |
+|Central Utah       |84070016       |
+|Southeast Utah     |84070017       |
+|Southwest Utah     |84070018       |
+|TriCounty          |84070019       |
+|Weber-Morgan       |84070020       |
 
-As of June 2020, the JHU data does not report any cases or deaths for individual
-counties in Rhode Island. These cases and deaths are reported unassigned to any
-individual county, and are hence available in state-level estimates only.
+These are mapped to the megaFIPS 49000 and accounted for in the state totals.
 
 ### Kansas City, Missouri
 
-Kansas City intersects the following four counties, which themselves report
-confirmed case and deaths data:
+Kansas City intersects the following four counties, which themselves report confirmed case and deaths data:
 
 |County Name        |FIPS Code      |
 |-------------------|---------------|
@@ -102,29 +93,17 @@ confirmed case and deaths data:
 |Cass County        |29037          |
 |Clay County        |29047          |
 
-**Data from Kansas City is given its own dedicated line, with FIPS
-code 70003.**  This is how JHU encodes their data.  However, the data in
-the four counties that Kansas City intersects is not necessarily zero.
-
-For the mapping to HRR and MSA, the counts for Kansas City are dispersed to
-these four counties in equal proportions.
+**We distribute count totals to each of these counties in proportion to the population.**  JHU reports all of these in the UID 84070003, which does not correspond to a Census Bureau FIPS.
 
 ### Dukes and Nantucket Counties, Massachusetts
 
-**The counties of Dukes and Nantucket report their figures together,
-and we (like JHU) list them under FIPS Code 70002.**  Here are the FIPS codes
-for the individual counties:
+**The counties of Dukes and Nantucket report their figures together; We distribute count totals to each of these counties in proportion to the population.**  JHU reports all of these in the UID 84070002, which does not correspond to a Census Bureau FIPS. Here are the FIPS codes for the individual counties:
 
 |County Name        |FIPS Code      |
 |-------------------|---------------|
 |Dukes County       |25007          |
 |Nantucket County   |25019          |
 
-For the mapping to HRR and MSA, the counts for Dukes and Nantucket are
-dispersed to the two counties in equal proportions.
-
-The data in the individual counties is expected to be zero.
-
 ### Mismatched FIPS Codes
 
 Finally, there are two FIPS codes that were changed in 2015 (see the [Census

docs/api/covidcast_changelog.md

@@ -44,6 +44,19 @@ Duplicate survey weights had corrupted historical figures for the following sign
 
 We now include figures on Puerto Rico for all `jhu-csse` signals at the state level.
 
+#### 1 September 2020
+
+NY Boroughs county FIPS (36005, 36047, 36061, 36081, 36085) are now split in proportion to the population of each county, instead of being reported in aggregate in FIPS 36061. 
+
+#### 7 October 2020
+
+The following changes were made to all `jhu-csse` signals related to geocoding
+- NY Boroughs county FIPS (36005, 36047, 36061, 36081, 36085) are now [differentiated by JHU](https://github.com/CSSEGISandData/COVID-19/issues/3084),
+- “Unassigned” and “Out of State” counts are assigned to a megaFIPS at the county level (XX000 where XX is the state FIPS code) and will now be incorporated into the state totals
+- The split of the Kansas City, Missouri into its four subcounties is now done with a more precise population proportion,
+- The split of the Dukes and Nantucket, Massachusetts FIPS codes into the two subcounties is now done with a more precise population proportion,
+- Some counts are reported in the Utah territories JHU UIDs 84070015-84070020. These are now properly aggregated into the state megaFIPS 49000, which will be aggregated in state level information.
+
 ### `indicator-combination`
 
 #### 3 June 2020

docs/api/covidcast_signals.md

@@ -38,7 +38,7 @@ map](https://covidcast.cmu.edu/):
 | Early Indicators | COVID-Related Doctor Visits                  | [`doctor-visits`](covidcast-signals/doctor-visits.md)                 | `smoothed_adj_cli`               |
 | Early Indicators | COVID Indicator Combination                         | [`indicator-combination`](covidcast-signals/indicator-combination.md) | `nmf_day_doc_fbc_fbs_ght`        |
 | Late Indicators  | COVID Antigen Test Positivity (Quidel)             | [`quidel`](covidcast-signals/quidel.md)                               | `covid_ag_smoothed_pct_positive` |
-| Late Indicators  | COVID Hospital Admissions              | [`hospital-admissions`](covidcast-signals/hospital-admissions.md)     | `smoothed_adj_covid19`           |
+| Late Indicators  | COVID Hospital Admissions              | [`hospital-admissions`](covidcast-signals/hospital-admissions.md)     | `smoothed_adj_covid19_from_claims`           |
 | Late Indicators  | Cases                            | [`indicator-combination`](covidcast-signals/indicator-combination.md) | `confirmed_7dav_incidence_num`   |
 | Late Indicators  | Cases per 100,000 People         | [`indicator-combination`](covidcast-signals/indicator-combination.md) | `confirmed_7dav_incidence_prop`  |
 | Late Indicators  | Deaths                           | [`indicator-combination`](covidcast-signals/indicator-combination.md) | `deaths_7dav_incidence_num`      |

docs/epidata_development.md

@@ -94,19 +94,28 @@ docker build -t delphi_database_epidata \
 
 # test
 
-At this point you're ready to bring the stack online. To do that, just start
-containers for the epidata-specific web and database images. As an aside, the
+At this point you're ready to bring the stack online. 
+
+First, make sure you have the docker network set up so that the containers can
+communicate:
+
+```
+docker network create --driver bridge delphi-net
+
+```
+
+Next, start containers for the epidata-specific web and database images. As an aside, the
 output from these commands (especially the web server) can be very helpful for
 debugging. For example:
 
 ```bash
 # launch the database
-docker run --rm -p 13306:3306 \
+docker run --rm -p 127.0.0.1:13306:3306 \
   --network delphi-net --name delphi_database_epidata \
   delphi_database_epidata
 
 # launch the web server
-docker run --rm -p 10080:80 \
+docker run --rm -p 127.0.0.1:10080:80 \
   --network delphi-net --name delphi_web_epidata \
   delphi_web_epidata
 ```