In this assignment, we will be working with an open data set provided by NYCOpenData, the open data platform of the City of New York. The data set contains over 3 million FDNY dispatch records for fire incidents in the City of New York. The scope of this assignment is intentionally defined in broad terms and gives room for creative expression, so please make sure to document all decisions and assumptions you make while tackling the given problems.
The data set can be obtained as a CSV file directly from NYCOpenData at https://data.cityofnewyork.us/Public-Safety/Fire-Incident-Dispatch-Data/8m42-w767.
The first task is concerned with profiling the data at hand. The focus should be put onto assessing the data quality and understanding the data set in depth, especially in respect to any biases inherent in the data and the distribution of values.
Now that we have acquired a deeper understanding of the data set, we want to train a prediction model on the data that is able to predict the number of dispatched vehicles for a randomly given dispatch record. You will need to clean and preprocess the data set in order to be able to use it as training data for a prediction model. You are free to choose a model (or multiple models) that you think is well-suited for the task and will deliver state-of-the-art prediction performance. Please make sure to include an evaluation and validation of your trained model(s).
The following should be included in your submission:
- The runnable code used to profile, clean, and preprocess the data and train the model(s)
- The result of the data reporting in a suitable format
- Documentation of your thought process in written form
- Any intermediary or derived data sets you create during this assignment
- Evaluation results for the final model(s)