m202-svi-tri-icis-places.ipynb

{
  "cells": [
    {
      "cell_type": "raw",
      "metadata": {},
      "source": [
        "---\n",
        "title: \"Exploring Air Quality, Social Vulnerability, and Health Outcomes in Metro Detroit\"\n",
        "author: \"Joshua Brinks\"\n",
        "date: \"September 4, 2024\"\n",
        "df-print: paged\n",
        "code-block-bg: true\n",
        "code-block-border-left: \"#315c86\"\n",
        "format: \n",
        "    html\n",
        "keep-ipynb: True\n",
        "bibliography: svi-references.bib\n",
        "---"
      ],
      "id": "547bc1c8"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Overview\n",
        "\n",
        "In this lesson, we will introduce you to 3 public health and air quality datasets: the United States [Environmental Protection Agency](https://www.epa.gov/)'s (EPA's) [Toxic Release Inventory](https://www.epa.gov/toxics-release-inventory-tri-program/tri-toolbox) (TRI), the EPA's [Integrated Compliance Information System for Air](https://www.epa.gov/enviro/icis-air-overview) (ICIS-AIR), and the U.S. [Centers For Diesease Control and Prevention's](https://www.cdc.gov/) (CDC's) [PLACES health outcomes](https://www.cdc.gov/places/index.html) dataset. You will learn how to form queries and retrieve each dataset from their respective API, process the returned object into a useable format, create visualizations and perform simple analysis. You will also use the CDC's [Social Vulnerability Index](https://sedac.ciesin.columbia.edu/data/set/usgrid-us-social-vulnerability-index-rev01) (SVI), available through NASA, to examine socioeconomic patterns in the greater Detroit metropolitan area and explore relationships with public health.\n",
        "\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "## Programming Reminder\n",
        "\n",
        "This lesson uses the Python programming environment.\n",
        ":::\n",
        "\n",
        "::: column-margin\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Data Science Review\n",
        "\n",
        "An Application Programming Interface (API) is a set of protocols, tools, and definitions that enable different software applications to communicate with each other. It acts as an intermediary, allowing one application to request specific data or actions from another application or service without needing access to its entire codebase. APIs define the methods and data formats that applications can use to request and exchange information, typically through specific *endpoints* (URLs) that accept requests and return responses in standardized formats like JSON. They are crucial for integrating external services, accessing databases, and enabling communication between different parts of larger open science software systems.\n",
        ":::\n",
        ":::\n",
        "\n",
        "## Learning Objectives\n",
        "\n",
        "After completing this lesson, you should be able to:\n",
        "\n",
        "-   Retrieve multiple public health datasets through the EPA and CDC's APIs.\n",
        "-   Create multipanel raster maps from multilayered SVI data.\n",
        "-   Geolocate tabular data using latitude and longitude.\n",
        "-   Create maps joining geolocated points and basemaps.\n",
        "-   Create maps with graduated point symbols to visualize point source pollution releases.\n",
        "-   Interpolate a raster surface from point spatial data.\n",
        "-   Perform raster math to create specialized environmental and socio-economic indexes.\n",
        "-   Perform spatial correlation analysis.\n",
        "\n",
        "::: callout-warning\n",
        "## Preface: A Note on Data Interpretation and Ethical Considerations\n",
        "\n",
        "The topics of environmental justice, institutional racism, socioeconomic conditions, and pollution are complex and multifaceted. The data and analyses presented in this lesson are not intended to draw definitive conclusions or suggest scientific evidence of cause and effect relationships. Rather, they are meant to equip you with the skills to investigate data and perform analyses that could be applied to your local communities.\n",
        "\n",
        "As you work through this lesson, remember that correlation does not imply [causation](https://www.coursera.org/articles/correlation-vs-causation). The patterns you may observe in the data could be influenced by factors not represented in these datasets. Approach your findings with caution, and consider the broader historical, social, and economic contexts that shape environmental and health outcomes in different communities.\n",
        "\n",
        "This lesson will empower you with data literacy and analytical skills. We encourage you to use these tools and skills responsibly. Consider the ethical implications of your analyses and the potential impact on the communities represented in the data. When drawing insights or making decisions based on such analyses, it’s crucial to involve community stakeholders, consider multiple perspectives, and seek additional expertise when necessary.\n",
        ":::\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In many urban areas, air quality issues disproportionately affect low-income communities and communities of color [@tessum2021; @liu2021]. This disparity is a key focus of environmental justice efforts. In cities like Detroit and Chicago, industrial facilities, highways, and other pollution sources are often concentrated in disadvantaged neighborhoods. Detroit has a history of industrial pollution and is working to address air quality issues in areas with relatively lower socio-economic status like Southwest Detroit, where residents face higher exposure to pollutants from nearby industries and heavy truck traffic [@rector2017; @air_quality_michigan; @restum2022; @hartig2024; @safta2024]. Similarly, Chicago has seen community efforts to address air pollution in areas like the Southeast Side, where residents have fought against polluting industries and advocated for stricter regulations [@gordon2022; @nature_conservancy_2021; @illgner2022].\n",
        "\n",
        "The EPA is the primary federal agency responsible for environmental protection in the United States. It sets and enforces air quality standards, including the [National Ambient Air Quality Standards (NAAQS)](https://www.epa.gov/criteria-air-pollutants/naaqs-table) for six criteria pollutants. The EPA also maintains the [AirNow](https://www.airnow.gov/about-airnow/) system, which provides real-time air quality data to the public. While primarily focused on public health, the CDC plays a crucial role in understanding the health impacts of air pollution. It conducts research on the relationships between air quality and various health outcomes, and provides guidance on protecting public health from environmental hazards [@cdc2024].\n",
        "\n",
        "In response to these challenges, community-driven science initiatives have emerged. These efforts involve local residents in collecting data on air quality and other environmental factors, often using low-cost sensors and mobile monitoring techniques [@morawska2018; @lu2022; @park2021; @su2024]. This approach helps fill gaps in official monitoring networks and empowers communities to advocate for themselves. Open data is crucial for community-driven solutions in several ways:\n",
        "\n",
        "-   Transparency: Open data allows communities to verify official reports and hold authorities accountable.\n",
        "-   Accessibility: When air quality data is freely available, communities can use it to inform local decision-making and advocacy efforts.\n",
        "-   Innovation: Open data enables researchers, activists, and tech developers to create new tools and analyses that can benefit communities.\n",
        "-   Collaboration: Open data facilitates collaboration between communities, scientists, and policymakers, leading to more effective solutions.\n",
        "\n",
        "While the EPA and CDC provide federal networks of open-access datasets like the Air Quality System (AQS), TRI, ICIS-AIR, and Population Level Analysis and Community Estimates (PLACES) collections, community driven data collection is a large part of driving change in traditionally underrepresented communities. In Chicago, the [Array of Things](https://arrayofthings.github.io/) project has installed sensors throughout the city, providing open data on various environmental factors including air quality, while Detroit's [Community Air Monitoring Project](https://www.michigan.gov/-/media/Project/Websites/egle/Documents/Reports/AQD/monitoring/2018-04-27-48217-air-monitoring-project.pdf?rev=33ae4f2a312149219bcdf9f25a538cca) uses low-cost sensors to collect and share air quality data in areas underserved by official monitoring stations.\n",
        "\n",
        "With access to open data, communities can:\n",
        "\n",
        "-   Identify local air quality hotspots that may be missed by sparse official monitoring networks and potential discrepancies in available data.\n",
        "-   Detect inconsistencies or errors in federal air quality data, ensuring more accurate and localized environmental assessments.\n",
        "-   Correlate air quality data with health outcomes to strengthen advocacy efforts.\n",
        "-   Develop targeted interventions, such as promoting indoor air filtration on high-pollution days.\n",
        "-   Create custom alerts and information systems tailored to local needs.\n",
        "\n",
        "Although open data provides many benefits, challenges still remain. These include: 1) ensuring data quality and consistency, especially when integrating data from various sources; 2) addressing the \"digital divide\" to ensure all community members can access and use the data; 3) balancing the need for detailed local data with privacy concerns; and 4) building capacity within communities to effectively use and interpret complex environmental data.\n",
        "\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### EJ in the News\n",
        "\n",
        "The [Clear the Air Coalition](https://cleartheairmi.org/), a new environmental justice advocacy group in Michigan, is calling for stronger protection of vulnerable communities from pollution. Launched in Detroit, the coalition argues that state regulators focus too much on technical compliance with environmental laws rather than public health. They claim the current permitting process favors polluters and fails to consider the cumulative impacts of pollution on overburdened communities. The group seeks changes in state law and a shift in mindset from environmental regulators.\n",
        "\n",
        "[![Clear the Air Coalition](https://npr.brightspotcdn.com/dims4/default/42a4f01/2147483647/strip/true/crop/960x720+0+0/resize/1760x1320!/format/webp/quality/90/?url=http%3A%2F%2Fnpr-brightspot.s3.amazonaws.com%2F78%2F01%2F1711cf6e442dbe579d452c0fc6f6%2Fclearair.jpg)](https://www.michiganpublic.org/environment-climate-change/2024-05-02/new-environmental-justice-coalition-aims-to-change-state-law-pressure-egle)\n",
        "\n",
        "The [Michigan Department of Environment, Great Lakes, and Energy (EGLE)](https://www.michigan.gov/egle/) emphasizes its commitment to protecting underserved communities and notes improvements in air quality over recent decades, but Coalition members argue that the current approach is insufficient. They call for regulators to consider the combined effects of multiple pollution sources in marginalized areas and to give local governments more power to reject new polluting facilities in already burdened communities.\n",
        ":::\n",
        "\n",
        "The Michigan EGLE and the Office of the [Environmental Justice Public Advocate](https://www.michigan.gov/egle/about/organization/environmental-justice) are two of the more active state departments aiming to address issues of the environment, public health, and social justice. Here are some of the projects enacted in greater Detroit area:\n",
        "\n",
        "-   [Detroit Environmental Agenda](https://detroitenv.org/): This community-led initiative works closely with EGLE to address environmental concerns in Detroit. It focuses on issues like air quality, water quality, and waste management.\n",
        "-   [48217 Community Air Monitoring Project](https://www.michigan.gov/-/media/Project/Websites/egle/Documents/Reports/AQD/monitoring/2018-04-27-48217-air-monitoring-project.pdf): Named after the zip code of a heavily industrialized area in Southwest Detroit, this project involves community members working with EGLE to monitor air quality using low-cost sensors.\n",
        "-   [Detroit Climate Action Plan](https://detroitenvironmentaljustice.org/climate-action-plan/): Developed in partnership with EGLE, this plan addresses climate change impacts on the city, with a focus on vulnerable communities.\n",
        "-   [Delray Neighborhood Initiatives](https://detroitmi.gov/departments/planning-and-development-department/neighborhood-plans/central-design-region/delray): EGLE has been involved in efforts to address air quality concerns in the Delray neighborhood, which is impacted by industrial emissions and heavy truck traffic.\n",
        "-   [Green Door Initiative](https://www.greendoorinitiative.org/): This Detroit-based organization collaborates with EGLE on various environmental justice projects, including lead abatement and air quality improvement efforts.\n",
        "-   [Detroit River Sediment Cleanup](https://www.epa.gov/great-lakes-aocs/detroit-riverwalk-great-lakes-legacy-act-project-detroit-river-aoc): EGLE has been involved in efforts to clean up contaminated sediments in the Detroit River, which disproportionately affects nearby low-income communities.\n",
        "-   [Asthma Prevention Programs](https://getasthmahelp.org/asthma-collaborative-of-detroit.aspx): EGLE supports community-based asthma prevention programs in Detroit, recognizing the link between air quality and asthma rates in disadvantaged neighborhoods.\n",
        "\n",
        "These are just a few examples that demonstrate the ongoing collaboration between community groups, local government, and EGLE to address environmental justice concerns in Detroit.\n",
        "\n",
        "Air quality issues in urban areas disproportionately affect low-income communities and communities of color, making it a critical focus for environmental justice efforts. Federal agencies like the EPA and CDC play crucial roles in setting standards and conducting research, but community-driven science initiatives have emerged as powerful tools for local action. Open data is key to these efforts, enabling transparency, accessibility, innovation, and collaboration. Cities like Detroit and Chicago are at the forefront of these initiatives, with projects that empower residents to monitor and advocate for better air quality. While challenges remain, particularly in data management and accessibility, the collaboration between community groups, local governments, and state agencies like Michigan's EGLE demonstrates a promising path forward.\n",
        "\n",
        "## Data Analysis and Exercises\n",
        "\n",
        "To better understand and address these complex issues of air pollution, public health impacts, and environmental inequities, we’ll work through a series of coding and data science exercises. These hands-on activities will allow users to explore some of the open datasets and tools that are available to community members and stakeholders that offer practical insights into air quality, public health, and social vulnerability.\n",
        "\n",
        "We'll begin with the ICIS-AIR dataset, then move on to the TRI Facility dataset, and finally work our way down to TRI Form A. After exploring these 3 air pollution datasets, we will take a look at the CDC PLACES health outcomes data and perform some simple analysis integrating the pollution and health outcomes data.\n",
        "\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Data Science Review\n",
        "\n",
        "This lesson uses the following Python modules:\n",
        "\n",
        "-   [`pandas`](https://pandas.pydata.org/): Essential for data manipulation and analysis.\n",
        "-   [`geopandas`](https://geopandas.org/): Extends pandas functionality to handle geospatial data.\n",
        "-   [`matplotlib`](https://matplotlib.org/): Used for creating static, animated, and interactive visualizations.\n",
        "-   [`numpy`](https://numpy.org/): Provides support for large, multi-dimensional arrays and matrices, along with mathematical functions to operate on these arrays.\n",
        "-   [`requests`](https://docs.python-requests.org/en/latest/): Allows you to send HTTP requests and interact with APIs easily.\n",
        "-   [`contextily`](https://contextily.readthedocs.io/en/latest/): Adds basemaps to your plots, enhancing the visual context of your geospatial data.\n",
        "-   [`pygris`](https://pygris.readthedocs.io/en/latest/): Simplifies the process of working with US Census Bureau TIGER/Line shapefiles.\n",
        "-   [`rasterio`](https://rasterio.readthedocs.io/en/latest/): Reads and writes geospatial raster datasets.\n",
        "-   [`xarray`](https://xarray.pydata.org/): Introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making it easier to work with labeled multi-dimensional arrays.\n",
        "-   [`shapely`](https://shapely.readthedocs.io/en/latest/): Manipulation and analysis of geometric objects in the Cartesian plane.\n",
        "-   [`scipy`](https://www.scipy.org/): Used for scientific and technical computing, particularly the `stats` and `interpolate` submodules.\n",
        "-   [`rioxarray`](https://corteva.github.io/rioxarray/stable/): Extends xarray to make it easier to handle geospatial raster data.\n",
        "-   [`time`](https://docs.python.org/3/library/time.html): Provides various time-related functions (part of Python's standard library).\n",
        "-   [`tabulate`](https://pypi.org/project/tabulate/): Used for creating nicely formatted tables in various output formats.\n",
        "-   [`pysal`](https://pysal.org/): A library of spatial analysis functions.\n",
        "-   [`splot`](https://splot.readthedocs.io/en/latest/): Provides statistical plots for spatial analysis.\n",
        "\n",
        "If you'd like to learn more about the functions used in this lesson, you can refer to the documentation on their respective websites.\n",
        "\n",
        "The `pandas` module is essential for data manipulation and analysis, while `geopandas` extends its functionality to handle geospatial data (spatial data format used to represent geographic information as a grid of cells or pixels). `matplotlib` is used for creating static, animated, and interactive visualizations. `numpy` provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays.\n",
        "\n",
        "The `requests` module allows you to send HTTP requests and interact with APIs easily. `contextily` adds basemaps to your plots, enhancing the visual context of your geospatial data. `pygris` simplifies the process of working with US Census Bureau TIGER/Line shapefiles.\n",
        "\n",
        "`rasterio` and `xarray` are used for working with geospatial raster data (spatial data format used to represent geographic information as a grid of cells or pixels). `rasterio` reads and writes geospatial raster datasets, while `xarray` introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making it easier to work with labeled multi-dimensional arrays.\n",
        "\n",
        "`shapely` is used for manipulation and analysis of geometric objects. `scipy` provides additional tools for scientific computing, including statistical functions and interpolation methods. `rioxarray` combines the functionality of rasterio and xarray for easier handling of geospatial raster data.\n",
        "\n",
        "`pysal` is a library for spatial analysis, providing tools for exploratory spatial data analysis. `splot` is a visualization module that works with pysal to create statistical plots for spatial analysis.\n",
        "\n",
        "Make sure these modules are installed before you begin working with the code in this document.\n",
        ":::\n",
        "\n",
        "### ICIS Air\n",
        "\n",
        "[ICIS-AIR (Integrated Compliance Information System for Air)](https://www.epa.gov/enviro/icis-air-overview) is a comprehensive database maintained by the U.S. Environmental Protection Agency (EPA) that focuses specifically on air quality compliance and enforcement data. It tracks information related to stationary sources of air pollution, including their compliance with various Clean Air Act regulations, permit data, and enforcement actions. While ICIS-AIR and the [Toxic Release Inventory (TRI)](https://www.epa.gov/toxics-release-inventory-tri-program) are separate systems, they have significant overlap in their coverage of industrial facilities and air emissions. Many facilities that report to TRI also have data in ICIS-AIR, providing complementary information.\n",
        "\n",
        "Where TRI focuses on the quantities of specific toxic chemicals released into the air, ICIS-AIR offers a broader picture of a facility's overall air quality compliance status, including non-toxic pollutants regulated under the Clean Air Act. Together, these systems provide a more comprehensive view of industrial air pollution sources, enabling researchers, regulators, and the public to assess both the types and quantities of air pollutants emitted (from TRI) and the regulatory compliance status of the emitting facilities (from ICIS-AIR).\n",
        "\n",
        "#### Data Processing\n",
        "\n",
        "##### Defining Our Study Area\n",
        "\n",
        "Before we can query the API for ICIS-AIR regulated facilities in the greater Detroit area, we need to create a spatial object that defines our study area and that we can use throughout this analysis. The U.S. Census Bureau defines the Detroit, MI Metropolitan Statistical Area (MSA) as including 6 counties (Wayne, Macomb, Oakland, Lapeer, Livingston, and St. Clair), but for this lesson we will focus on the core 3 counties (Wayne, Macomb, and Oakland); both population and air emissions related facilities are sparser in the outer counties.\n",
        "\n",
        "We can us **pygris** to get vector boundaries for the counties and dissolve them into a single boundary we can use for cropping data and restricting API searches.\n"
      ],
      "id": "d4cfcf12"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import geopandas as gpd\n",
        "import pandas as pd\n",
        "import pygris\n",
        "from shapely.geometry import box\n",
        "\n",
        "\n",
        "counties = ['Wayne', 'Oakland', 'Macomb']\n",
        "\n",
        "# Fetch the county boundaries for Michigan (MI) from the pygris dataset for the year 2022.\n",
        "# Then, filter the DataFrame to only include the counties in the Detroit metro area (Wayne, Oakland, Macomb).\n",
        "metro_counties = pygris.counties(state=\"MI\", year=2022)\n",
        "detroit_metro = metro_counties[metro_counties['NAME'].isin(counties)]\n",
        "\n",
        "# Combine the geometries of the selected counties into a single polygon (dissolve by state code).\n",
        "detroit_metro = detroit_metro.dissolve(by='STATEFP')\n",
        "\n",
        "# Convert the dissolved geometries into a GeoDataFrame and ensure the coordinate reference system (CRS) is EPSG:4269.\n",
        "detroit_metro = gpd.GeoDataFrame(detroit_metro, geometry='geometry', crs='EPSG:4269')\n",
        "\n",
        "# Obtain the total bounding box for the Detroit metro area polygon.\n",
        "bbox = detroit_metro.total_bounds\n",
        "\n",
        "# Create a bounding box polygon from the bounding coordinates and store it in a GeoDataFrame with the same CRS.\n",
        "bbox_polygon = gpd.GeoDataFrame(\n",
        "    geometry=[box(*bbox)],\n",
        "    crs=detroit_metro.crs\n",
        ")"
      ],
      "id": "61779f68",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's verify this by overlaying it on a basemap.\n"
      ],
      "id": "2430f608"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import contextily as ctx\n",
        "\n",
        "# Initialize a figure and axis with a 7x7 inch size for plotting.\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Reproject the Detroit metro area and bounding box to Web Mercator (EPSG:3857) to align with the basemap.\n",
        "detroit_metro_bm = detroit_metro.to_crs(epsg=3857)\n",
        "bbox_polygon_bm = bbox_polygon.to_crs(epsg=3857)\n",
        "\n",
        "# Plot the Detroit metro area with a light blue fill and dark blue edges at 30% opacity.\n",
        "detroit_metro_bm.plot(ax=ax, color='lightblue', edgecolor='darkblue', alpha=0.3)\n",
        "\n",
        "# Plot the boundary of the bounding box with a grey outline and a thicker line width.\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='grey', linewidth=2)\n",
        "\n",
        "# Add a basemap from OpenStreetMap (Mapnik) to provide background geographic context.\n",
        "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Adjust the map's visible area (extent) to fit the bounding box.\n",
        "ax.set_xlim(bbox_polygon_bm.total_bounds[0], bbox_polygon_bm.total_bounds[2])\n",
        "ax.set_ylim(bbox_polygon_bm.total_bounds[1], bbox_polygon_bm.total_bounds[3])\n",
        "\n",
        "# Remove axis lines and labels for a cleaner map visualization.\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Add a title to the plot.\n",
        "plt.title(\"Detroit Metro Study Area\", fontsize=16)\n",
        "\n",
        "# Optimize layout to ensure proper spacing around the plot.\n",
        "plt.tight_layout()\n",
        "\n",
        "# Display the final plot.\n",
        "plt.show()"
      ],
      "id": "0fc707f4",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This looks correct, and we'll use this boundary object repeatedly throughout our analysis.\n",
        "\n",
        "#### Querying the API\n",
        "\n",
        "The EPA's [Enforcement and Compliance History Online (ECHO)](https://echo.epa.gov/) website serves as a comprehensive portal for environmental compliance and enforcement data. It includes the ECHO Data Service, which provides programmatic access to EPA data through [RESTful](https://echo.epa.gov/tools/web-services) APIs. Among these is the ICIS-AIR API, which specifically offers access to air quality compliance and enforcement data. This API allows users to retrieve detailed information about stationary sources of air pollution, including facility details, permit data, emissions reports, and compliance status.\n",
        "\n",
        "The ECHO API requires that users follow a multi-step workflow to acquire data via the [Facility Search Air API](https://echo.epa.gov/tools/web-services/facility-search-air):\n",
        "\n",
        "1.  Use get_facilities to validate parameters, get summary statistics, and obtain a query_id (QID), valid for about 30 minutes.\n",
        "2.  Use get_qid with the obtained QID to paginate through facility results.\n",
        "3.  Use get_map with the QID to visualize and navigate facility locations.\n",
        "4.  Use get_download with the QID to generate a CSV file of facility information.\n",
        "\n",
        "The get_facility_info endpoint operates independently, returning either clustered summary statistics or an array of individual facilities. This API structure allows for efficient querying, visualization, and extraction of air quality compliance data, making it a valuable resource for environmental analysis and research.\n",
        "\n",
        "We'll begin by querying this database for just Michigan facilities in the workflow detailed above. From there we'll subset with our list of counties.\n",
        "\n",
        "We start with the base url for the ECHO API and our parameter list.\n"
      ],
      "id": "5ace69c7"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Base URL for the ECHO ICIS-AIR API, which provides access to air quality and compliance data from the EPA.\n",
        "base_url = \"https://echodata.epa.gov/echo/air_rest_services\"\n",
        "\n",
        "# Query parameters for the API call:\n",
        "# - \"output\": Specifies the response format as JSON.\n",
        "# - \"p_st\": Filters results to include only the state of Michigan (MI).\n",
        "params = {\n",
        "    \"output\": \"JSON\",\n",
        "    \"p_st\": \"MI\"\n",
        "}"
      ],
      "id": "744c2b7c",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we define 2 functions that will carry out the first 2 steps; 1) identifying the facilities in our search and 2) retrieve the facility data.\n"
      ],
      "id": "1d02fcef"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import requests\n",
        "\n",
        "# Define a function to query the ECHO ICIS-AIR API for facilities of interest in Michigan.\n",
        "# If successful, extract the Query ID (qid) and total number of facilities from the API response.\n",
        "def get_facilities():\n",
        "    response = requests.get(f\"{base_url}.get_facilities\", params=params)\n",
        "    if response.status_code == 200:\n",
        "        data = response.json()\n",
        "        if 'Results' in data:\n",
        "            qid = data['Results']['QueryID'] # Unique Query ID to fetch detailed facility data.\n",
        "            print(f\"Query ID: {qid}\")\n",
        "            print(f\"Total Facilities: {data['Results']['QueryRows']}\")  # Number of facilities in the query result.\n",
        "            return qid\n",
        "    # Log an error if the API request fails or the data is incomplete.\n",
        "    print(\"Failed to get facilities and QID\")\n",
        "    return None\n",
        "\n",
        "# Define a function to retrieve detailed facility data using the Query ID (qid).\n",
        "# Retrieves paginated data, adding all facility results to the all_facilities list.\n",
        "def get_facility_data(qid):\n",
        "    all_facilities = []\n",
        "    page = 1\n",
        "    while True:\n",
        "        # Query parameters include the Query ID and the page number for paginated results.\n",
        "        params = {\"qid\": qid, \"pageno\": page, \"output\": \"JSON\"}\n",
        "        response = requests.get(f\"{base_url}.get_qid\", params=params)\n",
        "        if response.status_code == 200:\n",
        "            data = response.json()\n",
        "            if 'Results' in data and 'Facilities' in data['Results']:\n",
        "                facilities = data['Results']['Facilities'] # List of facilities on the current page.\n",
        "                if not facilities:   # Break if no more facilities to retrieve.\n",
        "                    break\n",
        "                all_facilities.extend(facilities) # Add the facilities from the current page to the result list.\n",
        "                print(f\"Retrieved page {page}\")\n",
        "                page += 1\n",
        "            else:\n",
        "                break # Break if 'Results' or 'Facilities' key is missing.\n",
        "        else:\n",
        "            print(f\"Failed to retrieve page {page}\")  # Log an error if the API request fails for the current page.\n",
        "            break\n",
        "    return all_facilities"
      ],
      "id": "56f8edab",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that we've defined the functions we can make a request to the API with our Detroit metro counties.\n"
      ],
      "id": "16ea7b4c"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Step 1: Retrieve the Query ID (qid) by querying the API for ICIS-AIR facilities in Michigan.\n",
        "qid = get_facilities()\n",
        "\n",
        "if qid:\n",
        "    # Step 2: If the Query ID is retrieved, use it to get all relevant facility data.\n",
        "    print(\"Retrieving facility data...\")\n",
        "    facilities = get_facility_data(qid)\n",
        "    \n",
        "    # Convert the list of facilities into a pandas DataFrame for easier data analysis and manipulation.\n",
        "    df_icis_air = pd.DataFrame(facilities)\n",
        "    \n",
        "    # Print the total number of facilities retrieved and display the column names in the dataset.\n",
        "    print(f\"\\nSuccessfully retrieved {len(df_icis_air)} ICIS-AIR facilities for Michigan\")\n",
        "    print(\"\\nColumns in the dataset:\")\n",
        "    print(df_icis_air.columns)\n",
        "    \n",
        "else:\n",
        "    # If the Query ID is not retrieved, output an error message indicating the failure.\n",
        "    print(\"Failed to retrieve facility data\")"
      ],
      "id": "1a4dbc60",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are over three thousand facilities in the ICIS-AIR Michigan dataset. Here are the descriptions for some key columns.\n",
        "\n",
        "#### Key Columns in the [ICIS-AIR ECHO API Dataset](https://echo.epa.gov/help/facility-search/air-search-results-help)\n",
        "\n",
        "| **Key Column**                       | **Description**                                                        |\n",
        "|--------------------------------------|------------------------------------------------------------------------|\n",
        "| AIRName                              | The name of the facility                                               |\n",
        "| SourceID                             | A unique identifier for the air pollution source                       |\n",
        "| AIRStreet, AIRCity, AIRState, AIRZip | Address components of the facility                                     |\n",
        "| RegistryID                           | A unique identifier for the facility in the EPA’s registry             |\n",
        "| AIRCounty                            | The county where the facility is located                               |\n",
        "| AIREPARegion                         | The EPA region responsible for the facility                            |\n",
        "| AIRNAICS                             | North American Industry Classification System code(s) for the facility |\n",
        "| FacLat, FacLong                      | Latitude and longitude coordinates of the facility                     |\n",
        "| AIRPrograms                          | Air quality programs applicable to the facility                        |\n",
        "| AIRStatus                            | Current operational status of the facility                             |\n",
        "| AIRUniverse                          | Categorization of the facility within air quality regulation           |\n",
        "| AIRComplStatus                       | Current compliance status under the Clean Air Act                      |\n",
        "| AIRHpvStatus                         | High Priority Violator status                                          |\n",
        "| AIRQtrsWithViol                      | Number of quarters with violations                                     |\n",
        "| AIRLastViolDate                      | Date of the most recent violation                                      |\n",
        "| AIREvalCnt                           | Count of evaluations conducted                                         |\n",
        "| AIRLastEvalDate                      | Date of the most recent evaluation                                     |\n",
        "| AIRFceCnt                            | Count of Full Compliance Evaluations                                   |\n",
        "| AIRLastFceDate                       | Date of the last Full Compliance Evaluation                            |\n",
        "\n",
        "These columns provide information about each facility's location, operational characteristics, compliance history, and regulatory oversight under the Clean Air Act. It includes information on violations, evaluations, and compliance status that assess a facility's environmental performance and regulatory compliance.\n",
        "\n",
        "We can check some basic information like how many facilities are in Detroit metro and the breakdown by our 3 counties.\n"
      ],
      "id": "bb45b832"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Subset the DataFrame to include only the ICIS-AIR facilities located in the Detroit metro counties (Wayne, Oakland, Macomb).\n",
        "icis_air_detroit = df_icis_air[df_icis_air['AIRCounty'].isin(counties)]\n",
        "\n",
        "# Print a summary of the total number of ICIS-AIR facilities in Michigan and those specifically in the Detroit metro area.\n",
        "print(f\"Total ICIS-AIR facilities in Michigan: {len(df_icis_air)}\")\n",
        "print(f\"ICIS-AIR facilities in Detroit metro area: {len(icis_air_detroit)}\")\n",
        "\n",
        "# Display the count of ICIS-AIR facilities for each county within the Detroit metro area.\n",
        "print(\"\\nFacilities per county:\")\n",
        "print(icis_air_detroit['AIRCounty'].value_counts())"
      ],
      "id": "6877eaa8",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next we should check for facilities with missing latitude or longitude coordinates.\n"
      ],
      "id": "6ef9c9d6"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Identify and count the records in the Detroit metro subset where latitude or longitude values are missing.\n",
        "missing_coords = icis_air_detroit[(icis_air_detroit['FacLat'].isnull()) | (icis_air_detroit['FacLong'].isnull())]\n",
        "print(f\"Number of ICIS-AIR records with missing coordinates: {len(missing_coords)}\")\n",
        "\n",
        "# Remove records with missing latitude or longitude from the dataset to ensure only facilities with valid coordinates remain.\n",
        "icis_air_detroit = icis_air_detroit.dropna(subset=['FacLat', 'FacLong'])"
      ],
      "id": "0f33023e",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "No missing records! That's great. As you'll find out later, you won't always be so lucky.\n",
        "\n",
        "Now we can create a spatial object, reproject it so it matches the basemap of Detroit, and make a map showing the location of all the ICIS-AIR facilities in Detroit metro. The existing dataset in our Python environment includes the latitude (FacLat) and longitude (FacLong) coordinates, but we need to convert this to an explicit spatial object with GeoPandas using the coordinates so we can perform spatial operations on this object like creating maps, creating surfaces, or finding nearby objects.\n"
      ],
      "id": "a1937953"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the ICIS-AIR facilities data into a GeoDataFrame with valid point geometries (using longitude and latitude).\n",
        "gdf_icis_air = gpd.GeoDataFrame(\n",
        "    icis_air_detroit, \n",
        "    geometry=gpd.points_from_xy(icis_air_detroit.FacLong, icis_air_detroit.FacLat),\n",
        "    crs=\"EPSG:4326\" # Coordinate reference system set to WGS 84 (EPSG:4326)\n",
        ")\n",
        "\n",
        "# Reproject the ICIS-AIR facilities' GeoDataFrame to Web Mercator (EPSG:3857) for alignment with basemap data.\n",
        "gdf_icis_air_bm = gdf_icis_air.to_crs(epsg=3857)\n",
        "\n",
        "# Set up the plot with a 7x7 inch figure.\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Plot the Detroit metro area boundary and bounding box, using previously reprojected objects.\n",
        "detroit_metro_bm.plot(ax=ax, facecolor='none', edgecolor='blue', linewidth=2)\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Plot the ICIS-AIR facilities as cyan markers, with transparency (alpha=0.5) and grey outlines.\n",
        "gdf_icis_air_bm.plot(ax=ax, color='cyan', markersize=50, alpha=0.5, label='ICIS-AIR Facilities', edgecolor = \"grey\")\n",
        "\n",
        "# Add an OpenStreetMap basemap for geographical context.\n",
        "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Adjust the map's extent to fit the bounding box around the Detroit metro area.\n",
        "ax.set_xlim(bbox_polygon_bm.total_bounds[0], bbox_polygon_bm.total_bounds[2])\n",
        "ax.set_ylim(bbox_polygon_bm.total_bounds[1], bbox_polygon_bm.total_bounds[3])\n",
        "\n",
        "# Hide the axes to make the plot cleaner.\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Add a legend to the plot to label the ICIS-AIR facilities.\n",
        "ax.legend()\n",
        "\n",
        "# Add a title and improve the layout of the plot.\n",
        "plt.title(\"Detroit Metro Area ICIS-AIR Facilities\", fontsize=16)\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Output the total number of ICIS-AIR facilities plotted.\n",
        "print(f\"Number of ICIS-AIR facilities plotted: {len(gdf_icis_air)}\")"
      ],
      "id": "4547bc9a",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "These are distributed along business and industrial sectors as expected, but we can't infer much else from this map.\n",
        "\n",
        "Let’s look for regulated facilities that have been in violation status. We can plot all the facilities with at least one violation during a reporting quarter and use graduated symbols that reflect the total number of quarters with violations. This information is in the `AIRQtrsWithViol` column.\n"
      ],
      "id": "c3cfdc4a"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the 'AIRQtrsWithViol' column to numeric, replacing any non-numeric values with NaN.\n",
        "gdf_icis_air_bm['AIRQtrsWithViol'] = pd.to_numeric(gdf_icis_air_bm['AIRQtrsWithViol'], errors='coerce')\n",
        "\n",
        "# Subset the dataset to include only ICIS-AIR facilities with reported violations (quarters with violations > 0).\n",
        "gdf_icis_air_violators = gdf_icis_air_bm = gdf_icis_air_bm[gdf_icis_air_bm['AIRQtrsWithViol'] > 0]\n",
        "\n",
        "# Set up the plot with a 7x7 inch figure.\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Plot the Detroit metro area boundary and bounding box for geographical context.\n",
        "detroit_metro_bm.plot(ax=ax, facecolor='none', edgecolor='blue', linewidth=2)\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Plot the violators (facilities with violations) using graduated symbol sizes based on 'AIRQtrsWithViol' values.\n",
        "# The symbol size is scaled (multiplied by 10) to make differences more visible.\n",
        "scatter = ax.scatter(gdf_icis_air_violators.geometry.x,\n",
        "                     gdf_icis_air_violators.geometry.y, \n",
        "                     s=gdf_icis_air_violators['AIRQtrsWithViol']*10,  # Adjust marker size by violation count\n",
        "                     c='orangered', # Color the violators in 'orangered' for visibility\n",
        "                     edgecolor='yellow', # Add a yellow edge to make the points stand out\n",
        "                     linewidth=1, \n",
        "                     alpha=0.7) # Set transparency for better map visibility\n",
        "\n",
        "# Add an OpenStreetMap basemap to enhance the visual context of the plot.\n",
        "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Set the map's extent to match the bounding box around the Detroit metro area.\n",
        "ax.set_xlim(bbox_polygon_bm.total_bounds[0], bbox_polygon_bm.total_bounds[2])\n",
        "ax.set_ylim(bbox_polygon_bm.total_bounds[1], bbox_polygon_bm.total_bounds[3])\n",
        "\n",
        "# Hide axes for a cleaner map.\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Add a legend to label and distinguish facilities with violations.\n",
        "ax.legend()\n",
        "\n",
        "# Set the plot title and finalize the layout for professional presentation.\n",
        "plt.title(\"Detroit Metro Area ICIS-AIR Facilities w/ Violations\", fontsize=16)\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "id": "894d7d49",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can also create a table showing which businesses have been in violation status, their name, address, and the date of their most recent violation using the **tabulate** module.\n"
      ],
      "id": "f3f391b1"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from tabulate import tabulate\n",
        "\n",
        "# Filter the dataset to include only facilities with 10 or more quarters of violations.\n",
        "gdf_icis_air_violators = gdf_icis_air_bm = gdf_icis_air_bm[gdf_icis_air_bm['AIRQtrsWithViol'] >= 10]\n",
        "\n",
        "# Define the columns to be displayed in the final table.\n",
        "table_columns = ['AIRName', 'AIRStreet', 'AIRCity', 'AIRQtrsWithViol', 'AIRLastViolDate']\n",
        "\n",
        "# Create a new DataFrame with only the selected columns for table generation.\n",
        "table_data = gdf_icis_air_violators[table_columns].copy()\n",
        "\n",
        "# Convert the 'AIRLastViolDate' column to a datetime format for proper sorting and display.\n",
        "table_data['AIRLastViolDate'] = pd.to_datetime(table_data['AIRLastViolDate'])\n",
        "\n",
        "# Sort the DataFrame by the 'AIRLastViolDate' column in descending order to show the most recent violations first.\n",
        "table_data = table_data.sort_values('AIRLastViolDate', ascending=False)\n",
        "\n",
        "# Define a dictionary to map the original column names to more readable/pretty names for the table.\n",
        "pretty_names = {\n",
        "    'AIRName': 'Name',\n",
        "    'AIRStreet': 'Street',\n",
        "    'AIRCity': 'City',\n",
        "    'AIRQtrsWithViol': 'Violations',\n",
        "    'AIRLastViolDate': 'Violation Date'\n",
        "}\n",
        "\n",
        "# Apply the new readable column names.\n",
        "table_data.rename(columns=pretty_names, inplace=True)\n",
        "\n",
        "# Optionally, format the 'Violation Date' column to display dates in a 'YYYY-MM-DD' format.\n",
        "table_data['Violation Date'] = table_data['Violation Date'].dt.strftime('%Y-%m-%d')\n",
        "\n",
        "# Generate the table using the 'tabulate' library and format it as an HTML table for presentation.\n",
        "table = tabulate(table_data, headers='keys', tablefmt='html', showindex=False)\n",
        "\n",
        "# Display the resulting table.\n",
        "table "
      ],
      "id": "9ba91b18",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "ICIS-AIR is a powerful tool for communities and policy-makers to identify regulated facilities and their compliance status, but the dataset does have limitations. For example, we can not examine the types or amounts of chemicals being released into the community.\n",
        "\n",
        "Next, we’ll take a look at facilities required to report to the TRI.\n",
        "\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "Which of the following best describes the ICIS-AIR database?\n",
        "\n",
        "a)  A database of air quality measurements from monitoring stations\n",
        "b)  A comprehensive database of air quality compliance and enforcement data\n",
        "c)  A public health database tracking respiratory illnesses\n",
        "d)  A database of weather patterns affecting air quality\n",
        ":::\n",
        "\n",
        "### Toxic Release Inventory\n",
        "\n",
        "The [Toxic Release Inventory (TRI)](https://www.epa.gov/toxics-release-inventory-tri-program) is a crucial resource in understanding and addressing air quality issues in the United States. Established under the Emergency Planning and Community Right-to-Know Act of 1986, the TRI is a publicly accessible database maintained by the Environmental Protection Agency (EPA). It requires certain industrial facilities to report annually on their releases of toxic chemicals into the environment, including air emissions. The TRI provides valuable data on over 770 chemicals and chemical categories, offering insights into the types and quantities of pollutants released into the air by various industries. This information is vital to researchers, policymakers, and community advocates in assessing local air quality, identifying potential health risks, and developing targeted strategies to reduce toxic air emissions. By making this data publicly available, the TRI plays an important role in promoting transparency and supporting environmental justice initiatives focused on improving air quality in communities across the nation.\n",
        "\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Data Review\n",
        "\n",
        "The Toxics Release Inventory (TRI) and the Integrated Compliance Information System for Air (ICIS-AIR) are two important but distinct environmental reporting systems maintained by the U.S. Environmental Protection Agency (EPA). They have several key differences:\n",
        "\n",
        "-   **Regulatory Basis**\n",
        "    -   TRI: Established under the Emergency Planning and Community Right-to-Know Act (EPCRA) of 1986\n",
        "    -   ICIS-AIR: Part of the Clean Air Act (CAA) compliance and enforcement program\n",
        "-   **Focus**\n",
        "    -   TRI: Tracks the management of certain toxic chemicals that may pose a threat to human health and the environment\n",
        "    -   ICIS-AIR: Focuses specifically on air quality and emissions from facilities regulated under the Clean Air Act\n",
        "-   **Reported Information**\n",
        "    -   TRI: Facilities report on releases, waste management, and pollution prevention activities for specific toxic chemicals\n",
        "    -   ICIS-AIR: Tracks emissions data, compliance status, and enforcement actions related to air quality regulations\n",
        "-   **Facility Coverage**\n",
        "    -   TRI: Covers facilities in specific industries that manufacture, process, or use TRI-listed chemicals above certain thresholds\n",
        "    -   ICIS-AIR: Includes a broader range of facilities that emit air pollutants, regardless of the specific chemicals involved\n",
        "-   **Reporting Thresholds**\n",
        "    -   TRI: Has specific chemical thresholds that trigger reporting requirements\n",
        "    -   ICIS-AIR: Generally doesn't have chemical-specific thresholds; requirements are based on overall emissions and facility type\n",
        "-   **Public Accessibility**\n",
        "    -   TRI: Designed with a strong focus on public right-to-know, with data easily accessible to the public\n",
        "    -   ICIS-AIR: While public, it's primarily designed for regulatory and enforcement purposes\n",
        "-   **Data Frequency**\n",
        "    -   TRI: Annual reporting is required for covered facilities\n",
        "    -   ICIS-AIR: May involve more frequent reporting, depending on permit requirements and compliance status\n",
        "-   **Scope of Pollutants**\n",
        "    -   TRI: Focuses on a specific list of toxic chemicals and chemical categories\n",
        "    -   ICIS-AIR: Covers a wider range of air pollutants, including criteria air pollutants and hazardous air pollutants\n",
        "-   **Use in Environmental Management**\n",
        "    -   TRI: Often used for assessing long-term trends in toxic chemical releases and waste management practices\n",
        "    -   ICIS-AIR: More commonly used for day-to-day air quality management and enforcement activities\n",
        "-   **Geographic Coverage**\n",
        "    -   TRI: Nationwide program with consistent reporting across states\n",
        "    -   ICIS-AIR: While national, implementation can vary more by state or local air quality management district\n",
        ":::\n",
        "\n",
        "#### Envirofacts API\n",
        "\n",
        "[EnviroFacts](https://www.google.com/url?q=https://enviro.epa.gov/&sa=D&source=docs&ust=1733163481208997&usg=AOvVaw0dJjyDJg-FG1iSnwyCuuaH) is a comprehensive online database and information system maintained by the U.S. Environmental Protection Agency (EPA). It serves as a centralized hub for accessing a wide range of environmental data collected by the EPA and other federal agencies. EnviroFacts integrates information from multiple EPA databases, covering various aspects of environmental health, including air quality, water quality, hazardous waste, and toxic releases. One of the key components of EnviroFacts is the Toxic Release Inventory (TRI) data. Through EnviroFacts, users can easily access and query TRI information, allowing them to investigate toxic chemical releases and waste management activities in their local areas. The integration of TRI data within the broader EnviroFacts system enables researchers, policymakers, and community members to contextualize toxic release information alongside other environmental indicators.\n",
        "\n",
        "Envirofacts is available as a [web based search and data platform](https://enviro.epa.gov/) and also as a [programmatic API](https://www.epa.gov/enviro/envirofacts-data-service-api). The web platform is a great way to familiarize yourself with the available datasets and create simple downloads, however, for analytical purposes we recommend learning to navigate their API so you can create repeatable and reliable analysis.\n",
        "\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Environmental Justice In the News\n",
        "\n",
        "[The Michigan Department of Environment, Great Lakes and Energy (EGLE) has settled a civil rights complaint regarding a hazardous waste facility in Detroit](https://www.detroitnews.com/story/news/local/michigan/2024/08/29/egle-settles-civil-rights-complaint-about-detroit-hazardous-waste-approval-us-ecology-north/75001763007/). The complaint, filed in 2020 by environmental groups and local residents, challenged the renewal and expansion of [U.S. Ecology North's](https://www.michigan.gov/egle/about/organization/materials-management/hazardous-waste/liquid-industrial-byproducts/us-ecology-detroit-north/license-information) license, arguing it was unjust to increase hazardous waste storage in a predominantly low-income and minority neighborhood. As part of the settlement, EGLE will now consider environmental justice concerns in future licensing decisions for hazardous waste facilities.\n",
        "\n",
        "The agreement, described as \"groundbreaking\" by the [Sierra Club](https://www.sierraclub.org/), introduces several new measures. EGLE will conduct environmental justice and cumulative impact analyses for future licensing decisions, potentially denying licenses that would have an unlawful impact on human health and the environment. The settlement also includes provisions for improved community engagement, such as better translation services, public input processes, and the installation of air monitors around U.S. Ecology North. Additionally, the state will work with local residents to conduct a community health assessment. While some details remain to be clarified, environmental advocates view this as a significant step towards advancing environmental justice in Michigan.\n",
        ":::\n",
        "\n",
        "#### Querying the API\n",
        "\n",
        "We'll continue to use the Detroit metro boundary we created earlier. We assigned them the names `detroit_metro` and `detroit_metro_bm` (projected to mercator to match **b**ase**m**aps for plotting).\n"
      ],
      "id": "b35f8719"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Print the coordinates of the bounding box, which defines the rectangular boundary for the area of interest\n",
        "print(\"Bounding Box:\")\n",
        "\n",
        "# Display the minimum longitude (X-axis) of the bounding box\n",
        "print(f\"Minimum X (Longitude): {bbox[0]}\")\n",
        "\n",
        "# Display the minimum latitude (Y-axis) of the bounding box\n",
        "print(f\"Minimum Y (Latitude): {bbox[1]}\")\n",
        "\n",
        "# Display the maximum longitude (X-axis) of the bounding box\n",
        "print(f\"Maximum X (Longitude): {bbox[2]}\")\n",
        "\n",
        "# Display the maximum latitude (Y-axis) of the bounding box\n",
        "print(f\"Maximum Y (Latitude): {bbox[3]}\")"
      ],
      "id": "a3ed1024",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we'll create an empty list to store the TRI data, loop through each county making an API query, and place the retrieved data in the empty list.\n"
      ],
      "id": "7d5c7a78"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Initialize a list to hold TRI (Toxic Release Inventory) facility data for each county in Michigan\n",
        "tri_data = []\n",
        "\n",
        "# Loop through each county in the specified list to fetch TRI data separately\n",
        "# (avoiding issues encountered when querying all counties at once)\n",
        "for county in counties:\n",
        "    # Construct the API URL to query TRI facilities for the current county\n",
        "    api_url = f\"https://data.epa.gov/efservice/tri_facility/state_abbr/MI/county_name/{county}/JSON\"\n",
        "    response = requests.get(api_url)\n",
        "\n",
        "    # Check if the response was successful\n",
        "    if response.status_code == 200:\n",
        "        county_data = response.json()\n",
        "        # Append the current county's TRI data to the overall list\n",
        "        tri_data.extend(county_data)\n",
        "    else:\n",
        "        # Print an error message if data retrieval failed for the county\n",
        "        print(f\"Failed to fetch data for {county} County. Status code: {response.status_code}\")"
      ],
      "id": "d1842d22",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Processing the Data\n",
        "\n",
        "The TRI data comes in json format. This format can be confusing to interpret, because each record (traditionally a row in a table) is viewed with columns structured vertically. Let’s look at the first record.\n"
      ],
      "id": "4da84de4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Access and print the second entry in the TRI data list to inspect a sample facility's data structure\n",
        "tri_data[1]"
      ],
      "id": "4d78118b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This record shows all the information for *PPG GROW DETROIT* with columns and values listed side-by-side ('city_name': 'DETROIT'). This may seem difficult to deal with, but most programming languages have tools to easily parse json data into traditional tables. `pd.DataFrame` does it automatically when you attempt to create a **pandas** data frame.\n"
      ],
      "id": "c43d6a95"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the collected TRI data list into a Pandas DataFrame for easier data manipulation and analysis\n",
        "tri_df = pd.DataFrame(tri_data)\n",
        "\n",
        "# Output the total count of facilities retrieved to confirm successful data fetching\n",
        "print(f\"Number of facilities fetched: {len(tri_df)}\")\n",
        "\n",
        "# Display the first few rows of the DataFrame for a quick data preview (uncomment if needed for inspection)\n",
        "# tri_df.head()"
      ],
      "id": "9c290db1",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Checking the first few rows--everything looks as it should. We want to geocode the facility locations into a point layer using the latitude and longitude values. Therefore, we should start by making sure all the facilities have coordinate values.\n",
        "\n",
        "We'll check and remove those without.\n"
      ],
      "id": "a795408f"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Create a copy of the TRI DataFrame to prevent modifications that could trigger a SettingWithCopyWarning\n",
        "tri_df_clean = tri_df.copy()\n",
        "\n",
        "# Remove rows where latitude or longitude is missing to ensure data completeness\n",
        "tri_df_clean = tri_df_clean.dropna(subset=['pref_latitude', 'pref_longitude'])\n",
        "\n",
        "print(f\"Number of facilities after removing empty coordinates: {len(tri_df_clean)}\")\n",
        "\n",
        "# Convert latitude and longitude columns to numeric, handling non-numeric values by setting them to NaN\n",
        "tri_df_clean['pref_latitude'] = pd.to_numeric(tri_df_clean['pref_latitude'], errors='coerce')\n",
        "tri_df_clean['pref_longitude'] = pd.to_numeric(tri_df_clean['pref_longitude'], errors='coerce')"
      ],
      "id": "9f6a52ee",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Ouch! We lost roughly 300 records due to missing coordinate information (789 vs. 478). If this were an important analysis for policy-making or scientific research we would take the time to [geocode](https://www.mapbox.com/insights/geocoding) the listed addresses into latitude/longitude coordinates, but for the purposes of this exercise we’ll move on.\n",
        "\n",
        "In my personal investigations of this data, I noticed that some longitude coordinates were not negative, but had an absolute value (e.g. 83.25) that made sense for the Detroit metro area, therefore we will flip the sign on these records.\n"
      ],
      "id": "8f77a69e"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define a function to correct longitudes that are mistakenly positive, converting them to negative values for North America\n",
        "def correct_longitude(lon):\n",
        "    if lon > 0:\n",
        "        return -lon\n",
        "    return lon\n",
        "\n",
        "# Apply longitude correction to all facilities in the DataFrame\n",
        "tri_df_clean['pref_longitude'] = tri_df_clean['pref_longitude'].apply(correct_longitude)"
      ],
      "id": "465da46e",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Additionally, there are some records with wild longitudinal values that are not simply from a missing negative sign. We'll identify these outliers using the interquantile range. Anything outside the range will be tossed. One of the downsides of the TRI database is that many aspects are provided directly by the facility (and therefore subject to errors). Again, if this were a more important analysis we would carefully explore all of these missing records and possibly geocode using the address.\n",
        "\n",
        "Toss the quartile outliers.\n"
      ],
      "id": "84375589"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Calculate the Interquartile Range (IQR) for longitude to identify and remove outlier values\n",
        "Q1 = tri_df_clean['pref_longitude'].quantile(0.25)\n",
        "Q3 = tri_df_clean['pref_longitude'].quantile(0.75)\n",
        "IQR = Q3 - Q1\n",
        "\n",
        "# Define acceptable range for longitude, filtering out points beyond 1.5 * IQR\n",
        "lower_bound = Q1 - 1.5 * IQR\n",
        "upper_bound = Q3 + 1.5 * IQR\n",
        "\n",
        "lower_bound, upper_bound\n",
        "\n",
        "# Remove facilities with longitudes outside of the defined bounds to eliminate extreme outliers\n",
        "tri_df_clean = tri_df_clean[(tri_df_clean['pref_longitude'] >= lower_bound) & \n",
        "                            (tri_df_clean['pref_longitude'] <= upper_bound)]\n",
        "\n",
        "print(f\"Number of facilities after removing longitude outliers: {len(tri_df_clean)}\")"
      ],
      "id": "548c27aa",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Thankfully we only lost 2 more records from outliers.\n",
        "\n",
        "#### Visualizing the Data\n",
        "\n",
        "The data has been cleaned up a bit so now lets create a spatial point object from the data with **geopandas** and use **matplotlib** to plot the values on top of a basemap of Detroit we'll acquire using **contextily** and our original Wayne, Macomb, and Oakland counties boundary.\n"
      ],
      "id": "223a85a6"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Create a GeoDataFrame from the cleaned TRI data, converting latitude and longitude into geometric points\n",
        "detroit_tri = gpd.GeoDataFrame(\n",
        "    tri_df_clean, \n",
        "    geometry=gpd.points_from_xy(tri_df_clean.pref_longitude, tri_df_clean.pref_latitude),\n",
        "    crs=\"EPSG:4326\" # Set the coordinate reference system to WGS 84 (latitude/longitude)\n",
        ")\n",
        "\n",
        "# Reproject TRI data to the Web Mercator projection (EPSG:3857) for compatibility with the basemap\n",
        "detroit_tri = detroit_tri.to_crs(epsg=3857)\n",
        "\n",
        "# Initialize a plot with a fixed size\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Plot the Detroit metro area boundaries and the bounding box as background layers\n",
        "detroit_metro_bm.plot(ax=ax, facecolor='none', edgecolor='blue', linewidth=2)\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Plot TRI facilities as purple points with a grey edge, representing facilities' locations in the metro area\n",
        "detroit_tri.plot(ax=ax, color='purple', edgecolor='grey', markersize=50, alpha=0.5, label='TRI Facilities')\n",
        "\n",
        "# Plot ICIS-AIR facilities as cyan points with a grey edge, indicating the locations of air-quality-monitored facilities\n",
        "gdf_icis_air_bm.plot(ax=ax, color='cyan', edgecolor='grey', markersize=50, alpha=0.5, label='ICIS-AIR Facilities')\n",
        "\n",
        "# Overlay a basemap for geographical context, using OpenStreetMap tiles\n",
        "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Set the map's viewing area to the bounding box’s coordinates, zooming in on the Detroit metro area\n",
        "ax.set_xlim(bbox_polygon_bm.total_bounds[0], bbox_polygon_bm.total_bounds[2])\n",
        "ax.set_ylim(bbox_polygon_bm.total_bounds[1], bbox_polygon_bm.total_bounds[3])\n",
        "\n",
        "# Hide axis labels and ticks for a cleaner map presentation\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Add a legend to differentiate TRI and ICIS-AIR facilities on the map\n",
        "ax.legend()\n",
        "\n",
        "# Add a title for context and adjust layout for better spacing\n",
        "plt.title(\"Detroit Metro Area TRI and ICIS-AIR Facilities\", fontsize=16)\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Print the number of facilities included in the plot for confirmation\n",
        "print(f\"Number of TRI facilities plotted: {len(detroit_tri)}\")\n",
        "print(f\"Number of ICIS-AIR facilities plotted: {len(gdf_icis_air)}\")"
      ],
      "id": "802e722b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can see that there are far fewer facilities being tracked in TRI compared to ICIS-AIR. Some of this can be attributed to the records we tossed after checking for valid geographic coordinates and outliers, but generally speaking ICIS-AIR will contain far more as it tracks every facility regulated by the Clean Air Act as opposed to only those that are part of TRI.\n",
        "\n",
        "You may have also noticed that this dataset does not contain any actual pollution data; only facility information for those that are regulated by TRI.\n",
        "\n",
        "TRI is made up of numerous tables containing a wealth of information regarding the facilities themselves and all the regulated chemicals that are handled at each site. The TRI_FACILITY table we requested is the starter table used in most of the examples in the Envirofacts API documentation, however, it does not contain any chemical release data.\n",
        "\n",
        "Navigating the TRI Envirofacts interface can be overwhelming at times. Multiple tables can be queried with a single URL, however, for a proper join to be carried out both tables must contain at least one column in common. In most cases this can be accomplished using TRI_FACILITY_ID. That said, air pollution release data (column name [AIR_TOTAL_RELEASE](https://enviro.epa.gov/enviro/ef_metadata_html.tri_page?p_column_name=air_total_release)) is only found in the [TRI_FORM_R](https://enviro.epa.gov/enviro/ef_metadata_html.ef_metadata_table?p_table_name=TRI_FORM_R&p_topic=TRI) table, which does not contain the FACILITY_ID column.\n",
        "\n",
        "You can querry the V_TRI_FORM_R_EZ using the API even though it’s not a documented table listed in the API website. This form contains facility information and AIR_TOTAL_RELEASE reporting, but it only goes until 2021. To get the most recent data (2023) you would have to manually download the individual tables with separate calls, perform the joins, or use the [custom form search web portal](https://enviro.epa.gov/facts/tri/form_ra_download.html). We’ll demonstrate a quick API call for the V_TRI_FORM_R_EZ, but move forward with the most recent data available in the custom form search web portal.\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "What is the primary purpose of the Toxic Release Inventory (TRI)?\n",
        "\n",
        "a)  To track emissions of greenhouse gases\n",
        "b)  To monitor compliance with the Clean Air Act\n",
        "c)  To provide data on toxic chemical releases from industrial facilities\n",
        "d)  To record air quality index values for major cities\n",
        ":::\n",
        "\n",
        "### Toxic Release Inventory: Form R\n",
        "\n",
        "We can make the query to Envirofacts. Let's break down the call.\n"
      ],
      "id": "c0eab569"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define the URL for the CSV file containing Toxic Release Inventory Form R data\n",
        "# This query requests data from the EPA Envirofacts API, specifying:\n",
        "# - Reporting Year: 2021\n",
        "# - Air releases greater than 0\n",
        "# - State: Michigan (MI)\n",
        "# - Counties: Wayne, Oakland, and Macomb\n",
        "url = \"https://data.epa.gov/efservice/V_TRI_FORM_R_EZ/REPORTING_YEAR/2021/AIR_TOTAL_RELEASE/>/0/STATE_ABBR/MI/COUNTY_NAME/CONTAINING/WAYNE/CONTAINING/OAKLAND/CONTAINING/MACOMB/CSV\""
      ],
      "id": "9d23e398",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "-   Base url: https://data.epa.gov/efservice/\n",
        "-   The table we're requesting: V_TRI_FORM_R\\_EZ/\n",
        "-   For just 2021: REPORTING_YEAR/2021/\n",
        "-   Only facilities that report some air release: AIR_TOTAL_RELEASE/\\>/0/\n",
        "-   Just Michigan: STATE_ABBR/MI/\n",
        "-   Our counties: COUNTY_NAME/CONTAINING/WAYNE/CONTAINING/OAKLAND/CONTAINING/MACOMB/\n",
        "-   We want it as a csv: CSV\n"
      ],
      "id": "98cd16a9"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Read the TRI Form R data CSV file directly into a pandas DataFrame\n",
        "tri_form_r = pd.read_csv(url)\n",
        "\n",
        "# Confirm successful data loading by printing the number of records\n",
        "print(f\"Successfully read CSV. Number of records: {len(tri_form_r)}\")\n",
        "\n",
        "# Display column names to understand the structure of the dataset\n",
        "print(\"\\nColumns in the dataset:\")\n",
        "print(tri_form_r.columns)"
      ],
      "id": "e6f2c510",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We have roughly the same number of facilities as before, but this table has a lot of columns (568). We should make it more manageable.\n"
      ],
      "id": "cfac6e7b"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# The dataset contains numerous columns (568), so we will retain only essential columns for analysis\n",
        "# Select columns: facility name, address, city, reporting year, and air release amounts\n",
        "keeps = [\n",
        "    'facility_name', 'street_address', 'city_name',\n",
        "    'reporting_year', 'air_total_release']\n",
        "\n",
        "# Update DataFrame to only include the selected columns\n",
        "tri_form_r = tri_form_r[keeps]"
      ],
      "id": "4388535c",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This works well but we can get more recent data that is already filtered using the [web portal](https://enviro.epa.gov/facts/tri/form_ra_download.html). The user selects location, years of interest, and columns of interests. The portal serves up your requested dataset in an Amazon S3 cloud storage. We can import the dataset directly from the address they provide.\n"
      ],
      "id": "9785856a"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Read a secondary TRI Form R CSV file directly from the URL\n",
        "# This may contain additional or alternative data for further analysis\n",
        "tri_form_r = pd.read_csv(\"https://dmap-epa-enviro-prod-export.s3.amazonaws.com/396975438.CSV\")\n",
        "\n",
        "# Confirm successful reading by printing the number of records and dataset structure\n",
        "print(f\"Successfully read CSV. Number of records: {len(tri_form_r)}\")\n",
        "print(\"\\nColumns in the dataset:\")\n",
        "print(tri_form_r.columns)"
      ],
      "id": "9181e9a2",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This is a bit more manageable, but we have multiple records per facility because each chemical release has a separate record. This allows the user to investigate specific chemicals, but for the purposes of this exercise we will aggregate `AIR_TOTAL_RELEASE` for each facility, to get a single row per facility.\n"
      ],
      "id": "6c917371"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Display the first few rows of the dataset for a quick inspection\n",
        "# Note: this printout is limited to avoid excessive output in a web environment\n",
        "# tri_form_r.head"
      ],
      "id": "1bee8a13",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's aggregate it.\n"
      ],
      "id": "db001533"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Group the dataset by facility name and location details, summing the total air releases per facility\n",
        "# This aggregation step consolidates all entries for each unique facility based on name, latitude, longitude, and address\n",
        "tri_form_r = tri_form_r.groupby(['FACILITY_NAME', 'LATITUDE','LONGITUDE','STREET_ADDRESS'], as_index=False).agg({\n",
        "    'AIR_TOTAL_RELEASE': 'sum' # Sum the air release values for each facility\n",
        "})\n",
        "# Display the total number of unique facilities after grouping\n",
        "print(f\"Successfully read CSV. Number of records: {len(tri_form_r)}\")"
      ],
      "id": "00612b3b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There were only 211 unique listings in the 2023 data.\n",
        "\n",
        "As before, we should check for valid coordinates and missing air release data.\n"
      ],
      "id": "5b5c6ed0"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define column names for latitude, longitude, and air release values\n",
        "# Assumes column names match; adjust 'LATITUDE', 'LONGITUDE', and 'AIR_TOTAL_RELEASE' if necessary\n",
        "lat_col = 'LATITUDE'\n",
        "lon_col = 'LONGITUDE'\n",
        "release_col = 'AIR_TOTAL_RELEASE'  # Column for the total air release per facility\n",
        "\n",
        "# Remove any records with missing coordinates or air release data to ensure data integrity for spatial analysis\n",
        "df_tri_clean = tri_form_r.dropna(subset=[lat_col, lon_col, release_col])\n",
        "\n",
        "# Output the number of records remaining after removing rows with missing data\n",
        "print(f\"\\nNumber of records after removing missing data: {len(df_tri_clean)}\")"
      ],
      "id": "3b3e7632",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There were no missing coordinates or missing air release data.\n",
        "\n",
        "Now we can create a spatial object with **GeoPandas** using the latitude and longitude coordinates in the table.\n"
      ],
      "id": "e40fccfc"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the cleaned TRI data to a GeoDataFrame for spatial analysis\n",
        "# Creates a 'geometry' column based on longitude and latitude, setting the coordinate system to EPSG:4326 (WGS 84)\n",
        "gdf_tri_form_r = gpd.GeoDataFrame(\n",
        "    df_tri_clean, \n",
        "    geometry=gpd.points_from_xy(df_tri_clean[lon_col], df_tri_clean[lat_col]),\n",
        "    crs=\"EPSG:4326\"\n",
        ")"
      ],
      "id": "34220c97",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let’s check the distribution of the air release data for outliers.\n"
      ],
      "id": "b9c0519c"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Plot a histogram to visualize the distribution of total air release values across facilities\n",
        "plt.figure(figsize=(8, 8))  # Define plot size\n",
        "plt.hist(gdf_tri_form_r['AIR_TOTAL_RELEASE'], bins=10, edgecolor='black') # Set bin count and edge color\n",
        "plt.title('Distribution of Air Total Release Values') # Plot title\n",
        "plt.xlabel('Air Total Release Sum') # X-axis label for air release values\n",
        "plt.ylabel('Frequency') # Y-axis label for frequency of values\n",
        "plt.grid(True, alpha=0.3) # Add grid with slight transparency for readability\n",
        "\n",
        "# Display the histogram\n",
        "plt.show()"
      ],
      "id": "2b9bada3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Looks like there are outliers around 100,000 lbs and 400,000 lbs. Before we map the data, it would be a good idea to perform a log transformation of `AIR_TOTAL_RELEASE` to smooth out the visuals. You can't take the log of negative or zero values, so we'll use the `log1p` function, which adds 1 to every value before taking the log.\n"
      ],
      "id": "971901e2"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import numpy as np\n",
        "\n",
        "# Calculate the natural logarithm of air release values +1 to handle any zero values safely\n",
        "# This transformation helps normalize the data, reducing skew for clearer visualization\n",
        "gdf_tri_form_r['LOG_AIR_RELEASE'] = np.log1p(gdf_tri_form_r['AIR_TOTAL_RELEASE'])\n",
        "\n",
        "# Plot a histogram to visualize the distribution of the log-transformed air release values\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.hist(gdf_tri_form_r['LOG_AIR_RELEASE'], bins=10, edgecolor='black') # Set bin count and edge color\n",
        "plt.title('Distribution of Log-Transformed Air Release Values') # Plot title\n",
        "plt.xlabel('Log of Air Total Release Sum') # X-axis label for log air release values\n",
        "plt.ylabel('Frequency') # Y-axis label for frequency of values\n",
        "plt.grid(True, alpha=0.3) # Add grid with slight transparency for readability\n",
        "\n",
        "# Display the histogram\n",
        "plt.show()"
      ],
      "id": "4c3953a1",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "That looks much better.\n",
        "\n",
        "Now we can visualize the data, by overlaying the points on a basemap of Detroit while using graduated symbols to illustrate the amount of air releases for a given facility.\n"
      ],
      "id": "f812c76b"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Reproject TRI facility data to Web Mercator (EPSG:3857) to align with the basemap projection\n",
        "gdf_tri_form_r_bm = gdf_tri_form_r.to_crs(epsg=3857)\n",
        "\n",
        "# Create the map plot\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Plot the Detroit metro area boundary and bounding box with customized colors\n",
        "detroit_metro_bm.plot(ax=ax, facecolor='none', edgecolor='blue', linewidth=2)\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=2)\n",
        "\n",
        "# Scatter plot TRI facilities with symbol sizes proportional to the log of air releases\n",
        "# Scaling factor applied for better visibility; adjust as needed\n",
        "scatter = ax.scatter(gdf_tri_form_r_bm.geometry.x, gdf_tri_form_r_bm.geometry.y, \n",
        "                     s=gdf_tri_form_r_bm['LOG_AIR_RELEASE']*20,  # Scale size of symbols by log air releases\n",
        "                     c='orangered',  # Color fill of symbols\n",
        "                     edgecolor='yellow',  # Outline color of symbols\n",
        "                     linewidth=1,  # Outline thickness for clarity\n",
        "                     alpha=0.7) # Adjust transparency level\n",
        "\n",
        "# Add OpenStreetMap basemap for spatial context\n",
        "ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Set map view to focus on the bounding box area\n",
        "ax.set_xlim(bbox_polygon_bm.total_bounds[0], bbox_polygon_bm.total_bounds[2])\n",
        "ax.set_ylim(bbox_polygon_bm.total_bounds[1], bbox_polygon_bm.total_bounds[3])\n",
        "\n",
        "# Remove axes for a cleaner map view\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Add a legend with sample sizes for symbol scaling reference\n",
        "legend_sizes = [0,4,8,12]  # Adjust sample sizes based on data scale\n",
        "legend_elements = [plt.scatter([], [], s=size*20, c='orangered', edgecolor='yellow', \n",
        "                               linewidth=1, alpha=1, label=f'{size:,}') \n",
        "                   for size in legend_sizes]\n",
        "ax.legend(handles=legend_elements, title='Log Total Air Releases (lbs)', \n",
        "          loc='lower right', title_fontsize=12, fontsize=10) # Customize legend title and text\n",
        "\n",
        "# Set the plot title\n",
        "plt.title(\"Detroit Metro Area TRI Facilities - Total Air Releases (Custom Data)\", fontsize=16)\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Output summary statistics for TRI facilities plotted and air release values\n",
        "print(f\"\\nNumber of TRI facilities plotted: {len(gdf_tri_form_r)}\")\n",
        "print(f\"Total air releases: {gdf_tri_form_r[release_col].sum():,.2f} lbs\")\n",
        "print(f\"Average air release per facility: {gdf_tri_form_r[release_col].mean():,.2f} lbs\")"
      ],
      "id": "3008c210",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The point source air release data provides a good snapshot of the locations and relative amounts of chemical releases. With some basic understanding of the demographic dynamics of Detroit one might theorize who is being subjected to high levels of pollution, however, we’ll need additional data to develop any type of systematic analysis.\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "What key information does Form R of the Toxic Release Inventory provide?\n",
        "\n",
        "a)  Facility names and locations only\n",
        "b)  Chemical storage capacities of facilities\n",
        "c)  Detailed data on chemical releases, including air releases\n",
        "d)  Employee health records related to chemical exposure\n",
        ":::\n",
        "\n",
        "### Social Vulnerability Index\n",
        "\n",
        "NASA's [U.S. Social Vulnerability Index Grids (SVI), v1.01](https://sedac.ciesin.columbia.edu/data/set/usgrid-us-social-vulnerability-index-rev01) raster dataset, available through the NASA [Socioeconomic Data and Applications Center (SEDAC)](https://earthdata.nasa.gov/centers/sedac-daac), is a high-resolution geospatial resource that quantifies social vulnerability across the United States. This dataset is based on the CDC's Social Vulnerability Index but is presented in a raster format, providing continuous coverage at a 250-meter resolution. The SVI incorporates various socioeconomic and demographic factors such as poverty, lack of vehicle access, crowded housing, and minority status to assess communities' capacity to prepare for, respond to, and recover from hazards, including environmental threats like poor air quality.\n",
        "\n",
        "The raster format allows for more detailed spatial analysis and integration with other environmental datasets. This makes it particularly valuable for researchers and policymakers studying the intersection of social vulnerability and environmental risks, such as air pollution exposure. By overlaying this SVI data with air quality information, for instance, analysts can identify areas where socially vulnerable populations may be disproportionately affected by poor air quality, supporting environmental justice initiatives and targeted intervention strategies.\n",
        "\n",
        "As part of NASA’s [Earth Observing System Data and Information System (EOSDIS)](https://www.earthdata.nasa.gov/about/esdis/eosdis), SEDAC hosts this dataset along with other socioeconomic and environmental data, facilitating interdisciplinary research on human-environment interactions. The SVI raster dataset’s high resolution and comprehensive coverage make it a powerful tool for assessing environmental equity and informing policy decisions at various geographic scales.\n",
        "\n",
        "The SVI dataset is free to users with a NASA Earth Data account. The Earth Data account is free and gives you access to all SEDAC data as well as a wide range of NASA Earth science data. It’s a valuable resource for researchers, students, and anyone interested in environmental and socioeconomic data.\n",
        "\n",
        "If you don’t already have an Earth Data account you can follow these steps to download the SVI dataset on your local computer:\n",
        "\n",
        "1.  Visit the [NASA Earthdata website](https://urs.earthdata.nasa.gov/). Click on the “Register” button on the top right corner of the page.\n",
        "2.  Fill out the registration form with the required information, including your name, email address, and a password. Create a username.\n",
        "3.  Verify your email. NASA will send a verification email to the address you provided. Click on the link in this email to confirm your account.\n",
        "4.  Once your account is verified, log in to Earth Data using your username and password.\n",
        "5.  Visit the [SEDAC](https://sedac.ciesin.columbia.edu/) website and search for the U.S. Social Vulnerability Index Grids, v1.01 dataset by entering “SVI” in the search bar.\n",
        "6.  Download the 2020 WGS84 SVI grids; enter your Earth Data credentials to log in when prompted.\n",
        "\n",
        "#### Data Processing\n",
        "\n",
        "Once you’ve downloaded the dataset to your working directory, you can proceed with the analysis.\n",
        "\n",
        "The different layers of SVI are provided as individual files, but sometimes it's easier to work with a multilayer object. We can create one using **xarray**. To begin, we'll read in each file individually, clip it to our border of Detroit metro, and create an individual data array.\n"
      ],
      "id": "3c1e068d"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import xarray as xr\n",
        "import rasterio\n",
        "import rasterio.mask\n",
        "\n",
        "# Set xarray display options for enhanced attribute visibility when viewing data\n",
        "xr.set_options(\n",
        "    keep_attrs=True,  # Retain attributes in computations\n",
        "    display_expand_attrs=True, # Expand attribute display\n",
        "    display_expand_coords=True, # Show all coordinates\n",
        "    display_expand_data=False, # Collapse data display to save space\n",
        "    display_expand_data_vars=True # Expand variable display\n",
        ")\n",
        "\n",
        "# Define paths to SVI (Social Vulnerability Index) TIF files, each representing a different aspect of vulnerability\n",
        "tif_files = [\n",
        "    \"data/svi/svi_2020_tract_overall_wgs84.tif\",\n",
        "    \"data/svi/svi_2020_tract_minority_wgs84.tif\",\n",
        "    \"data/svi/svi_2020_tract_socioeconomic_wgs84.tif\",\n",
        "    \"data/svi/svi_2020_tract_housing_wgs84.tif\",\n",
        "    \"data/svi/svi_2020_tract_household_wgs84.tif\"\n",
        "]\n",
        "\n",
        "# Initialize an empty list to store DataArrays created from each raster\n",
        "data_arrays = []\n",
        "\n",
        "# Iterate over each TIF file to read, clip to Detroit metro area, and convert to xarray DataArray\n",
        "for file in tif_files:\n",
        "    with rasterio.open(file) as src:\n",
        "        # Align Detroit metro boundary to raster CRS for compatibility\n",
        "        metro_reprojected = detroit_metro.to_crs(src.crs)\n",
        "        \n",
        "        # Clip the raster to the Detroit metro area based on its geometry\n",
        "        out_image, out_transform = rasterio.mask.mask(src, metro_reprojected.geometry, crop=True)\n",
        "        out_meta = src.meta.copy() # Copy metadata for adjustments\n",
        "        \n",
        "        # Update metadata with new dimensions and transformation details after cropping\n",
        "        out_meta.update({\"driver\": \"GTiff\",\n",
        "                         \"height\": out_image.shape[1],\n",
        "                         \"width\": out_image.shape[2],\n",
        "                         \"transform\": out_transform})\n",
        "        \n",
        "        # Set up mesh grid coordinates to match the raster's new dimensions\n",
        "        height, width = out_meta['height'], out_meta['width']\n",
        "        cols, rows = np.meshgrid(np.arange(width), np.arange(height))\n",
        "        xs, ys = rasterio.transform.xy(out_transform, rows, cols)\n",
        "        \n",
        "        # Convert lists to numpy arrays and reshape to match the raster dimensions\n",
        "        xs, ys = np.array(xs).reshape(height, width), np.array(ys).reshape(height, width)\n",
        "        \n",
        "        # Create an xarray DataArray with coordinates for each cell based on the raster grid\n",
        "        da = xr.DataArray(out_image[0],  # Extract the first band of the clipped image\n",
        "                          coords={'y': ('y', ys[:, 0]),  # Latitude\n",
        "                                  'x': ('x', xs[0, :])}, # Longitude\n",
        "                          dims=['y', 'x'])\n",
        "        da.attrs['crs'] = str(src.crs)  # Store CRS as an attribute in string format\n",
        "        da.attrs['transform'] = out_transform # Add transform details to attributes\n",
        "        data_arrays.append(da) # Append DataArray to list"
      ],
      "id": "e6506646",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can combine them together and give the layers \"pretty\" names.\n"
      ],
      "id": "a1b25c22"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Combine all DataArrays from different SVI files into a single xarray DataSet\n",
        "svi_detroit = xr.concat(data_arrays, dim='layer')\n",
        "\n",
        "# Define and assign layer names to clarify SVI aspect for each data layer in the combined dataset\n",
        "layer_names = ['Overall', 'Minority', 'Socioeconomic', 'Housing', 'Household']\n",
        "svi_detroit = svi_detroit.assign_coords(layer=('layer', layer_names))\n",
        "svi_detroit # Display the DataSet for verification"
      ],
      "id": "a25761ae",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can plot each layer.\n"
      ],
      "id": "fce852cb"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define minimum and maximum values for the color scale; SVI ranges from 0 (low) to 1 (high vulnerability)\n",
        "vmin, vmax = 0, 1\n",
        "\n",
        "# Create a multi-panel plot with subplots arranged in a 3x2 grid\n",
        "fig, axes = plt.subplots(3, 2, figsize=(8, 10))\n",
        "axes = axes.flatten()  # Flatten the 2D array of axes for easier iteration\n",
        "\n",
        "# Plot each SVI layer in a separate subplot\n",
        "for i, layer in enumerate(svi_detroit.layer.values):\n",
        "    # Render the layer with a plasma colormap and fixed color range for consistency\n",
        "    im = svi_detroit.sel(layer=layer).plot(ax=axes[i], add_colorbar=False, vmin=vmin, vmax=vmax, cmap='plasma')\n",
        "    axes[i].set_title(layer) # Title each subplot with the SVI aspect\n",
        "    axes[i].axis('off') # Turn off axis ticks for cleaner appearance\n",
        "    \n",
        "    # Overlay Detroit metro boundary on each subplot\n",
        "    metro_reprojected.boundary.plot(ax=axes[i], color='red', linewidth=1)\n",
        "\n",
        "# Remove the unused subplot in the 3x2 grid layout\n",
        "fig.delaxes(axes[5])\n",
        "\n",
        "# Add a single shared colorbar to indicate the SVI score scale across all subplots\n",
        "cbar_ax = fig.add_axes([0.92, 0.15, 0.02, 0.7]) # Position colorbar at right side of plot\n",
        "cbar = fig.colorbar(im, cax=cbar_ax, label='SVI Score', \n",
        "                    fraction=0.047, pad=0.04, aspect=20) # Customize colorbar properties\n",
        "\n",
        "plt.tight_layout()  # Adjust layout for clarity and spacing\n",
        "plt.show() # Display the final plot"
      ],
      "id": "ccd0fd86",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This provides an excellent look at demographic trends in the Detroit metro area. Overall vulnerability is highest in the inner city. The most striking drivers are the concentration of minorities and socioeconomic vulnerability in the downtown area. The housing and household components are slightly more varied throughout the region.\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "What does the Social Vulnerability Index (SVI) primarily measure?\n",
        "\n",
        "a)  Air pollution levels in different communities\n",
        "b)  Economic growth rates of different regions\n",
        "c)  Communities' capacity to prepare for and respond to hazards\n",
        "d)  Population density in urban areas\n",
        ":::\n",
        "\n",
        "### Integrating TRI and SVI\n",
        "\n",
        "There are numerous ways to assess the relationships between the SVI and TRI data. For this lesson we’ll identify areas where air releases and vulnerability are highest together by creating a new rasterized index that combines both layers.\n",
        "\n",
        "To begin we need to rasterize the air release point data. We will create an empty raster grid and “fill” the cells with the sum of the air release values from any points that are overlapped by the cells.\n",
        "\n",
        "Start by getting the bounds from our boundary object, specify the desired resolution, and set up the transform.\n"
      ],
      "id": "d82e0aa0"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from rasterio.transform import from_origin\n",
        "\n",
        "# Retrieve the spatial boundaries (min/max x and y coordinates) of the Detroit metro area\n",
        "minx, miny, maxx, maxy = detroit_metro_bm.total_bounds\n",
        "\n",
        "# Define the resolution of each cell in meters (5,000 meters = 5km) to align with SVI data resolution\n",
        "resolution = 5000\n",
        "\n",
        "# Calculate the number of cells needed to cover the Detroit metro area extent\n",
        "# in both x (width) and y (height) directions based on the resolution\n",
        "nx = int((maxx - minx) / resolution) # Number of cells horizontally\n",
        "ny = int((maxy - miny) / resolution) # Number of cells vertically\n",
        "\n",
        "# Create the transformation for converting between the spatial extent and pixel grid of the raster\n",
        "transform = from_origin(minx, maxy, resolution, resolution)"
      ],
      "id": "1804f5f7",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we have to prepare the point values and rasterize them. A key thing to note here is the `merge_alg=rasterio.enums.MergeAlg.add` argument passed to the rasterize function. This argument makes sure that cells with more than 1 point will add the values together (the default is to simply replace).\n"
      ],
      "id": "8f153d99"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from rasterio import features\n",
        "\n",
        "# Prepare the geometries (points) and their associated values for rasterization\n",
        "# Each tuple contains a geometry (location) and the AIR_TOTAL_RELEASE value for raster cell assignment\n",
        "shapes = ((geom, value) for geom, value in zip(gdf_tri_form_r_bm.geometry, gdf_tri_form_r_bm.AIR_TOTAL_RELEASE))\n",
        "\n",
        "# Rasterize the point data into a grid, with each cell reflecting the air release values\n",
        "air_release_raster = features.rasterize(\n",
        "    shapes=shapes, # Geometries and values to rasterize\n",
        "    out_shape=(ny, nx), # Output raster shape, matching previously calculated cell count\n",
        "    transform=transform, # Transformation matrix for raster alignment with spatial extent\n",
        "    fill=0,  # Default fill value (0) for cells without data\n",
        "    all_touched=True, # Includes all cells touched by a geometry to ensure full coverage\n",
        "    merge_alg=rasterio.enums.MergeAlg.add) # Merge algorithm to sum overlapping values in cells"
      ],
      "id": "88a92ef9",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We rasterized with **rasterio**, but we need to convert this to an **xarray** object to continue and we need **rioxarray** to write the crs information for the **xarray** object.\n"
      ],
      "id": "81973be9"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import rioxarray\n",
        "\n",
        "# Convert the rasterized array to an xarray DataArray for enhanced data handling\n",
        "# Set coordinates to spatial extent using latitude (y) and longitude (x) for geospatial consistency\n",
        "air_release_raster_da = xr.DataArray(\n",
        "    air_release_raster, # Rasterized air release data\n",
        "    coords={'y': np.linspace(maxy, miny, ny), # Y-axis coordinates, from top to bottom\n",
        "        'x': np.linspace(minx, maxx, nx)}, # X-axis coordinates, from left to right\n",
        "    dims=['y', 'x']) # Dimension labels for DataArray\n",
        "\n",
        "# Write the Detroit metro area CRS to the DataArray for georeferencing\n",
        "air_release_raster_da.rio.write_crs(detroit_metro_bm.crs, inplace=True)"
      ],
      "id": "9b844e07",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The spaces between county borders can create odd values because there are no points right next to each other. We can clip the raster to our Detroit metro boundary to fix this.\n"
      ],
      "id": "e71e9dc9"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Clip the rasterized DataArray to the Detroit metro boundary to limit data to the target area\n",
        "air_release_raster_da = air_release_raster_da.rio.clip(\n",
        "    detroit_metro_bm.geometry.values,  # Boundary geometry for clipping\n",
        "    detroit_metro_bm.crs,  # Coordinate reference system of the boundary\n",
        "    drop=False,   # Retain original raster size; cells outside geometry are masked\n",
        "    all_touched=True)  # Include any cells touched by the boundary for full coverage"
      ],
      "id": "a49c1c71",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can see how our rasterized air release values look.\n"
      ],
      "id": "0fe11fa4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from matplotlib.colors import BoundaryNorm, ListedColormap\n",
        "\n",
        "# Define class intervals (breaks) for categorizing air release values into discrete color bands\n",
        "breaks = [0, 1, 10, 100, 1000, 10000, 100000, 250000, 500000]\n",
        "\n",
        "# Create a custom colormap to represent different levels of air release with distinct colors\n",
        "colors = ['#fffff3', '#FFFFCC', '#FFEDA0', '#FED976', '#FEB24C', '#FD8D3C', '#FC4E2A', '#E31A1C', '#B10026']\n",
        "cmap = ListedColormap(colors)\n",
        "\n",
        "# Create a normalization scheme to map values to the defined color breaks\n",
        "norm = BoundaryNorm(breaks, cmap.N)\n",
        "\n",
        "# Set up the plot with a defined size\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Plot TRI facility points in blue, with adjusted transparency (alpha) for visibility\n",
        "gdf_tri_form_r_bm.plot(ax=ax, color='blue', markersize=10, alpha=0.7)\n",
        "\n",
        "# Plot the air release raster, using the custom colormap and normalization scheme\n",
        "im = ax.imshow(\n",
        "    air_release_raster_da,  # Raster data for air release\n",
        "    extent=[minx, maxx, miny, maxy],  # Spatial extent to match the Detroit metro bounds\n",
        "    origin='upper',  # Ensure raster is oriented correctly (top-down)\n",
        "    cmap=cmap,  # Apply custom colormap\n",
        "    norm=norm)  # Normalize color according to defined breaks\n",
        "\n",
        "# Add a color bar with labels indicating air release levels (discrete)\n",
        "cbar = plt.colorbar(\n",
        "    im, ax=ax, extend='max',  # Display max extension on color bar\n",
        "    label='Total Air Releases (pounds)',  # Label for color bar\n",
        "    ticks=breaks,  # Set breaks as tick positions\n",
        "    fraction=0.047, pad=0.04, aspect=20)  # Adjust color bar size and position\n",
        "cbar.ax.set_yticklabels([f'{b:,}' for b in breaks])  # Format tick labels with thousands separator\n",
        "\n",
        "# Plot Detroit metro area boundary as a black outline\n",
        "detroit_metro_bm.boundary.plot(ax=ax, color='black', linewidth=2)\n",
        "\n",
        "# Set plot extent to focus on Detroit metro area\n",
        "ax.set_xlim(minx, maxx)\n",
        "ax.set_ylim(miny, maxy)\n",
        "\n",
        "# Add a title and axis labels\n",
        "\n",
        "ax.set_title('TRI Air Total Release (100m resolution sum) with Facility Locations', fontsize=16)\n",
        "ax.set_xlabel('X Coordinate')\n",
        "ax.set_ylabel('Y Coordinate')\n",
        "\n",
        "# Optimize layout and display the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Output summary information\n",
        "print(f\"Number of TRI facilities plotted: {len(gdf_tri_form_r_bm)}\")\n",
        "print(f\"Total air releases: {gdf_tri_form_r_bm['AIR_TOTAL_RELEASE'].sum():,.2f}\")\n",
        "print(f\"Maximum cell value in raster: {air_release_raster_da.max().values:,.2f}\")"
      ],
      "id": "51151e5e",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The raster looks as expected with the highest values in locations where our previous map had the largest circles. We overlaid the points to see the sources of the values. The cell resolution is set to 5000m. This was selected so we can evaluate the map and the process we used to create the raster, but any value could be set depending on the intended use.\n",
        "\n",
        "#### Air Release Vulnerability Index\n",
        "\n",
        "Now that we've created our air release raster layer we can combine it with the SVI raster to create a new index identifying areas with the highest combination of air releases and vulnerability.\n",
        "\n",
        "Start by selecting just the SVI Overall layer and converting it to a **rioxarray** object so we can perform raster calculations. We'll also ensure the 2 raster layers \"line up\" by reprojecting into the same coordinate reference system and matching their resolutions.\n"
      ],
      "id": "c03ea92e"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from rasterio.enums import Resampling\n",
        "\n",
        "# Select the 'Overall' layer from the Social Vulnerability Index (SVI) dataset specific to Detroit\n",
        "svi_overall = svi_detroit.sel(layer='Overall')\n",
        "\n",
        "# Convert SVI layer to rioxarray for CRS-based geospatial operations\n",
        "svi_overall = svi_overall.rio.write_crs(\"EPSG:4326\")\n",
        "\n",
        "# Reproject the SVI layer to match the CRS and spatial alignment of the air release raster\n",
        "svi_reprojected = svi_overall.rio.reproject_match(air_release_raster_da)\n",
        "\n",
        "# Adjust the resolution of air release data to match the SVI resolution using bilinear resampling\n",
        "air_release_disaggregated = svi_reprojected.rio.reproject_match(\n",
        "    svi_reprojected,\n",
        "    resampling=Resampling.bilinear\n",
        ")"
      ],
      "id": "6fbf3936",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There are a few more steps that will aid in interpretation.\n",
        "\n",
        "1.  We will take the log of the air release data to reduce the impact of major outliers.\n",
        "2.  We will scale the logged air release data to 0-1 to match the SVI data. Now the resulting index we create will have equal contributions from both datasets.\n"
      ],
      "id": "bc2de4f4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Apply log transformation (log1p) to compress wide-ranging air release values, then normalize to a 0-1 scale\n",
        "air_release_log = np.log1p(air_release_disaggregated)\n",
        "air_release_scaled = (air_release_log - air_release_log.min()) / (air_release_log.max() - air_release_log.min())\n",
        "\n",
        "\n",
        "# Multiply the scaled air release data with the reprojected SVI data to create a vulnerability indicator\n",
        "# This indicator combines environmental impact (air releases) with social vulnerability\n",
        "vulnerability_indicator = air_release_scaled * svi_reprojected"
      ],
      "id": "e69c0b01",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's visualize our index along with the inputs.\n"
      ],
      "id": "49dd4660"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Create a figure with 3 subplots stacked vertically for SVI, Air Release, and Vulnerability Indicator\n",
        "fig, axs = plt.subplots(3, 1, figsize=(8, 20))\n",
        "\n",
        "# Plot SVI Overall data with a color map (viridis), normalized between 0 and 1\n",
        "im1 = svi_reprojected.plot(ax=axs[0], cmap='viridis', vmin=0, vmax=1, add_colorbar=False)\n",
        "plt.colorbar(im1, ax=axs[0], label='SVI Overall', \n",
        "                    fraction=0.047, pad=0.04, aspect=20)\n",
        "axs[0].set_title('Social Vulnerability Index (Overall)', fontsize=16)\n",
        "detroit_metro.boundary.plot(ax=axs[0], color='black', linewidth=2) # Add Detroit metro boundary\n",
        "\n",
        "# Plot the log-transformed Air Release data for enhanced visual differentiation\n",
        "im2 = np.log1p(air_release_disaggregated).plot(ax=axs[1], cmap='YlOrRd', add_colorbar=False)\n",
        "plt.colorbar(im2, ax=axs[1], label='Log(Air Release + 1)', \n",
        "                    fraction=0.047, pad=0.04, aspect=20)\n",
        "axs[1].set_title('Air Release (Log-transformed)', fontsize=16)\n",
        "detroit_metro.boundary.plot(ax=axs[1], color='black', linewidth=2)\n",
        "\n",
        "# Plot the Vulnerability Indicator as the product of SVI and scaled air release data\n",
        "im3 = vulnerability_indicator.plot(ax=axs[2], cmap='YlOrRd', vmin=0, vmax=1, add_colorbar=False)\n",
        "plt.colorbar(im3, ax=axs[2], label='Air Release Vulnerability Indicator', \n",
        "                    fraction=0.047, pad=0.04, aspect=20)\n",
        "axs[2].set_title('Air Release Vulnerability Indicator\\n(Scaled Air Release * SVI)', fontsize=16)\n",
        "detroit_metro.boundary.plot(ax=axs[2], color='black', linewidth=2)\n",
        "\n",
        "# Set consistent axis limits and labels across all subplots\n",
        "for ax in axs:\n",
        "    ax.set_xlabel('Longitude')\n",
        "    ax.set_ylabel('Latitude')\n",
        "    ax.set_xlim(svi_reprojected.x.min(), svi_reprojected.x.max())\n",
        "    ax.set_ylim(svi_reprojected.y.min(), svi_reprojected.y.max())\n",
        "\n",
        "# Adjust layout and display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Print out the maximum value of the vulnerability indicator for reference\n",
        "print(f\"Maximum vulnerability indicator: {vulnerability_indicator.max().values:.4f}\")"
      ],
      "id": "4ac49562",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As one would expect, there are several areas in the downtown river corridor with very high combinations of SVI and TRI releases. The maximum score for our index is 0.9286! That is an extremely vulnerable community facing extraordinary levels of air pollution.\n",
        "\n",
        "We can get a better sense of where these communities are by extracting the location of the cells with the highest scores and placing them on a basemap of Detroit.\n",
        "\n",
        "We'll convert the raster into a data frame, sort the values in descending order, and then extract the coordinates.\n"
      ],
      "id": "f6ff16cb"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert vulnerability indicator to a DataFrame for easy sorting and manipulation\n",
        "vulnerability_df = vulnerability_indicator.to_dataframe(name='index').reset_index()\n",
        "\n",
        "# Sort by index value (descending) to identify top 10 most vulnerable areas\n",
        "top_10 = vulnerability_df.sort_values('index', ascending=False).head(10)\n",
        "\n",
        "# Create GeoDataFrame from the top 10 points using coordinates as geometry\n",
        "top_10['geometry'] = gpd.points_from_xy(top_10.x, top_10.y)\n",
        "top_10_gdf = gpd.GeoDataFrame(top_10, geometry='geometry', crs=vulnerability_indicator.rio.crs)"
      ],
      "id": "3760f605",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's make a map.\n"
      ],
      "id": "b5994d8a"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Create the final map plot to display top 10 vulnerable areas with basemap and boundaries\n",
        "fig, ax = plt.subplots(figsize=(8, 8))\n",
        "\n",
        "# Plot Detroit metro area boundary in black\n",
        "detroit_metro_bm.boundary.plot(ax=ax, color='black', linewidth=2)\n",
        "\n",
        "# Plot the top 10 highest vulnerability points with blue markers\n",
        "top_10_gdf.plot(ax=ax, color='blue', markersize=100, alpha=0.7)\n",
        "\n",
        "# Annotate each top point with its rank for clarity\n",
        "for idx, row in top_10_gdf.iterrows():\n",
        "    ax.annotate(f\"#{idx+1}\", (row.geometry.x, row.geometry.y), \n",
        "                xytext=(3, 3), textcoords=\"offset points\", \n",
        "                color='black', fontweight='bold')\n",
        "\n",
        "# Add a basemap to provide geographic context, using OpenStreetMap tiles\n",
        "ctx.add_basemap(ax, crs=vulnerability_indicator.rio.crs.to_string(), source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Overlay a bounding box in blue as a border around the plotted area\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Limit map extent to focus on Detroit metro\n",
        "ax.set_xlim(vulnerability_indicator.x.min(), vulnerability_indicator.x.max())\n",
        "ax.set_ylim(vulnerability_indicator.y.min(), vulnerability_indicator.y.max())\n",
        "\n",
        "# Add title and remove axis display for a cleaner map\n",
        "ax.set_title('Top 10 Areas with Highest Air Release Vulnerability Index', fontsize=16)\n",
        "ax.set_axis_off()\n",
        "\n",
        "# Display the final map\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Output index and vulnerability value of the top 10 areas for review\n",
        "print(\"Row index and index value of the top 10 points:\")\n",
        "for i, (idx, row) in enumerate(top_10_gdf.iterrows(), 1):\n",
        "    print(f\"Row index = {idx}, Index value = {round(row['index'], 2)}\")"
      ],
      "id": "1653c164",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As the input layers suggest, the most vulnerable and polluted areas are along the river corridor in Lincoln Park, Melvindale, Mexicantown, and River Rouge. There are multiple locations with indexes greater than 0.70. This is valuable information for targeted relief or mitigation programs.\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "What was the main purpose of combining the TRI and SVI data in this analysis?\n",
        "\n",
        "a)  To calculate total pollution levels for each county\n",
        "b)  To identify areas with both high air releases and high social vulnerability\n",
        "c)  To determine the most populous areas in Detroit\n",
        "d)  To predict future industrial development zones\n",
        ":::\n",
        "\n",
        "### CDC PLACES\n",
        "\n",
        "The CDC [Population Level Analysis and Community Estimates (PLACES)](https://www.cdc.gov/places/index.html) dataset is a collaboration between the [Centers for Disease Control and Prevention (CDC)](https://www.cdc.gov/index.html), the [Robert Wood Johnson Foundation](https://www.rwjf.org/), and the [CDC Foundation](https://www.cdcfoundation.org/). It provides model-based population-level analysis and community estimates of health indicators for all counties, places (incorporated and census designated places), census tracts, and ZIP Code Tabulation Areas (ZCTAs) across the United States. Some key points to consider when working with CDC PLACES:\n",
        "\n",
        "1.  **Spatial Extent:** Entire United States, including all 50 states, the District of Columbia, and Puerto Rico.\n",
        "2.  **Spatial Resolution:** Multiple levels including counties, cities/towns, census tracts, and ZIP codes.\n",
        "3.  **Indicators:** Wide range of chronic disease measures related to health outcomes, prevention, and health risk behaviors.\n",
        "4.  **Data Sources:**\n",
        "    -   [Behavioral Risk Factor Surveillance System (BRFSS)](https://www.cdc.gov/brfss/index.html)\n",
        "    -   [U.S. Census Bureau's American Community Survey (ACS)](https://www.census.gov/programs-surveys/acs)\n",
        "5.  **[Methodology:](https://www.cdc.gov/places/methodology/index.html)** Uses small area estimation methods for small geographic areas.\n",
        "6.  **Health Measures Include:**\n",
        "    -   Chronic diseases: e.g., asthma, COPD, heart disease, diabetes\n",
        "    -   Health risk behaviors: e.g., smoking, physical inactivity, binge drinking\n",
        "    -   Prevention practices: e.g., health insurance coverage, dental visits, cholesterol screening\n",
        "7.  **Socioeconomic Data:** Includes some socioeconomic and demographic variables.\n",
        "8.  **Annual Updates:** Providing recent estimates for local areas.\n",
        "\n",
        "This dataset is valuable for public health researchers, policymakers, and community organizations. It provides a standardized way to compare health indicators across different geographic areas and can be used to inform targeted interventions and policy decisions, especially in addressing health disparities at a local level.\n",
        "\n",
        "As with ICIS-AIR and TRI, PLACES is available through a web search interface and a programmatic API. We will focus on the API in this example.\n",
        "\n",
        "#### Processing\n",
        "\n",
        "We'll start by accessing the CDC PLACES data through their API and processing it for our analysis. First we set up our API request. We specify the endpoint URL, define our target counties, and create a filter to select data only for these counties in Michigan.\n"
      ],
      "id": "3604aec8"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define the GeoJSON API endpoint URL, specific to CDC health data\n",
        "url = \"https://data.cdc.gov/resource/cwsq-ngmh.geojson\"\n",
        "\n",
        "# Define the list of Detroit metro area counties of interest\n",
        "detroit_counties = ['Wayne', 'Oakland', 'Macomb']\n",
        "\n",
        "# Construct the query filter string to specify only Detroit metro counties\n",
        "county_filter = \" OR \".join([f\"countyname = '{county}'\" for county in detroit_counties])\n",
        "\n",
        "# Define query parameters to retrieve data for Detroit metro counties in Michigan\n",
        "params = {\n",
        "    \"$where\": f\"stateabbr = 'MI' AND ({county_filter})\",\n",
        "    \"$limit\": 50000   # Limit the results to a manageable number\n",
        "}"
      ],
      "id": "c2075278",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next, let's make the API request and process the response:\n"
      ],
      "id": "6802dd7d"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Perform the API request to retrieve GeoJSON data for Detroit area health statistics\n",
        "response = requests.get(url, params=params)\n",
        "\n",
        "# Check if the request was successful\n",
        "if response.status_code == 200:\n",
        "    data = response.json() # Parse the JSON data\n",
        "    print(f\"Successfully retrieved data\")\n",
        "else:\n",
        "    print(f\"Failed to retrieve data. Status code: {response.status_code}\")\n",
        "    print(response.text)\n",
        "\n",
        "# Convert the GeoJSON text to a GeoDataFrame for spatial analysis\n",
        "gdf = gpd.read_file(response.text)"
      ],
      "id": "30b6dbb1",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can inspect the `gdf` return object on your own; it's too large to print here. Let's just take a look at the available health measures in CDC PLACES.\n"
      ],
      "id": "3c6ad160"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Display unique health measures available in the dataset to understand data variety\n",
        "print(\"\\nAvailable health measures:\")\n",
        "print(gdf['measure'].unique())\n",
        "\n",
        "# Print essential details of the GeoDataFrame structure and content\n",
        "print(\"\\nGeoDataFrame Info:\")\n",
        "print(gdf.info())"
      ],
      "id": "cea8fef3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There's a lot of great data here considering the rarity of health outcomes data; especially at this resolution (census tracts). Because we’re dealing with air quality in this lesson we’ll use “Current asthma among adults.” The asthma data all comes from 2022 (you can verify with `gdf_asthma[‘year’].unique()`).\n",
        "\n",
        "Let's make a quick map to see how it looks.\n"
      ],
      "id": "c1a9ac3b"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Create a sample map to visualize data for a specific health measure, such as asthma prevalence\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Filter the data for the asthma measure and ensure numeric conversion of data values\n",
        "gdf_asthma = gdf[gdf['measure'] == 'Current asthma among adults'].copy()\n",
        "gdf_asthma['data_value'] = pd.to_numeric(gdf_asthma['data_value'], errors='coerce')\n",
        "\n",
        "# Plot asthma data with color variation based on prevalence\n",
        "gdf_asthma.plot(column='data_value', \n",
        "                ax=ax, \n",
        "                legend=True, \n",
        "                legend_kwds={'label': 'Asthma Prevalence (%)', 'orientation': 'horizontal'},\n",
        "                cmap='YlOrRd',\n",
        "                missing_kwds={'color': 'lightgrey'}) # Specify color for missing data\n",
        "\n",
        "# Add basemap for geographic context\n",
        "ctx.add_basemap(ax, crs=gdf_asthma.crs.to_string(), source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Overlay the Detroit metro area boundary\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Set map extent to focus on the Detroit metro area\n",
        "ax.set_xlim(gdf_asthma.total_bounds[0], gdf_asthma.total_bounds[2])\n",
        "ax.set_ylim(gdf_asthma.total_bounds[1], gdf_asthma.total_bounds[3])\n",
        "\n",
        "# Add map title and hide axis lines for a cleaner visualization\n",
        "plt.title('Asthma Prevalence in Detroit Metro Area', fontsize=16)\n",
        "ax.axis('off')\n",
        "\n",
        "# Adjust layout and show the map\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "id": "d644aa16",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you hadn't already noticed while inspecting the object we returned from the API, the census tract data are actually centroid points--not census tract boundaries.\n",
        "\n",
        "We created a choropleth map showing asthma prevalence across the Detroit metro area. With the `YlOrRd` colormap darker red indicates higher asthma prevalence.\n",
        "\n",
        "Finally, let's print some statistics about the asthma data:\n"
      ],
      "id": "e210fff4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Calculate and display key statistics for asthma prevalence in the dataset\n",
        "print(\"\\nAsthma Statistics:\")\n",
        "print(f\"Average asthma prevalence: {gdf_asthma['data_value'].mean():.2f}%\")\n",
        "print(f\"Minimum asthma prevalence: {gdf_asthma['data_value'].min():.2f}%\")\n",
        "print(f\"Maximum asthma prevalence: {gdf_asthma['data_value'].max():.2f}%\")\n",
        "\n",
        "# Count and display the number of census tracts in each county to understand data distribution\n",
        "print(\"\\nNumber of census tracts per county:\")\n",
        "print(gdf_asthma['countyname'].value_counts())"
      ],
      "id": "1f59747e",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Wow. Some census tracts along the river front and central downtown area have adult asthma prevalence of nearly 20%! I’m not an epidemiologist, but that seems extremely high; especially when you compare to the outer suburbs and more affluent areas in the Royal Oak-Ferndale area. However, in light of our exploration of TRI and ICIS-AIR data it might not be surprising to see elevated asthma rates in these areas.\n",
        "\n",
        "Given that we've mostly been working with and creating raster data in the previous sections, it might be easier to draw systematic comparisons between pollution and health outcomes if the CDC data is also in a raster format.\n",
        "\n",
        "#### Interpolate a Surface w/ IDW\n",
        "\n",
        "While the choropleth map provides a good overview, it can be affected by the varying sizes of census tracts. To get a smoother representation of the data that resembles our air quality data, we can use Inverse Distance Weighting (IDW) interpolation to create a continuous surface.\n",
        "\n",
        "First, let’s prepare our data for interpolation by reprojecting to the basemap projection, extracting the coordinates and the asthma value, and masking the data for any asthma values that may be NA.\n"
      ],
      "id": "fc6c195e"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from scipy.interpolate import griddata\n",
        "from rasterio.transform import from_origin\n",
        "from rasterio.warp import transform_bounds\n",
        "\n",
        "# Ensure the GeoDataFrame for asthma data is in EPSG:4326 (latitude and longitude)\n",
        "gdf_asthma = gdf_asthma.to_crs(epsg=4326)\n",
        "\n",
        "# Extract longitude (X), latitude (Y), and asthma prevalence values (Z)\n",
        "X = gdf_asthma.geometry.x.values\n",
        "Y = gdf_asthma.geometry.y.values\n",
        "Z = gdf_asthma['data_value'].values\n",
        "\n",
        "# Mask out any NaN values in the prevalence data to ensure valid input for interpolation\n",
        "mask = ~np.isnan(Z)\n",
        "X, Y, Z = X[mask], Y[mask], Z[mask]"
      ],
      "id": "e1845cc3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "::: column-margin\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Data Science Review\n",
        "\n",
        "Inverse Distance Weighting (IDW) is a spatial interpolation method used to estimate values at unknown locations based on known values at nearby points. The fundamental principle of IDW is that points closer to the location being estimated have more influence than those farther away. IDW calculates the estimated value as a weighted average of nearby known values, with weights inversely proportional to the distance between the known point and the estimation location. IDW is popular in geographic information systems (GIS) and environmental sciences due to its simplicity and effectiveness in creating continuous surfaces from point data, such as elevation, temperature, or pollution levels. While IDW has advantages like preserving local variation and working well with evenly distributed points, it also has limitations; including potential \"bull's-eye\" patterns around data points, inability to account for spatial trends or barriers, and sensitivity to outliers and clustering of input points.\n",
        ":::\n",
        ":::\n",
        "\n",
        "Now let’s create the structure of the raster we’ll create. This is based on our desired resolution and the extent of our data.\n"
      ],
      "id": "10337800"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Define grid properties to perform interpolation over a defined spatial extent and resolution\n",
        "\n",
        "# Set the grid resolution in degrees for interpolation\n",
        "grid_resolution = 0.025\n",
        "# Define the spatial bounds (extent) of the grid based on asthma data coverage\n",
        "x_min, y_min, x_max, y_max = gdf_asthma.total_bounds\n",
        "\n",
        "# Generate equally spaced coordinates based on grid resolution for X and Y dimensions\n",
        "grid_x = np.arange(x_min, x_max, grid_resolution)\n",
        "grid_y = np.arange(y_min, y_max, grid_resolution)\n",
        "\n",
        "# Create a 2D grid of coordinates by meshing grid_x and grid_y\n",
        "grid_xx, grid_yy = np.meshgrid(grid_x, grid_y)"
      ],
      "id": "ea1f0b7f",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we're ready to interpolate the surface.\n"
      ],
      "id": "2a26aca4"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Perform interpolation using Inverse Distance Weighting (IDW)\n",
        "\n",
        "# Stack longitude and latitude into coordinate pairs for IDW interpolation\n",
        "points = np.column_stack((X, Y))\n",
        "\n",
        "# Use linear interpolation to estimate asthma prevalence values on the grid\n",
        "grid_z = griddata(points, Z, (grid_xx, grid_yy), method='linear')"
      ],
      "id": "0ab67398",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "First we created a regular grid over our study area and then used the `griddata` function to perform linear interpolation (which is a form of IDW) on our asthma prevalence data.\n",
        "\n",
        "Now, let's convert our interpolated data into an xarray Dataset so it's easier to manipulate.\n"
      ],
      "id": "a9a3047f"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Convert the interpolated grid into an xarray Dataset for easier handling and export\n",
        "\n",
        "# Create an xarray dataset with interpolated asthma prevalence data, labeled by grid coordinates\n",
        "ds = xr.Dataset({\n",
        "    'asthma': (['y', 'x'], grid_z),  # 'asthma' is the interpolated variable over the grid\n",
        "    'x': grid_x,  # X-coordinates (longitude)\n",
        "    'y': grid_y  # Y-coordinates (latitude)\n",
        "})\n",
        "\n",
        "# Assign a coordinate reference system (CRS) to the dataset for spatial consistency\n",
        "ds.rio.write_crs(\"EPSG:4326\", inplace=True)"
      ],
      "id": "0e2ae1ab",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This step allows us to work with our interpolated data using **xarray**, which provides powerful tools for working with multi-dimensional arrays.\n",
        "\n",
        "Next, we'll clip our interpolated surface to the Detroit metro area. This ensures our data matches our other rasters. Then we'll project it to match basemaps so we can create a map of the data.\n"
      ],
      "id": "988181f2"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Clip the asthma dataset to the boundaries of the Detroit metro area\n",
        "# Ensures data outside of the metro area is excluded\n",
        "ds_clipped = ds.rio.clip(detroit_metro.geometry, drop=False)\n",
        "\n",
        "# Reproject the clipped dataset to Web Mercator (EPSG:3857) for basemap compatibility\n",
        "ds_3857 = ds_clipped.rio.reproject(\"EPSG:3857\")"
      ],
      "id": "cedd3b1d",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally, let's create a map of our interpolated asthma prevalence:\n"
      ],
      "id": "e2ce97fe"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Initialize the plot for visualizing interpolated asthma prevalence\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "\n",
        "# Display the interpolated asthma prevalence data, adjusting transparency and color map\n",
        "im = ds_3857.asthma.plot(ax=ax, cmap='YlOrRd', \n",
        "                         alpha=0.7, add_colorbar=False)\n",
        "\n",
        "# Add a colorbar with custom size and label for asthma prevalence\n",
        "cbar = plt.colorbar(im, ax=ax, label='Asthma Prevalence', \n",
        "                    fraction=0.047, pad=0.04, aspect=20)\n",
        "\n",
        "# Add a basemap for geographical context using Web Mercator projection\n",
        "ctx.add_basemap(ax, crs=ds_3857.rio.crs, \n",
        "                source=ctx.providers.OpenStreetMap.Mapnik)\n",
        "\n",
        "# Plot Detroit metro area boundary for clear region distinction\n",
        "bbox_polygon_bm.boundary.plot(ax=ax, color='#315c86', linewidth=3)\n",
        "\n",
        "# Set plot boundaries to focus on the Detroit metro area only\n",
        "bounds_3857 = transform_bounds(\"EPSG:4326\", \"EPSG:3857\", \n",
        "                               x_min, y_min, x_max, y_max)\n",
        "ax.set_xlim(bounds_3857[0], bounds_3857[2])\n",
        "ax.set_ylim(bounds_3857[1], bounds_3857[3])\n",
        "\n",
        "# Add title and remove default axis for clean presentation\n",
        "plt.title('IDW Interpolated Asthma Prevalence in Detroit Metro Area', fontsize=16)\n",
        "ax.axis('off')\n",
        "\n",
        "# Optimize layout\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "id": "a46d9754",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This code creates a map of the interpolated asthma prevalence, providing a smooth surface that shows the spatial variation in asthma rates across the Detroit metro area. The use of IDW interpolation helps to create a continuous surface from our discrete census tract data, potentially revealing patterns that might be less apparent in the choropleth map.\n",
        "\n",
        "By comparing this interpolated surface with our earlier maps of air pollution and social vulnerability, we can start to explore potential relationships between environmental factors, social conditions, and health outcomes in the Detroit metro area.\n",
        "\n",
        "We can start with a simple correlation analysis and then visualize it with a scatterplot. The simplest way to perform the analysis is to extract the raw value pairs and calculate the Pearson correlation.\n"
      ],
      "id": "36b427d5"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from scipy import stats\n",
        "\n",
        "# Reproject the air release data to align with the asthma dataset for comparison\n",
        "air_release_aligned = air_release_raster_da.rio.reproject_match(ds_clipped)\n",
        "\n",
        "# Flatten the raster arrays for correlation analysis\n",
        "air_release_flat = air_release_aligned.values.flatten()\n",
        "# Apply logarithmic transformation to air release data to reduce impact of extreme values\n",
        "air_release_flat = np.log1p(air_release_flat)\n",
        "asthma_flat = ds_clipped.asthma.values.flatten()\n",
        "\n",
        "# Mask to exclude NaN values for accurate correlation calculation\n",
        "mask = ~np.isnan(air_release_flat) & ~np.isnan(asthma_flat)\n",
        "\n",
        "# Compute Pearson correlation coefficient to measure association between air releases and asthma prevalence\n",
        "correlation, p_value = stats.pearsonr(air_release_flat[mask], asthma_flat[mask])\n",
        "print(f\"Correlation coefficient: {correlation}\")\n",
        "print(f\"P-value: {p_value}\")"
      ],
      "id": "05d58adb",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Creating a scatterplot of the paired values can provide a better look at what's going on.\n"
      ],
      "id": "ef752306"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Initialize the figure for scatter plot\n",
        "plt.figure(figsize=(7, 7))\n",
        "\n",
        "# Create a scatter plot comparing TRI Air Releases against Asthma Prevalence\n",
        "plt.scatter(air_release_flat[mask], asthma_flat[mask], alpha=0.5)\n",
        "\n",
        "# Label axes for clarity\n",
        "plt.xlabel('TRI Air Releases')\n",
        "plt.ylabel('Asthma Prevalence')\n",
        "\n",
        "# Add a descriptive title to convey the analysis purpose\n",
        "plt.title('TRI Air Releases vs Asthma Prevalence')\n",
        "\n",
        "# Display the scatter plot\n",
        "plt.show()"
      ],
      "id": "885ff03d",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This paints a clearer picture. Although there is a modest positive correlation, there are numerous cells with 0 TRI Air Release and a wide variety of Asthma prevalence values, which negates any correlation we might be seeing at higher release levels. This is a simple pixel by pixel comparison of matching values. We can perform a more robust analysis accounting for the spatial structure using a global autocorrelation analysis and Moran’s I.\n",
        "\n",
        "Pearson’s correlation and Moran’s I are both measures of association, but they serve different purposes and account for different aspects of data. Pearson’s correlation measures the linear relationship between two variables without considering their spatial context. It simply looks at how two variables change together, regardless of where the data points are located in space. \n",
        "\n",
        "In contrast, Moran’s I is specifically designed for spatial data and measures spatial autocorrelation – the degree to which a variable is correlated with itself across geographic space. Moran’s I takes into account both the values of observations and their spatial relationships, typically using a spatial weights matrix. \n",
        "\n",
        "While Pearson’s correlation can tell you if two variables are related overall, Moran’s I can reveal whether high or low values of a variable tend to cluster together in space, or if they’re randomly distributed. This makes Moran’s I particularly useful for analyzing geographical patterns and spatial dependencies, which are crucial in fields like environmental science, epidemiology, and urban planning.\n",
        "\n",
        "We can perform this analysis in Python using **pysal**. We already know our data is spatially aligned from the previous code chunks. We can create a spatial data frame and continue on.\n"
      ],
      "id": "74fc307e"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "from pysal.explore import esda\n",
        "from pysal.lib import weights\n",
        "import shapely\n",
        "\n",
        "# Convert aligned raster data to a GeoDataFrame for spatial analysis\n",
        "df = gpd.GeoDataFrame(\n",
        "    {\n",
        "        # Include air release and asthma prevalence data as columns\n",
        "        'air_release': air_release_aligned.values.flatten(),\n",
        "        'asthma': ds_clipped.asthma.values.flatten(),\n",
        "\n",
        "        # Create geometry points for each raster cell\n",
        "        'geometry': [\n",
        "            shapely.geometry.Point(x, y) \n",
        "            for x, y in zip(\n",
        "                np.repeat(air_release_aligned.x.values, len(air_release_aligned.y)),\n",
        "                np.tile(air_release_aligned.y.values, len(air_release_aligned.x))\n",
        "            )\n",
        "        ]\n",
        "    }\n",
        ")"
      ],
      "id": "80be5a4b",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "::: column-margin\n",
        "::: {.callout-tip style=\"color: #5a7a2b;\"}\n",
        "#### Statistics Review\n",
        "\n",
        "A p-value, or probability value, is a statistical concept used to determine the significance of results in hypothesis testing. It represents the probability of obtaining results at least as extreme as the observed results, assuming that the null hypothesis is true. In simpler terms, it measures the strength of evidence against the null hypothesis. P-values range from 0 to 1, with lower values indicating stronger evidence against the null hypothesis. Typically, a p-value below a predetermined threshold (often 0.05) is considered statistically significant, suggesting that the observed results are unlikely to have occurred by chance alone. However, it's important to note that while p-values can indicate the strength of evidence against the null hypothesis, they do not prove or disprove hypotheses, nor do they measure the size or importance of an effect. They should be interpreted in conjunction with other factors such as effect size, sample size, and practical significance.\n",
        ":::\n",
        ":::\n",
        "\n",
        "Check for missing values and create the spatial weights matrix that determines how wide of a net we want to cast as we search for a local dependence structure (you could change these values and see how it impacts the results). You can read more about settings for spatial autocorrelation and Moran's I analysis [here](https://pro.arcgis.com/en/pro-app/latest/tool-reference/spatial-statistics/spatial-autocorrelation.htm).\n"
      ],
      "id": "b2b207e2"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Remove rows with any NaN values to ensure valid spatial statistics\n",
        "df = df.dropna()\n",
        "\n",
        "# Create a spatial weights matrix based on the 8 nearest neighbors for each observation\n",
        "# The row-standardization ('r') ensures that each row sums to 1, which is useful for Moran's I calculation\n",
        "w = weights.distance.KNN.from_dataframe(df, k=8)  \n",
        "w.transform = 'r'  "
      ],
      "id": "dc36b41f",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we'll cycle through the Moran function for air release, current asthma among adults, and the 2 combined (bivariate). First, air releases.\n"
      ],
      "id": "f74963fb"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Import the moran_scatterplot function for visualizing spatial autocorrelation\n",
        "from splot.esda import moran_scatterplot\n",
        "\n",
        "# Calculate Moran's I statistic to measure spatial autocorrelation in air release data\n",
        "moran_air = esda.Moran(df['air_release'], w)\n",
        "\n",
        "# Print Moran's I value and the associated p-value for significance assessment\n",
        "print(\"Moran's I for Air Release:\")\n",
        "print(f\"I: {moran_air.I}\")\n",
        "print(f\"p-value: {moran_air.p_sim}\")\n",
        "\n",
        "# Generate a Moran scatter plot to visualize spatial autocorrelation for air release values\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "moran_scatterplot(moran_air, ax=ax)\n",
        "\n",
        "# Set plot limits to ensure both axes have the same range for accurate spatial comparison\n",
        "xlim = ax.get_xlim()\n",
        "ylim = ax.get_ylim()\n",
        "vmin = min(xlim[0], ylim[0])\n",
        "vmax = max(xlim[1], ylim[1])\n",
        "ax.set_xlim(vmin, vmax)\n",
        "ax.set_ylim(vmin, vmax)\n",
        "\n",
        "# Set the aspect ratio to 1:1 to maintain equal scaling on both axes\n",
        "ax.set_aspect('equal', adjustable='box')\n",
        "\n",
        "# Add a descriptive title to the plot\n",
        "ax.set_title(\"Moran Scatter Plot: Air Release\")\n",
        "plt.show()"
      ],
      "id": "6d72e450",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When evaluating Moran's I for single variables we are assessing how clustered like values are, i.e. do high or low values of air release tend to appear next to each other. On a scale of -1 to 1 a 0.12 suggests a slight positive correlation. High values of air release tend to cluster in similar spots. That said the correlation is moderate so we can state that there are a large amount of 0 values that are mixed in all throughout the data. You could attempt to address this issue by using a larger resolution when creating the raster layer depicting air release sums (we used 5000m), or by including different sources of air pollution.\n"
      ],
      "id": "6eec4801"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Calculate Moran's I statistic for spatial autocorrelation in asthma prevalence values\n",
        "moran_asthma = esda.Moran(df['asthma'], w)\n",
        "\n",
        "# Display Moran's I value and p-value to assess the spatial pattern in asthma prevalence\n",
        "print(\"\\nMoran's I for Asthma Prevalence:\")\n",
        "print(f\"I: {moran_asthma.I}\")\n",
        "print(f\"p-value: {moran_asthma.p_sim}\")\n",
        "\n",
        "# Generate a Moran scatter plot to visualize spatial autocorrelation in asthma prevalence data\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "moran_scatterplot(moran_asthma, ax=ax)\n",
        "\n",
        "# Set plot limits and aspect ratio to ensure a 1:1 scale\n",
        "xlim = ax.get_xlim()\n",
        "ylim = ax.get_ylim()\n",
        "vmin = min(xlim[0], ylim[0])\n",
        "vmax = max(xlim[1], ylim[1])\n",
        "ax.set_xlim(vmin, vmax)\n",
        "ax.set_ylim(vmin, vmax)\n",
        "ax.set_aspect('equal', adjustable='box')\n",
        "\n",
        "# Set title for the plot\n",
        "ax.set_title(\"Moran Scatter Plot: Asthma Prevalence\")\n",
        "plt.show()"
      ],
      "id": "faa374e3",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Asthma prevalence has a much stronger correlation structure. High asthma levels are clustered together spatially and have a value of 0.52. Also note that the scatterplot shows a nice fit without any odd artifacts or outliers.\n"
      ],
      "id": "eeafa703"
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# Calculate Bivariate Moran's I to examine spatial correlation between air release and asthma prevalence\n",
        "moran_bv = esda.Moran_BV(df['air_release'], df['asthma'], w)\n",
        "\n",
        "# Display Bivariate Moran's I value and p-value to assess significance\n",
        "print(\"\\nBivariate Moran's I (Air Release vs Asthma):\")\n",
        "print(f\"I: {moran_bv.I}\")\n",
        "print(f\"p-value: {moran_bv.p_sim}\")\n",
        "\n",
        "# Generate a bivariate Moran scatter plot to visualize spatial correlation between air release and asthma prevalence\n",
        "fig, ax = plt.subplots(figsize=(7, 7))\n",
        "moran_scatterplot(moran_bv, ax=ax)\n",
        "\n",
        "# Set plot limits and aspect ratio for consistent scaling\n",
        "xlim = ax.get_xlim()\n",
        "ylim = ax.get_ylim()\n",
        "vmin = min(xlim[0], ylim[0])\n",
        "vmax = max(xlim[1], ylim[1])\n",
        "ax.set_xlim(vmin, vmax)\n",
        "ax.set_ylim(vmin, vmax)\n",
        "ax.set_aspect('equal', adjustable='box')\n",
        "\n",
        "# Set title for the bivariate Moran scatter plot\n",
        "ax.set_title(\"Bivariate Moran Scatter Plot: Air Release vs Asthma\")\n",
        "plt.show()"
      ],
      "id": "b00091dc",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Similar to the Pearson analysis, when we analyze their combined spatial dependence the relationship is not as strong; largely due to the high levels of 0 air release values. That said, the relationship is positive and it is statistically significant (p-value 0.001).\n",
        "\n",
        "This could be due to several factors that went unaccounted for in our simple analysis.\n",
        "\n",
        "-   Our choice of size/resolution when creating our summed air release raster.\n",
        "-   The impacts of TRI air releases could be much more widespread than the 5km resolution we specified.\n",
        "-   We only analyze TRI regulated facilities and their emissions; we do not have data from unregulated facilities\n",
        "-   We did not account for traffic density or other sources of particulate matter emissions.\n",
        "-   We did not control for socioeconomic status, preventative healthcare access, and a whole host of other confounding variables.\n",
        "-   There is no underlying relationship. This is unlikely, based on large amounts of peer reviewed literature, but a possibility nonetheless.\n",
        "\n",
        "::: {.callout-tip style=\"color: #9f0b64;\"}\n",
        "#### Knowledge Check\n",
        "\n",
        "What type of data does the CDC PLACES dataset provide?\n",
        "\n",
        "a)  Real-time air quality measurements\n",
        "b)  Estimates of various health indicators at local levels\n",
        "c)  Hospital admission rates for air pollution-related illnesses\n",
        "d)  Locations of healthcare facilities in each county\n",
        ":::\n",
        "\n",
        "## Conclusion\n",
        "\n",
        "This lesson provided a comprehensive exploration of four environmental and public health datasets, focusing on the Detroit metropolitan area. By analyzing data from ICIS-AIR, the Toxic Release Inventory (TRI), the Social Vulnerability Index (SVI), and CDC PLACES, we gained insights into the complex relationships between industrial air pollution, social vulnerability, and health outcomes. The integration of these datasets, combined with geospatial analysis techniques, allowed us to identify areas where environmental hazards and social vulnerabilities intersect, highlighting potential environmental justice concerns. This approach demonstrates the power of combining diverse datasets and utilizing geospatial tools to inform policy decisions and target interventions in areas of greatest need.\n",
        "\n",
        "## Key Learning Points\n",
        "\n",
        "Congratulations! In this lesson you:\n",
        "\n",
        "-   Accessed and processed data from multiple EPA and CDC APIs, including ICIS-AIR, TRI, and PLACES datasets\n",
        "-   Created and manipulated geospatial objects using libraries such as **GeoPandas** and **rasterio**\n",
        "-   Visualized point data on maps using **Matplotlib** and **Contextily**\n",
        "-   Performed data cleaning and preprocessing, including handling missing coordinates and outliers\n",
        "-   Created choropleth maps to visualize health outcomes data at the census tract level\n",
        "-   Utilized Inverse Distance Weighting (IDW) interpolation to create continuous surfaces from point data\n",
        "-   Integrated multiple datasets to create a composite index (Air Release Vulnerability Index)\n",
        "-   Analyzed spatial patterns of air pollution, social vulnerability, and health outcomes\n",
        "-   Applied raster math and resampling techniques to align and compare different spatial datasets\n",
        "-   Interpreted results in the context of environmental justice and public health concerns\n",
        "-   Performed Pearson and Moran’s I correlation analyses\n",
        "\n",
        "## Lesson 3\n",
        "\n",
        "In this lesson, we explored ....\n",
        "\n",
        "[Lesson 3](https://ciesin-geospatial.github.io/TOPSTSCHOOL-air-quality/m203-grdiv1-pm25.html){.btn .btn-primary .btn role=\"button\"}"
      ],
      "id": "618c63ae"
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3 (ipykernel)"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}