RDM_metrics_from_Asana_csv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ABOUT THIS SCRIPT\n",
    "\n",
    "This script takes a csv file exported from the Consultations project in the Research IT Operations Asana: https://app.asana.com/0/531982708588662/list\n",
    "\n",
    "The script manipulates each row of the spreadsheet as a list. It uses list notation to refer to each column -- or what had been a column in the spreadsheet. For convenience, here are the list indices and headers associated with each column: \n",
    "\n",
    "| Index | Column header |\n",
    "| ------| ------------- |\n",
    "| [0]   | Task ID |\n",
    "| [1]   | Created At |\n",
    "| [2]   | Completed At |\n",
    "| [3]   | Last Modified |\n",
    "| [4]   | Name |\n",
    "| [5]   | Section/Column |\n",
    "| [6]   | Assignee |\n",
    "| [7]   | Assignee Email |\n",
    "| [8]   | Start Date |\n",
    "| [9]   | Due Date |\n",
    "| [10]  | Tags |\n",
    "| [11]  | Notes |\n",
    "| [12]  | Projects |\n",
    "| [13]  | Parent Task |\n",
    "| [14]  | Researcher |\n",
    "| [15]  | Position |\n",
    "| [16]  | PI/Lab |\n",
    "| [17]  | Department / ORU |\n",
    "| [18]  | RIT Service Area |\n",
    "| [19]  | RIT Service Area 2 |\n",
    "| [20]  | RIT Service Area 3 |\n",
    "| [21]  | Source (referrer) |\n",
    "| [22]  | Hand-off and/or referral |\n",
    "| [23]  | Complexity |\n",
    "\n",
    "\n",
    "NOTE: Column 5, Section/Column, added by Asana in 2019. All data sets exported prior to that change will no longer work with this notebook as it currently stands because we've had to change the list element numbers to compensate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORT STATEMENTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv, sys\n",
    "import re\n",
    "from collections import Counter\n",
    "from datetime import date, datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DATA INPUT CONFIGURATION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The .csv file containing the data is now generated from the Consultations project in Asana, \n",
    "then filtered for date and the tag 'RDM'.\n",
    "\n",
    "These are the parameters (arguments) that must be provided to the script:\n",
    "1. The filepath that points to the .csv file containing our data (which is now being generated from the the Consultations project in Asana).\n",
    "2. The filename of the .csv file.\n",
    "3. The 'report_period_descriptor' will be the text included in the first line of the report created by the script.\n",
    "4. The 'report_start' is the earliest consultation start date of the period covered by the report.\n",
    "5. The 'report_end' is the latest consultation start date of the period covered by the report\n",
    "\n",
    "You can load these from a file if you wish. The file should be a .py file that contains exactly the information shown below in lines 2 through 7. Place the file in the same folder as this Jupyter Notebook file, or provide the path to it in the cell magic command '%load.' (Remove the '#' before '%load' to run the command.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_args/RDM_metrics_args_CY2019_DriveFileStream.py'\n",
    "# These are the arguments provided to the script\n",
    "# Replace filepath with path on your machine and filename with time-stamped name of your csv export from Asana.\n",
    "\n",
    "filepath = '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_data/'\n",
    "filename = 'Consultations_20200325.csv'\n",
    "report_period_descriptor = 'the period from January 1, 2019 through December 31, 2019'\n",
    "report_start = '2019-01-01'\n",
    "report_end = '2019-12-31'\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DICTIONARY CONFIGURATION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This script utilizes dictionaries to store values for counting and to aggregate (\"roll up\"\") individual values \n",
    "under parent values. There are dictionaries for:\n",
    "1. The consultant involved in the consultation\n",
    "2. The department or ORU of the researcher/client\n",
    "3. The position or role of the researcher/client\n",
    "4. The category (RDM service area or lifecycle stage). NOTE: In 2019-2020 we switched to tracking all service areas, Research IT-wide\n",
    "5. The source of the consultation (or 'referral in'), rolled up to the organizational unit of the individual\n",
    "6. The referral (out), i.e., the person to whom the case was referred, rolled up to their organizational unit\n",
    "7. The complexity of the case\n",
    "8. The parent school, college or division of the researcher's/client's department or unit\n",
    "9. The College of Letters & Science as a whole (includes four divisions).\n",
    "\n",
    "We also use two other dictionaries to hold text values that vary by column:\n",
    "1. The labels used in our output for empty values\n",
    "2. The headings used in our output.\n",
    "\n",
    "Before we load the data file, we must initialize and configure several variables and dictionaries. The configuation files are kept in a directory called Metrics_config_files. There are two dictionaries at this point, one for referral data and one for organization data. We load those files using the cell magic command '%load.' (The file can be located in the same folder as the notebook file itself, or, as we have done, in a folder at the path provided in the %load command.) \n",
    "\n",
    "As new values are added to Asana, they should be added to the second value list within the correct dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we must set variables and initialize our dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Positional elements (columns) with the following indices -- and only those elements, at present --\n",
    "# will be gathered and counted using a dictionary\n",
    "# THE POSITIONAL ELEMENTS ASSIGNED HAVE CHANGED.\n",
    "# IDENTIFYING CONSULTANTS WILL BE DONE MUCH DIFFERENTLY, TOO, I SUSPECT.\n",
    "dictable_cols = [6, 17, 15, 18, 19, 20, 21, 22, 23] #Data from Asana csv\n",
    "\n",
    "# In this version, we bring the \"dictable columns\" into dictionaries called:\n",
    "# • ccounter (consultants) \n",
    "# • dcounter (departments/ORUs)\n",
    "# • pcounter (position)\n",
    "# • sacounter1 (RIT service area 1; was category)\n",
    "# • sacounter2 (RIT service area 2)\n",
    "# • sacounter3 (RIT service area 3)\n",
    "# • scounter (source)\n",
    "# • rcounter (referrals out) and \n",
    "# • cocounter (complexity).\n",
    "\n",
    "# Next we do modifications -- rolling up departmental values to their school, college or organizational parent; \n",
    "# L&S is first rolled up to its divisions, then aggregated as a college -- and store in new dictionaries: \n",
    "# • pacounter for values rolled-up by parent\n",
    "# • lscounter for Letters & Science divisions rolled up into a single total\n",
    "# • sacounter_all for three service area fields rolled up into one\n",
    "\n",
    "# Added sacounter_all_rdm_all Feb 29, 2020 to aggregate RDM areas in sacounter_all (i.e., service area - all values) \n",
    "# when we started doing consulting metrics Research IT-wide and stopped filtering on the RDM tag.\n",
    "\n",
    "#Initialize dictionaries that we'll use later\n",
    "ccounter, dcounter, pcounter, sacounter1, sacounter2, sacounter3, scounter, rcounter, cocounter,\\\n",
    "pacounter, lscounter, sacounter_all = {},{},{},{},{},{},{},{},{},{},{},{}\n",
    "sacounter_all_rdm_all = {}\n",
    "\n",
    "# Initialize lists of labels and headings\n",
    "labels = ['Unassigned', 'Unknown department', 'Unknown status', 'Not specified',  'None or Not specified',\\\n",
    "          'None or Not specified', 'Not specified', 'No hand-off or not specified', 'Not specified']\n",
    "\n",
    "orig_headings = ['Consultants, number of consultations',\\\n",
    "                 'Departments Served, number of engagements',\\\n",
    "                 'Researcher Status, number of researchers',\\\n",
    "                 'RIT Service Area 1: service area, number of cases',\\\n",
    "                 'RIT Service Area 2: service area, number of cases',\\\n",
    "                 'RIT Service Area 3: service area, number of cases',\\\n",
    "                 'Referrals In: from group or unit, number of cases',\\\n",
    "                 'Hand-offs or Referrals Out: to group or unit, number of cases',\\\n",
    "                 'Consultation Complexity: complexity, number of cases']\n",
    "mod_headings = ['School, College or Office: organization, number of engagements',\\\n",
    "                'School, College or Office, with L&S combined: organization, number of engagements',\\\n",
    "                'RIT Service Area (all 3 fields): service area, number of cases'\\\n",
    "                'RIT Service Area (all 3 fields), RDM aggregrated: service area, number of cases'] # for copied values\n",
    "all_headings = ['Consultants, number of consultations',\\\n",
    "                'Departments Served, number of engagements',\\\n",
    "                'Researcher Status, number of researchers',\\\n",
    "                'RIT Service Area 1: service area, number of cases',\\\n",
    "                'RIT Service Area 2: service area, number of cases',\\\n",
    "                'RIT Service Area 3: service area, number of cases',\\\n",
    "                'Referrals In: from group or unit, number of cases',\\\n",
    "                'Hand-offs or Referrals Out: to group or unit, number of cases',\\\n",
    "                'Consultation Complexity: complexity, number of cases',\\\n",
    "                'School, College or Office: organization, number of engagements',\\\n",
    "                'School, College or Office, with L&S combined: organization, number of engagements',\\\n",
    "                'RIT Service Area (all 3 fields): service area, number of cases'\\\n",
    "                'RIT Service Area (all 3 fields), RDM aggregrated: service area, number of cases']\n",
    "\n",
    "# List of dictionaries with modified values\n",
    "mod_dicts = [pacounter, lscounter, sacounter_all, sacounter_all_rdm_all]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the dictionary that aggregrates Source (referrals in) and Hand-offs and Referrals (out) values. . Uncomment the first line and load the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_config/config_ref_rollups_20200402.py'\n",
    "# Dictionary to aggregate Source (referrals in) and Handoffs and Referrals (out) values\n",
    "\n",
    "# These should be updated when new Asana tasks contain names not yet included here.\n",
    "ref_rollups = {\n",
    "    'lib': ['The Library', ['Harrison Dekker', 'Jamie Wittenberg', 'Susan Edwards', 'Steve Mendoza', 'Steven Mendoza',\n",
    "                            'Margaret Phillips', 'data-consult list (Library)', 'Brian Quigley', 'Library',\n",
    "                            'Data Storage/Sharing and the Social Sciences Working Group', 'Erik Mitchell',\n",
    "                            'Susan Powell', 'Anna Sackmann', 'David Eiffler', 'Yasmin Alnoamany', 'Co-consult: Celia Emmelhainz',\n",
    "                            'Stacy Reardon', 'Lisa Ngo','Co-consult: Kortney Rupp','co-consult: Kendra Levine',\n",
    "                            'Celia Emmelhainz', 'Hilary Schiraldi', 'Amy Neeser', 'Cody Hennesy','Mackie Mason / Library',\n",
    "                            'David Eifler in the CED library', 'Email to Anna Sackmann', 'Josh', 'Co-consult: Sam Teplitsky',\n",
    "                            'Michael Sholinbeck']],\n",
    "    'css': ['Campus Shared Services - IT', ['Brett Larsen', 'Daniel Bass', 'Johnathon Kogelman',\n",
    "                                            'Johnathon Kogelman (CSS-IT)', 'CSS-IT',\n",
    "                                            'Referred by Johnathon Kogelman (CSS-IT)',\n",
    "                                            'request to Rick from Daniel Bass',\n",
    "                                            'email to Rick from Daniel Bass',\n",
    "                                            'Daniel Bass (CSS-IT)',\n",
    "                                            'CSS_IT (Jon Valmores)']],\n",
    "    'dlab': ['D-Lab', ['D-Lab Consulting List', 'D-Lab', 'Jon Stiles', 'Zawadi Rucks Ahidiana',\n",
    "                       'Rick Jaffe (via D-Lab Consulting web page)', 'dlab-consultants@lists.b.e',\n",
    "                       'd-lab consultants list', 'referred to D-Lab/Jon Stiles', 'D-Lab consultants list',\n",
    "                       'Rick at d-lab consulting Ticket #29430', 'D-Lab ticket#29433', 'Chris Hench (D-Lab)',\n",
    "                       'Susan Grand (D-Lab)', 'Aaron Culich / D-Lab', 'D-Lab (Aaron Culich, Patty Frontiera']],\n",
    "    'scf': ['Statistical Computing Facility', ['Chris Paciorek', 'Ryan Lovett']],\n",
    "    'brc': ['Berkeley Research Computing',\n",
    "            ['Patrick Schmitz', 'Aron Roberts', 'Aaron Culich', 'Jason Christopher', 'Kelly Rowland', 'Gary Jung',\n",
    "             'BRC Cloud Consulting', 'Jason Huff (Computational Genomics Resource Lab)',\n",
    "             'Berkeley Research Computing - Cloud', 'Yong Qin', 'Deb McCaffrey', 'email to BRC',\n",
    "             'brc@berkeley.edu', 'brc@b.e.', 'brc@b.e', 'Maurice Manning', 'Savio ticket', 'BRC',\n",
    "             'Savio ticket forwarded by Chris Paciorek', 'BRC/Secure AEoD Service?', 'Krishna & Savio Folks',\n",
    "             'Oliver Muellerklein and Christopher Hann-Soden','Ticket to RIT-BRC-HPC']],\n",
    "    'dh': ['Digital Humanities @ Berkeley', ['Quinn Dombrowski', 'Camille Villa', 'Digital Humanities',\n",
    "                                             'Claudia Natalia Von Vacano']],\n",
    "    'rdm': ['RDM Consulting', ['researchdata@berkeley.edu', 'Rick Jaffe', 'Chris Hoffman', 'John B Lowe',\n",
    "                               'BRC Survey 2016 (Response to follow-up from Jamie)', 'email to Rick Jaffe',\n",
    "                               'Follow-up', 'Rick', 'follow up', 'email to Rick and Jason',\n",
    "                               'researchdata@b.e. (after browsing web site)', 'researchdata@b.e.',\n",
    "                               'researchdata@b.e', 'email to Rick from Jessica', 'email to Rick from Carla',\n",
    "                               'email to Rick from Laura', 'email to Rick from Phuong', 'email to Rick from Sarah',\n",
    "                               'Anna Sackman (RDM)', 'Email to Rick', 'email to Rick',\n",
    "                               'email to Rick in response to bConnected Box corruption thread',\n",
    "                               'follow-up to earlier RDM consult', 'email to Anna', 'e-mail to researchdata@b.e.', 'email to researchdata@b.e',\n",
    "                               'RDM Consulting', '(partial hand-off) to RDM for consultation on sensitive data',\n",
    "                               'Andy Peterson email to Jason and Rick', 'Rick Jaffe (RDM)']],\n",
    "    'cdl': ['California Digital Library', ['Joan Starr', 'Stephanie Simms',\n",
    "                                           'Daniella Lowenberg (DASH), via Quinn Dombrowski']],\n",
    "    'istapi': ['Information Services & Technology - API', ['Jennifer Bellenger', 'Jon Broshious',\n",
    "                                                           'Ian Crew', 'Ian Crew (bConnected)',\n",
    "                                                           'Jon Hays',\n",
    "                                                           'bConnected',\n",
    "                                                           'bconnected',\n",
    "                                                           'Michael Leefers',\n",
    "                                                           'Alex Walton',\n",
    "                                                           'referred by Ian (bConnected)',\n",
    "                                                           'referred to Rick by Jennifer Bellenger (bConnected)',\n",
    "                                                           'Forwarded by Beth Muramoto (GSE) to Ian Crew (bConnected), who forwarded it in turn to researchdata@berkeley.edu']],\n",
    "    'ist': ['Information Services & Technology', ['Jeff Makaiwi (IST Unix team)', 'IST',\n",
    "                                                  'IST Windows ticket',\n",
    "                                                  'IST PI ticket', 'Joe Silva, IST',\n",
    "                                                  'Isaac Orr / NOS',\n",
    "                                                  'Co-consult: Tim Gotch; Jack Shnell; Greg German',\n",
    "                                                  'IST Unix team']],\n",
    "    'micronet': ['Micronet', ['micronet', 'Micronet', 'micronet list', 'Micronet list']],\n",
    "    'iao': ['Industry Alliances Office', ['Nicole Hensley', 'Nicole Hensley (IAO/IPIRA)', 'Nicole Hensley (IAO)',\n",
    "                                         'email to Chris and Rick from Eric Giegerich', 'IAO',\n",
    "                                         'email from Nicole Hensley to Rick and Chris',\n",
    "                                         'Email from Nicole Hensley to Chris and to Rick',\n",
    "                                         'email from Janina Marie Maniaol/IAO',\n",
    "                                         'Nicole Hensley/IAO, via email to Chris and Rick',\n",
    "                                         'Eric Giegerich (IPIRA)','IPIRA',\n",
    "                                         'Greg Howell / IAO', 'Greg Howell (IPIRA)',\n",
    "                                         'Eric Giegerich/IAO', 'Janina Marie Maniaol (IAO)'\n",
    "                                         ]],\n",
    "    'ssw': ['School of Social Welfare', ['David Fullmer']],\n",
    "    'bids': ['Berkeley Institute for Data Science', ['BIDS']],\n",
    "    'brdo': ['Berkeley Research Development Office (VCRO)', ['Barbara Ustanko via Chris Hoffman']],\n",
    "    'lsit': ['Letters & Science IT', ['Michael Quan (Letters & Science IT)']],\n",
    "    'ais': ['Academic Innovation Studio', ['AIS drop-in (handled by Rick)']],\n",
    "    'musinf': ['Museum Informatics', ['BIDS Faire CSpace Portals poster']],\n",
    "    'rit': ['Research IT', ['research-it@berkeley.edu', 'rit@berkeley.edu', 'RIT email']],\n",
    "    'noid':  ['Unspecified', ['Unspecified', 'No hand-off or not specified']],\n",
    "    'isp': ['Information Security Office', ['Information Security and Policy', 'ISO',\n",
    "                                            'Chris Doane', 'Jake Harwood (ISP)']],\n",
    "    'irb': ['Internal Review Board', ['Daisy at IRB', 'IRB']],\n",
    "    'fac': ['Faculty', ['Carl Boettiger / ESPM 288', 'Corinne Riddell Faculty',\n",
    "                       'Deby Wilzbach JOHNS', 'Heather Haveman','Jim Demmel EECS',\n",
    "                       'Gabriel Gonzales', 'Reference by Niek Veldhuis']],\n",
    "    'citris':['CITRIS', ['citris']],\n",
    "    'outside': ['Outside Orgs', ['CSIRO (Commonwealth Scientific and Industrial Research Organisation) Australia',\n",
    "                                'NSF GRFP -> XSEDE Campus Champion list']],\n",
    "    'cao': ['Chief Administrative Officers Group', ['Dana Kowalski (CAO)', 'Dana Kowalski CAO']]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the dictionary that aggregrates Organization values. Uncomment the first line and load the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_config/config_org_rollups_20200402.py'\n",
    "# Dictionaries to aggregate Organization values.\n",
    "\n",
    "# These should be updated when new Asana tasks contain names not yet included here. \n",
    "\n",
    "# Also: note that this dictionary was initiated before we moved to standardized names, \n",
    "# so it contains multiple versions of some items. \n",
    "# It should work if pruned to only include the standardized names. \n",
    "# For list of standardized names, see: \n",
    "# https://docs.google.com/spreadsheets/d/1frNC_A9mERFfLrECX7GWWRBHkUZpGPqMWy_Ia3EYjnQ/\n",
    "\n",
    "org_rollups = {\n",
    "    'anr': ['UC Agriculture and Natural Resources', ['UC Agriculture and Natural Resources (UCANR)', \n",
    "                                                   'UC Natural Reserve System']],\n",
    "    'cchem': ['College of Chemistry', ['Chemistry', \n",
    "                                       'Chemical & Biomolecular Engineering', \n",
    "                                       'Chemical Engineering']],\n",
    "    'ced': ['College of Environmental Design', ['Environmental Design (CED)',\n",
    "                                                'Architecture',\n",
    "                                                'City & Regional Planning',\n",
    "                                                'Landscape Architecture & Environmental Planning']],\n",
    "    'citris':['Center for Information Technology Research in the Interest of Society', ['Center for Information Technology Research in the Interest of Society (CITRIS)']],\n",
    "    'cnr': ['College of Natural Resources', ['College of Natural Resources (CNR)', \n",
    "                                             'Agricultural & Resource Economics (ARE)',\n",
    "                                             'Energy and Resources Group',\n",
    "                                             'Environmental Science, Policy & Management (ESPM)',\n",
    "                                             'Plant and Microbial Biology']],\n",
    "    'coe': ['College of Engineering', ['Engineering (CoE)',\n",
    "                                       'Bioengineering',\n",
    "                                       'Civil and Environmental Engineering',\n",
    "                                       'Industrial Engineering & Operations Research',\n",
    "                                       'Institute for Environmental Science and Engineering',\n",
    "                                       'Electrical Engineering and Computer Science (EECS)',\n",
    "                                       'Electrical Engineering/EECS', \n",
    "                                       'Materials Science and Engineering',\n",
    "                                       'Mechanical Engineering (ME)',\n",
    "                                       'Nuclear Engineering']],\n",
    "    'dh':  ['Digital Humanities at Berkeley', ['Digital Humanities at Berkeley', \n",
    "                                               'Digital Humanities at Berkeley (DH)', \n",
    "                                               'Digital Humanities @ Berkeley']],\n",
    "    'eslib':['Ethnic Studies Library', ['Ethnic Studies Library']],\n",
    "    'gbsmrc':['Golden Bear Sleep and Mood Research Clinic', ['Golden Bear Sleep & Mood Research Clinic']],\n",
    "    'gse': ['Graduate School of Education', ['Education (GSE)']],\n",
    "    'gsj': ['Graduate School of Journalism', ['School of Journalism']],\n",
    "    'haas':['Haas School of Business', ['Haas School of Business']],\n",
    "    'intl':['International programs', ['International Computer Science Institute (ICSI)']],\n",
    "    'ist': ['Information Services & Technology', ['Information Services & Technology (IST)']],\n",
    "    'law': ['Berkeley Law', ['School of Law', \n",
    "                             'Boalt Hall School of Law', \n",
    "                             'Jurisprudence and Social Policy', \n",
    "                             'Legal Studies Program',\n",
    "                             'Law (Boalt Hall, Berkeley Law)']],\n",
    "    'lib': ['Library', ['Bancroft Library', \n",
    "                        'Bancroft Library - Oral History Center', \n",
    "                        'C.V. Starr East Asian Library', \n",
    "                        'Ethnic Studies Library', \n",
    "                        'Library']],\n",
    "    'ls':  ['College of Letters & Science - College-wide', ['American Studies',\n",
    "                                                            'College of Letters and Science (L&S)', \n",
    "                                                            'D-Lab',\n",
    "                                                            'Haas Scholars'\n",
    "                                                            'Undergraduate & Interdisciplinary Studies']],\n",
    "    'lsa': ['College of Letters & Science - Arts & Humanities', ['Art Practice',\n",
    "                                                                 'Classics',\n",
    "                                                                 'History of Art',\n",
    "                                                                 'Music', \n",
    "                                                                 'Near Eastern Studies']],\n",
    "    'lsb': ['College of Letters & Science - Biological Sciences', \n",
    "                                            ['Integrative Biology', \n",
    "                                             'Molecular & Cell Biology',\n",
    "                                             'Molecular & Cell Biology (MCB)']],\n",
    "    'lsm': ['College of Letters & Science - Math & Physical Sciences', \n",
    "                                            ['Astronomy',\n",
    "                                             'Earth and Planetary Science (EPS)', \n",
    "                                             'Physics',\n",
    "                                             'Statistics']],\n",
    "    'lss': ['College of Letters & Science - Social Sciences', \n",
    "                                            ['African American Studies',\n",
    "                                             'Anthropology',\n",
    "                                             'Economics',\n",
    "                                             'Geography',\n",
    "                                             'History', \n",
    "                                             'Political Science',\n",
    "                                             'Psychology',\n",
    "                                             'Sociology']],\n",
    "    'nat': ['National programs', ['Robert Wood Johnson Berkeley (Scholars in Health Policy Research Program)', \n",
    "                                  'Department of Economics / Robert Wood Johnson Berkeley (Scholars in Health Policy Research Program)']],\n",
    "    'neuro':['Berkeley Neuroscience', ['Berkeley Neuroscience', \n",
    "                                       'Helen Wills Neuroscience Institute (HWNI)']],\n",
    "    'noid':['Not specified', ['Unknown value', 'unidentified', 'Unassigned', 'Unknown department', \n",
    "                              'Unknown status', 'Unspecified', 'Unknown division']],\n",
    "    'other':['Other', ['UCLA - Humanities CIO', 'Italian Institute of Technology - Fondazione Istituto Italiano di Tecnologia']],\n",
    "    'qb3': ['California Institute for Quantitative Biosciences (QB3)', ['CIRM/QB3 Shared Stem Cell Facility and High-Throughput Screening Facility']],\n",
    "    'sph': ['School of Public Health', ['Public Health (SPH)', \n",
    "                                        'Berkeley Center for Health Technology (BCHT)',\n",
    "                                        'Biostatistics/Public Health',\n",
    "                                        'Environmental Health Sciences (SPH)',\n",
    "                                        'Health Policy (SPH)',\n",
    "                                        'UC Berkeley-UCSF Joint Medical Program', \n",
    "                                        'UC Berkeley-UCSF Joint Medical Program (JMP)',\n",
    "                                        'Epidemiology (SPH)', \n",
    "                                        'Center for Lean Engagement & Research in Healthcare (CLEAR) (SPH)']], \n",
    "    'ssw': ['School of Social Welfare', ['School of Social Welfare', 'Social Welfare (SSW)']],\n",
    "    'tchi':['Terner Center for Housing Innovation', ['Terner Center for Housing Innovation']],\n",
    "    'vcaf':['Vice Chancellor for Administration and Finance', ['Procurement Services - Supply Chain Management']],\n",
    "    'vcei':['Vice Chancellor for Equity & Inclusion', ['Equity & Inclusion', 'E&I']],\n",
    "    'vcr': ['Vice Chancellor for Research Office', ['Archaeological Research Facility (ARF)',\n",
    "                                                    'Berkeley Initiative for Transparency in the Social Sciences (BITSS)',\n",
    "                                                    'Berkeley Institute for Data Science (BIDS)', \n",
    "                                                    'Berkeley Seismological Lab',\n",
    "                                                    'Center for Effective Global Action (CEGA)',\n",
    "                                                    'Center for Studies in Higher Education',\n",
    "                                                    'Center for Race and Gender', 'Center for Study of Race and Gender',\n",
    "                                                    'Haas Institute for a Fair & Inclusive Society',\n",
    "                                                    'Human Evolution Research Center',\n",
    "                                                    'Industry Alliances Office (Vice Chancellor for Research)',\n",
    "                                                    'Institute of European Studies',\n",
    "                                                    'Institute of Human Development', \n",
    "                                                    'Institute for Research on Labor and Employment (IRLE)',\n",
    "                                                    '(California Policy Lab)',\n",
    "                                                    'Institute for the Study of Societal Issues',\n",
    "                                                    'Transportation Studies (ITS)',\n",
    "                                                    'Transportation Studies (ITS) - PATH',\n",
    "                                                    'Phoebe A. Hearst Museum of Anthropology', 'Hearst Museum of Anthropology',\n",
    "                                                    'UC Botanical Garden',\n",
    "                                                    'University of California Transportation Center'\n",
    "                                                    'Pacific Earthquake Engineering Center']],\n",
    "    'vcue':['Vice Chancellor for Undergraduate Education', ['Berkeley Resource Center for Online Education (BRCOE)', 'Research IT']],\n",
    "}\n",
    "\n",
    "# This dictionary rolls up the Letters & Science divisions into a single total.\n",
    "\n",
    "ls_rollup = {\n",
    "    'lsall': ['College of Letters & Science - All', ['College of Letters & Science - College-wide', \n",
    "                                                     'College of Letters & Science - Arts & Humanities', \n",
    "                                                     'College of Letters & Science - Biological Sciences', \n",
    "                                                     'College of Letters & Science - Math & Physical Sciences', \n",
    "                                                     'College of Letters & Science - Social Sciences']]\n",
    "}\n",
    "\n",
    "sa_rollup = {\n",
    "    'rdm': ['RDM', ['Active Data', 'Data Analysis Guidance', 'Data Backup', 'Data Collection', \n",
    "                    'Data Finding & Acquisition','Data Management Plans (DMP)', 'Data Organization', \n",
    "                    'Data Preservation', 'Data Publication & Sharing', 'Data Security', 'Data Transfer', \n",
    "                    'Digital Asset Management Systems', 'Persistant Identifiers']]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FUNCTIONS TO USE LATER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Couldn't get this to work!\n",
    "# import itertools\n",
    "# def remove_header_row(rows):\n",
    "#     #first header value is 'Task ID'; remove that row\n",
    "#     rows = itertools.filterfalse(lambda row: row[0] == 'Task ID', rows)\n",
    "#     return rows\n",
    "\n",
    "# Function for merging lists, keeping only unique values. \n",
    "# mergesource and mergetarget must be lists; skip must be string value\n",
    "\n",
    "def mergelists(mergesource, mergetarget, skip):\n",
    "    for s in mergesource:\n",
    "            if s == skip:    # skip this value\n",
    "                continue\n",
    "            elif s in mergetarget:  # if a value in the merge source is already in merge target, skip\n",
    "                continue\n",
    "            else:   # capture this value\n",
    "                mergetarget.append(s)\n",
    "    return(mergetarget)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "READ, FILTER AND CLEAN THE DATA\n",
    "\n",
    "With the dictionaries in place, the action begins. \n",
    "\n",
    "In the next cell, we read the .csv file and filter out all but the rows we want to keep: tasks with the tag \"RDM\" between the start and end dates of the report. At the same time, we gather information from subtasks (parent task name and subtask assignee) -- but we don't want to keep those rows because we don't want to count the subtasks as separate consultations. \n",
    "\n",
    "UPDATE, Feb 2020: We have stopped filtering rows based on RDM tags and now count all rows. We now aggregate data-related service areas to come up with the total number of RDM-related consultations. (We roll-up the RDM-related service areas after merging the counts from the three service area fields.)\n",
    "\n",
    "In subsequent cells, we prepare the data for counting and further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# TODO - Catch and handle missing arguments or errors in the arguments\n",
    "\n",
    "# Convert report_start and report_end arguments to datetime format\n",
    "reportstart = datetime.strptime(report_start, '%Y-%m-%d')\n",
    "reportend = datetime.strptime(report_end, '%Y-%m-%d')\n",
    "\n",
    "# Initialize lists and dictionary that will be used to gather parent task name and subtask assignee values \n",
    "# from subtasks and add these to the list of task assignees \n",
    "parenttasks = [] # parent tasks list\n",
    "assignees = []  # co-assignees list\n",
    "#coassignees = {}  # dictionary of parent tasks and co-assignees names\n",
    "\n",
    "myrows = []   # this will hold the values from each row that we keep, each row stored as a list within this list\n",
    "\n",
    "# Read data into a list of lists, keeping only RDM consultation tasks between report start and end date\n",
    "\n",
    "with open(filepath + filename) as csvfile:\n",
    "    reader = csv.reader(csvfile, delimiter=\",\")\n",
    "    next(reader) # skip header row  -- TODO: Save header row for writing to output file\n",
    "    rows = [r for r in reader]\n",
    "    for row in rows:\n",
    "        # Filter Asana csv to include only consultations during the desired period that have the tag 'RDM'.\n",
    "        \n",
    "        # Convert Created At values (second column) to datetime format and \\\n",
    "        # compare against report-start and report-end arguments. Skip if start date is not in report period range\n",
    "        # First, replace slashes with dashes\n",
    "        stDate = row[1].replace(\"/\", \"-\")\n",
    "        startdate = datetime.strptime(stDate, '%Y-%m-%d')\n",
    "        if not reportstart <= startdate <= reportend: continue\n",
    "            \n",
    "        # All RDM consultations have been tagged 'RDM'. \n",
    "        # Remove rows in which tag values -- row[10] -- do not include the 'RDM' tag\n",
    "        # First, split tag values into a list (tags values are separated by a comma)\n",
    "        row[10] = row[10].split(',')\n",
    "\n",
    "        # if not 'RDM' in row[10]: continue\n",
    "                \n",
    "        # We create subtasks in Asana to manage the consultation work, and sometimes assign the subtask to \n",
    "        # another consultant. We want to track the participation of the additional consultant(s), but not count\n",
    "        # the subtask as a separate consultation.\n",
    "        \n",
    "        # To do this, we need to find those rows that are subtasks and copy the parent task name and \n",
    "        # the assignee name into temporary data structures, but not add the row to the list of rows. \n",
    "        # Eventually, the assignees will be added to the Assignee field (row[6]) \n",
    "        # of the parent task. \n",
    "        \n",
    "        # (Our plan was to identify co-consultants in Asana by assigning that co-consultant a sub-task whose\n",
    "        # name begins \"CO-CONSULT\". Now we're thinking of adding a custom 'Co-consultee' field to track this.\n",
    "        # Still, we want to note the network of all people working on consultations.)\n",
    " \n",
    "        if row[13] != '':   #if this row has a parent task name, i.e., if this is a subtask\n",
    "            # if row[4] startswith 'CO_CONSULT':   # Leave this for now\n",
    "            parenttasks.append(row[13])\n",
    "            assignees.append(row[6])\n",
    "            continue\n",
    "            \n",
    "        myrows.append(row)\n",
    "        \n",
    "#print(myrows)  #debug only\n",
    "#print(parenttasks)  #debug only\n",
    "#print(assignees)  #debug only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we need to add subtask assignees to the list of assignees of the parent task.\n",
    "# To do that, we'll create a coassignees dictionary with key = parent task name and value = assignee name.\n",
    "\n",
    "# Note that there can be multiple subtasks per parent task, and, though unlikely,\n",
    "# two or more parent tasks with the same name. \n",
    "# We have to check for existing keys when creating the dictionary so we don't overwrite existing values       \n",
    "\n",
    "coassignees = {}  # dictionary of parent tasks and co-assignees names\n",
    "\n",
    "for p, a in zip(parenttasks, assignees):\n",
    "    if p in coassignees:   # If key already exists... \n",
    "        avals = []    # initialize temporary list of assignee values\n",
    "        avals.append(coassignees[p])   # add existing value of that key to temporary list\n",
    "        avals = avals[0] + ', ' + a   # add new value to existing\n",
    "        a = avals  # re-assign to variable\n",
    "    coassignees[p] = a  # assign the new value(s) to the key\n",
    "        \n",
    "# print(coassignees)  # For debugging purposes\n",
    "# for k, v in coassignees.items():    # For debugging purposes - alternate view\n",
    "#     print(k, v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that we've isolated the Asana tasks (rows in the csv file) that we want to count, \n",
    "# iterate through the rows and prepare them for counting. We need to:\n",
    "# • replace empty values with appropriate labels; \n",
    "# • split multiple values in the columns that we will count; \n",
    "# • add subtask assignees to parent tasks; and finally \n",
    "# • make sure that the values we will count are elements in a list. (If they are strings, the Counter function\n",
    "# will iterate over the string and count the occurences of each letter, rather than of the name.)\n",
    "\n",
    "for row in myrows:\n",
    "    \n",
    "    # REPLACE EMPTY VALUES WITH THE APPROPRIATE LABELS\n",
    "    \n",
    "    # For Consultant(s), Department/ORU, Patron status, RIT Service Area (aka Category), \n",
    "    # Source (aka referral in), Hand-off or referral (aka referral out), Consultation complexity: \n",
    "    # replace empty values with appropriate label.\n",
    "        \n",
    "    # for n, label in zip(config_20171104.dictable_cols, config_20171104.labels):  ## USE THIS IN PYCHARM\n",
    "    for n, label in zip(dictable_cols, labels):\n",
    "\n",
    "        # Fill in empty cells with appropriate label\n",
    "        if row[n] == '':\n",
    "            row[n] = label\n",
    "\n",
    "            \n",
    "    # SPLIT MULTIPLE VALUES IN COLUMNS THAT WE WILL COUNT\n",
    "        \n",
    "    for n in [17, 21, 22]:   # Dept/ORU, Source and Hand-offs... may have multiple values separated by '; '\n",
    "        if type(row[n]) == str:\n",
    "            row[n] = row[n].split('; ')\n",
    "        \n",
    "    # ADD SUBTASK ASSIGNEES TO PARENT TASKS\n",
    "        \n",
    "    if row[4] in coassignees.keys():  # If task is parent of a subtask...\n",
    "        \n",
    "        subassigns = []  \n",
    "        uniqueassigns = []\n",
    "        assigns = []\n",
    "        \n",
    "        for c in coassignees[row[4]].split(', '):\n",
    "            subassigns.append(c)          # add each task owner to a list\n",
    "        #print(subassigns)  #for debugging purposes  \n",
    "        # Merge subassignees list into a list of unique values, skipping empty values\n",
    "        subassignees = mergelists(subassigns, uniqueassigns, '')\n",
    "        assigns.append(row[6])  # start a list beginning with task assignee\n",
    "        # Merge unique subassignees list created above into list we just created that contains \n",
    "        # the parent task assignee, skipping empty values        \n",
    "        assignees = mergelists(subassignees, assigns, '')\n",
    "        #print(assignees)  #for debugging purposes\n",
    "        # Assign the merged values as consultants on the task\n",
    "        row[6] = assignees\n",
    "        \n",
    "    else:     # No subtasks for this task. Convert assignee value from string to list\n",
    "        consultant = []\n",
    "        consultant.append(row[6])\n",
    "        row[6] = consultant\n",
    " \n",
    "    # ASSURE CELL VALUES ARE LIST ELEMENTS SO COUNTER FUNCTION DOESN'T ITERATE OVER STRINGS \n",
    "\n",
    "    for n in [15, 17, 18, 19, 20, 23]:\n",
    "        txt = []\n",
    "        if type(row[n]) == list:\n",
    "            for r in row[n]:\n",
    "                txt.append(r)\n",
    "        elif type(row[n]) == str:\n",
    "            txt.append(row[n])    \n",
    "        row[n] = txt\n",
    "        \n",
    "# print(myrows)  # debug only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AGGREGATE DATA\n",
    "\n",
    "Now we do our first aggregation: we roll-up the individual sources of referrals (in) and the individuals to whom we referred (out) researchers into their unit or division affiliation.  \n",
    "\n",
    "Eventually, we will aggregate/roll-up a series of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# AGGREGATE (ROLL-UP) VALUES \n",
    "myrows_aggregated = myrows\n",
    "\n",
    "for row in myrows_aggregated:\n",
    "    \n",
    "    # Replace individual names with the corresponding org name in Source (aka Referral In) and\n",
    "    # Hand-off or referral (aka Referral Out) fields\n",
    "    # (positional elements [21] and [22])\n",
    "    for n in [21, 22]:\n",
    "        ref_x = []\n",
    "        if type(row[n]) == list:\n",
    "            for r in row[n]:\n",
    "                ref_x.append(r)\n",
    "        elif type(row[n]) == str:\n",
    "            ref_x.append(row[n])\n",
    "        row[n] = []  # Empty cell to ready it for being re-filled\n",
    "        for term in ref_x:\n",
    "            #for key in config_20171104.ref_rollups.keys():    ## USE THIS IN PYCHARM\n",
    "            for key in ref_rollups.keys():\n",
    "                #if term in config_20171104.ref_rollups[key][1]:   ## USE THIS IN PYCHARM\n",
    "                if term in ref_rollups[key][1]: \n",
    "                    #term = config_20171104.ref_rollups[key][0]    ## USE THIS IN PYCHARM\n",
    "                    term = ref_rollups[key][0]\n",
    "        row[n].append(term)\n",
    "                    \n",
    "# print(myrows_aggregated)  # Debug only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WRITE TO FILE\n",
    "\n",
    "Write our cleaned and partially aggregated data set to a file for safe keeping and other uses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a filename with a simple timestamp so we don't overwrite file each time we write.\n",
    "path = '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_output/'\n",
    "basename = 'rdm_filteredtasks_aggregatedreferrals_'\n",
    "datestr = str(datetime.today())\n",
    "datestr = datestr[:-7]   # Remove microseconds\n",
    "datestr = datestr.replace(' ', 'T')  # Replace space with 'T' so filename doesn't cause problems elsewhere\n",
    "datestr = datestr.replace(':','-')  # The colons in the timestamp may cause problems, so swap them out, too\n",
    "fileoutname = path + basename + datestr + '.csv'\n",
    "\n",
    "with open(fileoutname, 'w', newline='') as csvfile:\n",
    "    rdmwriter = csv.writer(csvfile)\n",
    "    rdmwriter.writerows(myrows_aggregated)        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "COUNT DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We count the number of consultations (number of rows gathered above) and determine how many have been resolved successfully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ***** COUNT THE DATA *****\n",
    "# Each row (list) represents a consulting engagement\n",
    "# Feb 2020: We now count all Research IT consultations, not just RDM Consulting ones.\n",
    "print('\\nIn %s, Research IT Consulting provided %d consultations.' % (report_period_descriptor, (len(myrows)))) \n",
    "\n",
    "# # Count how many engagements are resolved successfully\n",
    "# #TODO: FIGURE OUT HOW TO MEASURE THIS\n",
    "# yesrows = []\n",
    "\n",
    "# for r in myrows_aggregated:\n",
    "#     r15 = r[15]\n",
    "#     if 'Resolved' in r15:\n",
    "#         yesrows.append(r)\n",
    "\n",
    "# #print('We reached a successful resolution in %d of those engagements.' % len(yesrows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we count and store the unique values in several of the erstwhile columns and print the totals to the screen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gather and count (subtotal) the values for consultant(s), department/oru, researcher status, RIT Service Area \n",
    "# (formerly category), source (referrals in), hand-off or referral (referrals out) and consultation complexity.\n",
    "\n",
    "#for i, n in enumerate(config_20171104.dictable_cols):   ## USE THIS IN PYCHARM\n",
    "for i, n in enumerate(dictable_cols):  # the i identifies heading to use when printing results to screen (below)\n",
    "    counter = Counter()\n",
    "    for row in myrows_aggregated:\n",
    "        for z in row[n]:\n",
    "            counter[z] += 1\n",
    "        if n == 6:\n",
    "            ccounter.update(counter.most_common())\n",
    "        elif n == 17:\n",
    "            dcounter.update(counter.most_common())     \n",
    "        elif n == 15:\n",
    "            pcounter.update(counter.most_common())\n",
    "        elif n == 18:\n",
    "            sacounter1.update(counter.most_common())\n",
    "        elif n == 19: \n",
    "            sacounter2.update(counter.most_common())\n",
    "        elif n == 20:   \n",
    "            sacounter3.update(counter.most_common())\n",
    "        elif n == 21:\n",
    "            scounter.update(counter.most_common())\n",
    "        elif n == 22:\n",
    "            rcounter.update(counter.most_common())\n",
    "        else:\n",
    "            cocounter.update(counter.most_common())\n",
    "    \n",
    "    # Print results to screen\n",
    "    #print('\\n' + config_20171104.orig_headings[j] + ':')  # Use this in Pycharm or command-line invocation\n",
    "    print('\\n' + orig_headings[i] + ':')\n",
    "    for (k, v) in counter.most_common():\n",
    "        print(k + ': ' + str(v))\n",
    "    \n",
    "# Uncomment to view data structures and to debug        \n",
    "# print('\\n')\n",
    "# print(ccounter)\n",
    "# print('\\n')\n",
    "# print(dcounter)\n",
    "# print('\\n')\n",
    "# print(pcounter)\n",
    "# print('\\n')\n",
    "# print(sacounter1)\n",
    "# print('\\n')\n",
    "# print(sacounter2)\n",
    "# print('\\n')\n",
    "# print(sacounter3)\n",
    "# print('\\n')\n",
    "# print(scounter)\n",
    "# print('\\n')\n",
    "# print(rcounter)\n",
    "# print('\\n')\n",
    "# print(cocounter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, modify the values and write to new dictionaries: roll-up department/ORU key/value pairs into schools, colleges and divisions, first with the divisions of the College of Letters & Science separated and then with the divisions all counted within the College. [TODO: Do the same for the divisions of the College of Engineering?]. Do the same with the data-related service areas to create one tally for 'RDM'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregate (\"roll-up\") granular Department value into corresponding parent organization value\n",
    "\n",
    "for k, v in dcounter.items():\n",
    "    #for key in config_20171104.org_rollups.keys():    # Use this in Pycharm or command-line invocation\n",
    "    for key in org_rollups.keys():\n",
    "        #if k in config_20171104.org_rollups[key][1]:   # Use this in Pycharm or command-line invocation\n",
    "        if k in org_rollups[key][1]:         # No else to this if: org_rollup dict updated to include all keys (Depts)\n",
    "            #k = config_20171104.org_rollups[key][0]    # Use this in Pycharm or command-line invocation\n",
    "            k = org_rollups[key][0]\n",
    "            if k in pacounter.keys():\n",
    "                pacounter[k] = (pacounter[k] + v)\n",
    "            else:\n",
    "                pacounter[k] = v\n",
    "\n",
    "# Additionally, roll-up all Letters & Science in one L&S tally\n",
    "\n",
    "for k, v in pacounter.items():\n",
    "    #for key in config_20171104.ls_rollup.keys():    # Use this in Pycharm or command-line invocation\"\n",
    "    for key in ls_rollup.keys():\n",
    "        #if k in config_20171104.ls_rollup[key][1]:   # Use this in Pycharm or command-line invocation\n",
    "        if k in ls_rollup[key][1]:\n",
    "            #k = config_20171104.ls_rollups[key][0]    # Use this in Pycharm or command-line invocation\n",
    "            k = ls_rollup[key][0]\n",
    "            if k in lscounter.keys():\n",
    "                lscounter[k] = (lscounter[k] + v)  # L&S-All key added previously; add value to existing value\n",
    "            else:\n",
    "                lscounter[k] = v  # No other item rolled-up yet; add L&S-All key, assign value \n",
    "        else:\n",
    "            lscounter[k] = v  # key not in set of keys to be rolled up, guaranteed to be unique within this dict\n",
    "            \n",
    "# Merge dictionaries sacounter1, sacounter2 and sacounter3, eliminating 'None or Unspecified' in 2 and 3\n",
    "\n",
    "d1 = {}\n",
    "d2 = {}\n",
    "d3 = {}\n",
    "\n",
    "for k, v in sacounter1.items():\n",
    "    d1[k] = v\n",
    "    \n",
    "for k, v in sacounter2.items():\n",
    "    d2[k] = v\n",
    "\n",
    "for k, v in sacounter3.items():\n",
    "    d3[k] = v\n",
    "\n",
    "del d2['None or Not specified']\n",
    "del d3['None or Not specified']\n",
    "\n",
    "d = Counter(d1) + Counter(d2) + Counter(d3)\n",
    "sacounter_all = dict(d)\n",
    "\n",
    "# Print results to screen\n",
    "# for h, d in zip(config_20171104.mod_headings, config_20171104.mod_dicts):  # Use this in Pycharm or command-line invocation\n",
    "# for h, d in zip(mod_headings, mod_dicts):\n",
    "#     print('\\n' + h + ':')\n",
    "#     for (k, v) in d.items():\n",
    "#         print(k + ': ' + str(v))     \n",
    "\n",
    "# Uncomment to view data and to debug        \n",
    "# print('\\n')\n",
    "# print(pacounter)\n",
    "# print('\\n')\n",
    "# print(lscounter)\n",
    "# print('\\n')\n",
    "# print(sacounter_all)\n",
    "\n",
    "# Roll-up data-related service areas into 'RDM' - do this for the merged counts for all three service areas.\n",
    "\n",
    "for k, v in sacounter_all.items():\n",
    "    for key in sa_rollup.keys():\n",
    "        if k in sa_rollup[key][1]:\n",
    "            k = sa_rollup[key][0]\n",
    "            if k in sacounter_all_rdm_all.keys():\n",
    "                sacounter_all_rdm_all[k] = (sacounter_all_rdm_all[k] + v)  # RDM key added previously; add value to existing value\n",
    "            else:\n",
    "                sacounter_all_rdm_all[k] = v  # No other item rolled-up yet; add RDM key, assign value \n",
    "        else:\n",
    "            sacounter_all_rdm_all[k] = v  # key not in set of keys to be rolled up, guaranteed unique within this dict\n",
    "\n",
    "# Uncomment to view data and to debug        \n",
    "# print('\\n')\n",
    "# print(sacounter_all_rdm_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With numbers in hand, we prepare data structures to use in visualization and further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For graphing and other analysis, we need to create portable, persistent data structures.\n",
    "# We will do this by creating a list of lists, each list comprising the key/value pairs from one dictionary. \n",
    "# Then we will sort each of those lists and add each to a list of sorted lists\n",
    "\n",
    "# Initialize the lists - unsorted and sorted (the dicts have been created above). \n",
    "# Name each list for the dictionary it will be drawn from.\n",
    "# sacount_all_rdm_all (based on sacounter_all_rdm_all dict) added 2/29/2020\n",
    "ccount, dcount, pcount, sacount1, sacount2, sacount3, scount, rcount, cocount, pacount, \\\n",
    "lscount, sacount_all, sacount_all_rdm_all = [[],[],[],[],[],[],[],[],[],[],[],[],[]]\n",
    "sccount, sdcount, spcount, ssacount1, ssacount2, ssacount3, sscount, srcount, scocount, spacount, \\\n",
    "slscount, ssacount_all, ssacount_all_rdm_all = [[],[],[],[],[],[],[],[],[],[],[],[],[]]  # 's' for sorted\n",
    "\n",
    "# Create containers with the names of our dicts and lists so we can do all the processing in a loop\n",
    "dcts = [ccounter, dcounter, pcounter, sacounter1, sacounter2, sacounter3, scounter, rcounter, cocounter,\\\n",
    "        pacounter, lscounter, sacounter_all, sacounter_all_rdm_all]\n",
    "lsts = [ccount, dcount, pcount, sacount1, sacount2, sacount3, scount, rcount, cocount,\\\n",
    "    pacount, lscount, sacount_all, sacount_all_rdm_all]\n",
    "srtdlsts = [sccount, sdcount, spcount, ssacount1, ssacount2, ssacount3, sscount, srcount, scocount, \\\n",
    "            spacount, slscount, ssacount_all, ssacount_all_rdm_all]\n",
    "\n",
    "# For each dictionary item, append the key/value pair as a tuple to the named list related to the dictionary\n",
    "for d, l, s in zip(dcts, lsts, srtdlsts):\n",
    "    for key in d.keys():\n",
    "        tup = (key, d[key])\n",
    "        l.append(tup)\n",
    "    # Sort each list by the second element in each tuple (the value of the original key/value pair) and add\n",
    "    # the sorted list to list of sorted lists\n",
    "    s.extend(sorted(l, key=lambda x: x[1], reverse=True))  # reverse means descending order\n",
    "    \n",
    "# Uncomment the next lines view data on screen    \n",
    "# for lst in lsts:\n",
    "#     print(lst)\n",
    "#     print('\\n')\n",
    "# for srtdlst in srtdlsts:\n",
    "#     print(srtdlst)\n",
    "#     print('\\n')\n",
    "\n",
    "# Uncomment these print commands to review individual lists by name\n",
    "# print(str(ccount) + '\\n')\n",
    "# print(str(dcount) + '\\n')\n",
    "# print(str(pcount) + '\\n')\n",
    "# print(str(sacount1) + '\\n')\n",
    "# print(str(sacount2) + '\\n')\n",
    "# print(str(sacount3) + '\\n')\n",
    "# print(str(scount) + '\\n')\n",
    "# print(str(rcount) + '\\n')\n",
    "# print(str(cocount) + '\\n')\n",
    "# print(str(pacount) + '\\n')\n",
    "# print(str(lscount) + '\\n')\n",
    "# print(str(sacount_all) + '\\n')\n",
    "# print(str(sacount_all_rdm_all) + '\\n')\n",
    "# print(str(sccount) + '\\n')\n",
    "# print(str(sdcount) + '\\n')\n",
    "# print(str(spcount) + '\\n')\n",
    "# print(str(ssacount1) + '\\n')\n",
    "# print(str(ssacount2) + '\\n')\n",
    "# print(str(ssacount3) + '\\n')\n",
    "# print(str(sscount) + '\\n')\n",
    "# print(str(srcount) + '\\n')\n",
    "# print(str(scocount) + '\\n')\n",
    "# print(str(spacount) + '\\n')\n",
    "# print(str(slscount) + '\\n')\n",
    "# print(str(ssacount_all) + '\\n')\n",
    "# print(str(ssacount_all_rdm_all) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(srtdlsts) #for debugging only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we write these sorted counts (\"tabulations\") to a file? (Of course we can.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "# Create a filename with a simple timestamp so we don't overwrite file each time we write.\n",
    "path = '/Volumes/GoogleDrive/My Drive/RDM Program/Metrics/Metrics_output/'\n",
    "basename = 'rdm_tabulations_'\n",
    "datestr = str(datetime.today())\n",
    "datestr = datestr[:-7]   # Remove microseconds\n",
    "datestr = datestr.replace(' ', 'T')  # Replace space with 'T' so filename doesn't cause problems elsewhere\n",
    "datestr = datestr.replace(':','-')  # The colons in the timestamp may cause problems, so swap them out, too\n",
    "fileoutname = path + basename + datestr + '.txt'\n",
    "#print(filename) #debug\n",
    "\n",
    "with open(fileoutname, 'w') as f:\n",
    "    \n",
    "    f.write('\\nIn %s, RDM Consulting provided %d consultations.\\n\\n' % (report_period_descriptor, (len(myrows))))\n",
    "    \n",
    "    for lst, hdr in zip(srtdlsts,all_headings):\n",
    "        h = hdr\n",
    "        f.write(h + ':\\n\\n')\n",
    "        for item in lst:\n",
    "            k = item[0]\n",
    "            v = item[1]\n",
    "            f.write(k + ': ' + str(v))\n",
    "            f.write('\\n')\n",
    "        f.write('\\n\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot pie charts for each category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't want pie charts for RIT Service Area 2 and RIT Service Area 3 -- those values were rolled up into sacount_all. So, let's eliminate those elements from the list of sorted lists. And let's plot the combined values for all three RIT Service Areas (sacount_all) right after those for RIT Service Area 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "srtdlsts_wo_sa2_sa3 = []  # initialize list where we will alter order of sorted lists for printing purposes\n",
    "\n",
    "for s in srtdlsts:   # Copy (reconstruct) list of sorted lists into this new list, where it will be edited\n",
    "    srtdlsts_wo_sa2_sa3.append(s) \n",
    "\n",
    "sa_all = srtdlsts_wo_sa2_sa3[11]\n",
    "srtdlsts_wo_sa2_sa3.remove(srtdlsts_wo_sa2_sa3[11]) # Remove because we've copied it and will move the values up \n",
    "\n",
    "srtdlsts_wo_sa2_sa3.remove(srtdlsts_wo_sa2_sa3[4])  # Remove values for RIT Service Area 2\n",
    "srtdlsts_wo_sa2_sa3.remove(srtdlsts_wo_sa2_sa3[4])  # With element 4 removed, service area 3 becomes new element 4\n",
    "srtdlsts_wo_sa2_sa3.insert(4, sa_all)  # Insert service_area_all values prior to element 4\n",
    "\n",
    "#print('\\n')    # debug only\n",
    "#print(srtdlsts_wo_sa2_sa3)  # debug only\n",
    "\n",
    "# Now move sorted list ssacount_all_rdm_all (service_area_all values with RDM aggregated) \n",
    "# to follow sorted list of service_area_all\n",
    "ssacount_all_rdm_all = srtdlsts_wo_sa2_sa3[10]\n",
    "srtdlsts_wo_sa2_sa3.remove(srtdlsts_wo_sa2_sa3[10]) # Remove - we've copied ssacount_all_rdm_all, will move values up \n",
    "srtdlsts_wo_sa2_sa3.insert(5, ssacount_all_rdm_all)  # Insert ssacount_all_rdm_all prior to element 5\n",
    "\n",
    "#print('\\n')    # debug only\n",
    "#print(srtdlsts_wo_sa2_sa3)  # debug only\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Pie charts of each category, where the slices will be ordered and plotted counter-clockwise:\n",
    "# Don't need pie charts of ssacount2 and ssacount3\n",
    "\n",
    "# Titles for the figures -- a quick fix. \n",
    "# This configuration shhould probably be set earlier, where we defined other headers, dictionaries, etc.\n",
    "figtitles = ['Consultants','Departments and ORUs served','Researcher status','Service areas (primary values only)',\n",
    "             'Service areas (all values)','Service areas (all values) - RDM combined','Referrals In',\n",
    "             'Hand-offs, Referrals out and Co-consults (to group, unit, or consultant)',\n",
    "             'Complexity','Schools, colleges and divisions served (L&S disaggregated)',\n",
    "             'Schools, colleges and divisions served (L&S aggregated)']\n",
    "for s in srtdlsts_wo_sa2_sa3:  # list of sorted lists without values for RIT service area 2 and 3\n",
    "    \n",
    "    labels = []\n",
    "    sizes = []\n",
    "    labelsplusn = []\n",
    "    \n",
    "    for pair in s:\n",
    "        labels.append(pair[0])\n",
    "        sizes.append(pair[1])\n",
    "\n",
    "    #print(labels) #debug\n",
    "    #print(sizes)  #debug\n",
    "    \n",
    "    for label, size in zip(labels, sizes):\n",
    "        labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "        labelsplusn.append(labelplusn)\n",
    "        \n",
    "    #print(labelsplusn)  #debug\n",
    "    \n",
    "    explode = [.1 for _ in range(len(s))]  # one value for each element in len(sortedlist)\n",
    "    \n",
    "    fig1, ax1 = plt.subplots()\n",
    "    ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=37)  # originally labels=labels\n",
    "    ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "    f = figtitles.pop(0)\n",
    "    fig1.suptitle(f, fontsize=16)\n",
    "        \n",
    "    plt.show()\n",
    "    #print('\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next set of cells generate each pie chart separately so we can customize them. \n",
    "\n",
    "NOTE: These need to be customized based on the date range: the number of arguments in the statement 'explode = ()'\n",
    "must be equal to the number of items in the sorted list, and that will vary depending upon the range of time covered by the reporting period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# Pie chart of schools, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in spacount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "#print(labels) #debug\n",
    "#print(sizes)  #debug\n",
    "#print(len(labels))\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "    \n",
    "explode = (.13, .1, .1, .1, .1, .1, .1, .1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=37)  # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Schools, colleges and divisions served (L&S disaggregated)', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_spacount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of schools with all L&S combined, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in slscount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "#print(labels) #debug\n",
    "#print(sizes)  #debug\n",
    "#print(len(labels))\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.12, .1, .1, .1, .1, .1, .1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=37)  #originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Schools, colleges and divisions served (L&S aggregated)', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_slscount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of original department/oru values, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in sdcount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "#print(labels) #debug\n",
    "#print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (2.1, 1.9, 1.9, 1.8, 1.9, 2.1, 2.3, 2.5, 2.7, 2.9, 3.1, 3.3, 3.5, 3.7, 3.9, 4.3, 4.7, 5.1, 5.5, 10.1, 9.7, 9.5, 9.3, 9.1, 8.9)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Departments and ORUs served', fontsize=16)\n",
    "plt.show()\n",
    "#plt.savefig('pie_sdcount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of position of client on campus, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in spcount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "#print(labels) #debug\n",
    "#print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Researcher status', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_spcount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of RIT Service Area 1 values, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in ssacount1:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "# print(labels) #debug\n",
    "# print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1, .1, .1, .1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Service areas (primary values only)', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_ssacount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of RIT Service Area - All (merged) values, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in ssacount_all:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "# print(labels) #debug\n",
    "# print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1, .1, .1, .1, .1, .1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Service areas (all values)', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_ssacount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of sources (referrals in), where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in sscount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "# print(labels) #debug\n",
    "# print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1, .1, .1, .1, .2, .3)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Referrals in', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_sscount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of referrals out, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in srcount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "# print(labels) #debug\n",
    "# print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1, .1, .1, .1, .2, .3)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=50) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Hand-offs, Referrals out and Co-consults (to group, unit, or consultant)', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_srcount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pie chart of complexity measures, where the slices will be ordered and plotted counter-clockwise:\n",
    "labels = []\n",
    "sizes = []\n",
    "labelsplusn = []\n",
    "\n",
    "for pair in scocount:\n",
    "    labels.append(pair[0])\n",
    "    sizes.append(pair[1])\n",
    "#print(labels) #debug\n",
    "#print(sizes)  #debug\n",
    "\n",
    "for label, size in zip(labels, sizes):\n",
    "    labelplusn = str(label) + ' (' + str(size) + ')'\n",
    "    labelsplusn.append(labelplusn)\n",
    "\n",
    "#print(labelsplusn)  #debug\n",
    "\n",
    "explode = (.1, .1, .1, .1)  \n",
    "\n",
    "fig1, ax1 = plt.subplots()\n",
    "ax1.pie(sizes, explode=explode, labels=labelsplusn, autopct='%1.f%%',\n",
    "        shadow=True, startangle=40) # originally labels=labels\n",
    "ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "fig1.suptitle('Complexity', fontsize=16)\n",
    "\n",
    "plt.show()\n",
    "#plt.savefig('pie_scocount.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the top five values in each category (or all values, if less than five)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for lst, hdr in zip(srtdlsts, all_headings):\n",
    "    print('\\n' + 'Top Five: ' + hdr)\n",
    "    for i in range(5):\n",
    "        if i >= len(lst): continue\n",
    "        (k,v) = lst[i]\n",
    "        print(str(i+1) + '. ' + k + ': ' + str(v))\n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}