Development

.gitignore

@@ -17,6 +17,11 @@ secret.py
 *.sh
 !/data/more_gtfs_sources.csv
 !/data/cleaned_sources.csv
+/notebooks/cache/*
+/notebooks/checkpoints/*
+/notebooks/files/*
+/notebooks/lightning_logs/*
+*/*.zip
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

README.md

@@ -1,4 +1,35 @@
-# open_bus_tools
+# Open Bus Tools
 Tools for open bus data collection, travel time prediction, energy consumption and visualization.
 
-If not installed as package, add [export PYTHONPATH="${PYTHONPATH}:/my/other/path"] to .bashrc
+This is companion code for a dissertation in Civil and Environmental Engineering at the University of Washington. It will be distributed as a Python package when completed.
+
+If not installed as package, you may need to add [export PYTHONPATH="${PYTHONPATH}:/my/other/path/to/open_bus_tools"] to .bashrc to run the scripts.
+
+### Workflow
+There are two folders in the base directory /scripts and /notebooks. These are useful starting points for understanding how different parts of the code work together. The two main modules of the package are:
+* drivecycle: Used to estimate energy use based on vehicle and driving profile from a given actual or predicted GPS trajectory.
+
+* traveltime: Used to predict travel times using ML-based geospatial data mining.
+
+The /webscraper folder has code used to gather GTFS and GTFS-RT feeds. These data are used in training the travel time models.
+
+### Motivation
+Standardized and open source bus data including static schedules and realtime positions have become widely available in public web APIs. Though these data primarily underly popular map-based trip planning applications, they also enable new analyses in understanding, forecasting and improving bus operations across agencies and cities. Due to their lower resolution and simpler, anonymized features, these data are more challenging to work with than those of the underlying sensors, making them unfavorable for direct operations analysis by transit agencies. However, the wide scale of these data and their open source nature make them a valuable resource for researchers and planners.
+
+This work develops and tests a set of tools for analyzing bus operations using open source data. Central to this endeavor is the ongoing collection of a multi-year dataset from the King County Metro transit network in Seattle, Washington, rapidly approaching one billion tracked bus locations. First, these data are used in basic roadway segment matching and aggregation to visualize the spatiotemporal dynamics of different types of delays in the transit system. The findings are used to identify locations in the network where targeted transit priority treatments can generate the greatest benefit. Then a set of deep learning models and baselines are developed to forecast bus travel times under different data availability scenarios. Their generalizability is tested across different cities and transit networks. Finally, these models are used to estimate the drive cycle energy demands of a system-wide bus electrification project, and are validated using results of a more traditional approach with direct sensor data.
+
+### Past Work
+This work will replace code in these repositories.
+* [ML-Based Bus Travel Time Models](https://github.com/zackAemmer/generalizable_travel_times)
+* [Bus Performance Visualization Tool](https://github.com/zackAemmer/transit_vis)
+* [TransitVis Web-App](https://www.transitvis.com)
+
+### Data Sample
+The full dataset is stored on Amazon S3 and downloaded for local analysis as needed, please feel free to reach out if you would like a sample.
+
+### Other Libraries and Code You Might Find Useful
+* https://github.com/kraina-ai/srai
+* https://github.com/NREL/fastsim
+* https://github.com/smohiudd/drivecycle
+* https://github.com/EricaEgg/Route_Dynamics
+* https://github.com/mrcagney/gtfs_kit

data/cleaned_sources.csv

@@ -17,7 +17,6 @@ US,Georgia,Atlanta,Metropolitan Atlanta Rapid Transit Authority (MARTA),https://
 US,Illinois,Springfield,Springfield Mass Transit District (SMTD),http://data.smtd.org/gtfs/smtd_gtfs_feed.zip,http://ride.smtd.org/gtfsrt/vehicles,-89.75837,-89.527032,39.667827,39.898621,dba09c2c-6433-4924-acdd-61745c439460,America/Chicago,32616
 US,Wisconsin,Madison,Metro Transit,http://transitdata.cityofmadison.com/GTFS/mmt_gtfs.zip,http://transitdata.cityofmadison.com/Vehicle/VehiclePositions.pb,-89.563863,-89.244818,42.987718,43.176537,38a8b09e-078a-491f-b4bf-4b001ee51a61,America/Chicago,32616
 US,Ohio,Cleveland,Greater Cleveland Regional Transit Authority,http://www.riderta.com/sites/default/files/gtfs/latest/google_transit.zip,http://gtfs.gcrta.org/TMGTFSRealTimeWebService/Vehicle/VehiclePositions.pb,-81.96736,-81.437811,41.227828,41.637051,1656b240-d1ee-45ac-afc4-0bf17f4b4466,America/New_York,32617
-US,Pennsylvania,Pittsburgh,Port Authority of Allegheny County,https://www.portauthority.org/developerresources/GTFS.zip,http://truetime.portauthority.org/gtfsrt-train/vehicles,-80.258865,-79.70545,40.273078,40.667597,2f18ef13-fd56-4e71-94ce-e96973a0175c,America/New_York,32617
 US,Pennsylvania,Pittsburgh,Port Authority of Allegheny County,https://www.portauthority.org/developerresources/GTFS.zip,https://truetime.portauthority.org/gtfsrt-bus/vehicles,-80.258865,-79.70545,40.273078,40.667597,ead6fd88-9158-44c2-b71f-cd093a358072,America/New_York,32617
 US,Massachusetts,Boston,Massachusetts Bay Transportation Authority (MBTA),https://cdn.mbta.com/MBTA_GTFS.zip,https://cdn.mbta.com/realtime/VehiclePositions.pb,-71.848488,-70.276583,41.581289,42.797837,1287a85e-ca1c-47d3-b51f-d58fa9bd6784,America/New_York,32619
 US,Virginia,Arlington,Arlington Transit,https://www.arlingtontransit.com/shared/content/gtfs/art/google_transit.zip,https://realtime.arlingtontransit.com/gtfsrt/vehicles,-77.162277,-77.049105,38.839365,38.925147,1161e355-c52c-4925-a92b-1eabc12d0153,America/New_York,32618
@@ -44,7 +43,6 @@ CA,Québec,La Vallée-du-Richelieu,Exo Vallée du Richelieu,https://exo.quebec/x
 CA,Québec,Sainte-Julie,Exo Ville de Sainte-Julie,https://exo.quebec/xdata/omitsju/google_transit.zip,https://opendata.exo.quebec/ServiceGTFSR/VehiclePosition.pb?agency=OMITSJU,-73.56448,-73.288934,45.497418,45.63169,8f392db0-4b1b-4b7f-8036-4fb7504bb196,America/Toronto,32618
 CA,Québec,L'Assomption,Exo L'Assomption,https://exo.quebec/xdata/mrclasso/google_transit.zip,https://opendata.exo.quebec/ServiceGTFSR/VehiclePosition.pb?agency=MRCLASSO,-73.539828,-73.277404,45.589483,45.901438,9523f994-7c76-4cfc-ba9b-2b5751f90f28,America/Toronto,32618
 CA,British Columbia,Port Alberni,Port Alberni,https://bct.tmix.se/Tmix.Cap.TdExport.WebApi/gtfs/?operatorIds=11,https://bct.tmix.se/gtfs-realtime/vehicleupdates.pb?operatorIds=11,-124.85047,-124.78265,49.21847,49.27743,16c4e44f-108d-4e04-80ab-b305093f1328,America/Vancouver,32610
-FI,Keski-Suomi,Jyväskylä,"Jyväskylän Liikenne Oy, Koivuranta Oy, Uuraisten Liikenne Ky, Savo-Karjalan Linja Oy, Linja-Karjala Oy, Oulun taksipalvelut Oy, Oubus Oy, Suorsan Liikenne Ky, Koskilinjat Oy, Oy Pohjolan liikenne Ab, Kuopion Liikenne Oy, Autolinjat Oy, ML-Charter Oy, Etelä-Suomen Linjaliikenne Oy, Linjaliikenne Martti Laurila Oy, Kari Väisänen Ky, Matkatoimisto Matka-Majuri Ky, Linja-autoliikenne P. Puolakka Ky, Elimäen Liikenne Oy, Anjalankosken Linja Oy, Savonlinja Oy, Liikenne M. Heikura Oy, Jääskeläisen Auto Oy, Kilpailutus käynnissä, TILAUSAJOT MENNÄÄN BUSSILLA OY, Revon Turistiliikenne Oy, Matkamestarit Oy, R-Kioski, A. Valppu Oy, Invataksi Niemi Oy, Oy Wiik & Ström Ab, Oravaisten Liikenne Oy, Vaasan Paikallisliikenne Oy, Koiviston Auto Oy, Lehtimäen Liikenne Oy, Järvisen Liikenne Oy, Niemisen Linjat Oy, Bus Travel Oy Reissu Ruoti, Taksi- ja linja-autoliikenne Timo Lahtinen, SL-Autoyhtymä Oy, Linja-autoliike E Ahonen Ky, Lehdon Liikenne Oy, HämeBus Oy, Tuuri Jaakko Juhani, Pekolan Liikenne Oy, Mikkolan Liikenne Oy, Vekka Liikenne Oy, Inter Kuljetus Oy, Valkeakosken Liikenne Oy, Matkahuolto Oy, ELY, Ventoniemi Oy, Yhdysliikenne Oy, MIKA K. NISKANEN OY, Kantamatkat, LINJA-AUTOLIIKE YRJÖ MAKKONEN & KUMPP, SOISALON LIIKENNE OY, Pohjolan Matka, HENKILÖKULJETUS RISSANEN OY, V.Alamäki Oy, Kai Heikkinen Oy, Savo Kainuun Tilausmatkat Ky, Kajaanin Tila-autot Oy, A Kyllönen Oy, Puolangan Bussipalvelu Ky, Kainuun Tilausliikenne P. Jääskeläinen Ky, Matka-Kyllönen Oy, Jyrkilä Oy, Liikenne Vuorela Oy, Pyhtään Kuljetuspalvelut Oy, Erkki Itkonen Oy, Ihastjärven Linja Oy, Tilausliikenne Himanen, Mikkelin Palvelutaksit Oy, Tuplataxi, Matka Mäkelä Oy, Tilausliikenne Hokkinen Ky, S. J. TOIVONEN OY, E. RANTANEN OY, Kuljetus Mikkonen KY, Linja-autoliike Veljekset Laitinen Oy, TILAUSLIIKENNE HÄNNINEN OY, JOUTSENON AUTO- JA KULJETUSPALVELU OY, TOIMI VENTO KY, NINA´S ERIKOISTAKSIPALVELUT OY, Tarkiainen Matti Olavi, Anssin Tilausliikenne Oy, Rautalammin Auto Oy, Oy Rytkönen & Co, SAIMAAN TURISTILIIKENNE OY, Studio Ticket Pusatec-DO-NOT-USE, Studio Ti",https://tvv.fra1.digitaloceanspaces.com/209.zip,https://data.waltti.fi/jyvaskyla/api/gtfsrealtime/v1.0/feed/vehicleposition,25.3666653747545,26.4944285887841,61.7425528275114,62.62864111217454,cd173e54-d152-4aaf-9c45-ede279ffbc98,Europe/Helsinki,32635
 US,California,Commerce,Commerce Municipal Bus Lines,https://citycommbus.com/gtfs,https://citycommbus.com/gtfs-rt/vehiclepositions,-118.264252,-118.127642,33.965568,34.063075,97b3ab44-c33b-4ac8-9468-cfe582f201ca,America/Los_Angeles,32611
 FI,Päijät-Häme,Lahti,"Haarasilta Toivo Samuli, Järvisen Liikenne Oy, Koiviston Auto Oy, Lehtimäen Liikenne Oy, Bus Travel Oy Reissu Ruoti, Tilausliikenne Kuisma Ky",https://tvv.fra1.digitaloceanspaces.com/223.zip,https://data.waltti.fi/lahti/api/gtfsrealtime/v1.0/feed/vehicleposition,25.2492841801289,26.2609529033383,60.6772517226169,61.5800619928604,8ccc2bab-ae57-44b9-b989-e6dbae4d41ed,Europe/Helsinki,32635
 FI,Pohjois-Karjala,Joensuu,"Linja-Karjala Oy, Savo-Karjalan Linja Oy",https://tvv.fra1.digitaloceanspaces.com/207.zip,https://data.waltti.fi/joensuu/api/gtfsrealtime/v1.0/feed/vehicleposition,28.924184940398234,30.936747,62.092129,63.325670884853885,ff767665-d709-43a5-a91a-f301c076b7f8,Europe/Helsinki,32635