mapreader-spatial-humanities-workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MapReader Spatial Humanities Workshop (2024)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up for google colab - this cell will take a while to run!\n",
    "!git clone https://github.com/Living-with-machines/spatial-humanities-mapreader-workshop\n",
    "!pip install mapreader[dev]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enable custom widgets in colab\n",
    "from google.colab import output\n",
    "output.enable_custom_widget_manager()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download maps\n",
    "\n",
    "https://mapreader.readthedocs.io/en/latest/using-mapreader/step-by-step-guide/1-download.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this workshop, we have pre-selected and downloaded two maps from the OS 6-inch 1st edition map series.\n",
    "These were downloaded from a tile layer hosted by the NLS. You can find more information about NLS tile layers [here](https://maps.nls.uk/).\n",
    "\n",
    "The two maps and their metadata are saved in the `maps` directory of the `spatial-humanities-mapreader-workshop` repository (which we cloned earlier).\n",
    "\n",
    "We will use one to demonstrate annotating and training, and the other to demonstrate inference.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load maps and patchify\n",
    "\n",
    "https://mapreader.readthedocs.io/en/latest/using-mapreader/step-by-step-guide/2-load.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now load both maps and their metadata using the `loader`.\n",
    "\n",
    "From here, we can patchify our maps, visualise metadata and add further information about our maps/patches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mapreader import loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps = loader(\"./spatial-humanities-mapreader-workshop/maps/*png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps.add_metadata(\"./spatial-humanities-mapreader-workshop/maps/metadata.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_maps) # see which maps you have loaded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Patchify maps\n",
    "\n",
    "Choosing a patch size is an important part of using MapReader.\n",
    "\n",
    "Before patchifying our maps, we need to think about which visual features we want to find in our maps. This will help us pick a suitable patch size.\n",
    "\n",
    "Types of features we might want to label are:\n",
    "\n",
    "- Continuous features (e.g. roads, rivers, lines/patterns/shading)\n",
    "- Discrete features (e.g. buildings, trees, other symbology)\n",
    "- Abstract or composite concepts features (e.g. farmland, urban/rural areas)\n",
    "\n",
    "The patch size should be large enough to distinguish our visual features but small enough to get useful results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have added metadata to our maps, we have information about their coordinates and so can use the \"meters\" method to patchify our maps.\n",
    "\n",
    "We will be annotating buildings and will slice our maps into 100x100 meter patches for this workshop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps.patchify_all(method=\"meters\", patch_size=100) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you now look in your files you will see a `patches_100_meters` directory which contains all the patches of your two maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_maps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show a sample of the patches\n",
    "my_maps.show_sample(num_samples=3, tree_level=\"patch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our model will be looking at pixel values to classify our patches, it can be useful to know some statistics about these. \n",
    "\n",
    "- The mean pixel value for a patch gives an idea of the average \"brightness\" of the patch, with higher values indicating lighter patches so more empty/white space. \n",
    "- The standard deviation of pixel values gives an idea of the variation in pixel values across the patch. Higher values indicate more variation in pixel values.\n",
    "\n",
    "We can calculate these statistics using the `calc_pixel_stats()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps.calc_pixel_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at what information we have about each of our parent images and patches (including the pixel statistics we just calculated).\n",
    "\n",
    "The easiest way to do this is to create dataframes containing parent and patch information using the `convert_images()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_df, patch_df = my_maps.convert_images()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_df.head() # parent information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_df.head() # patch information (showing only first 5 rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Annotate\n",
    "\n",
    "https://mapreader.readthedocs.io/en/latest/using-mapreader/step-by-step-guide/3-annotate.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mapreader import Annotator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we begin annotating, we need to set up our annotation task by specifying labels, a task name and a username for the person annotating.\n",
    "\n",
    "There are two options for picking good labels:\n",
    "- Binary labels: e.g. \"building\" and \"not building\"/\"no\"\n",
    "- Multi-class labels (but these must be mutually exclusive!): e.g. \"building\", \"road\", \"building and road\" and \"neither building nor road\"/\"no\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">__*NOTE*__: You can change the labels in the cell below if you'd like to annotate something else!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "task_name = \"workshop\" \n",
    "labels = [\"building\", \"no\"] # change these to the labels you want to use\n",
    "username = \"rosie\" # change this to your username"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we only want to annotate one map, we will filter our patch dataframe for only patches with \"map_74427695.png\" as their `parent_id`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patches_to_annotate = patch_df[patch_df[\"parent_id\"]==\"map_74427695.png\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator = Annotator(\n",
    "    patches_to_annotate, # the information about our patches\n",
    "    parent_df, # the information about our parent images\n",
    "    task_name=task_name, \n",
    "    labels=labels,\n",
    "    username=username,\n",
    "    resize_to=300, # resize the patches to 300x300 pixels in the annotation interface\n",
    "\t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will annotate with no context image.\n",
    "This is representative of what the model sees during training so can be helpful for understanding what visual features are a good choice for labelling.\n",
    "\n",
    "> Do as many annotations as you want here (we will do more annotations in the cell below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator.annotate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(annotator.get_labelled_data())) # see number of annotations\n",
    "annotator.get_labelled_data() # see annotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make annotating easier, it can be helpful to see the patch in its surrounding context.\n",
    "This is done by setting `show_context=True`. \n",
    "\n",
    "We will now have a go at this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator.annotate(show_context=True, resize_to=600) # show the context of the patch and resize to 600x600 pixels to make it easier to see"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(annotator.get_labelled_data())) # see number of annotations\n",
    "annotator.get_labelled_data() # new annotations are added to the existing ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you now look in your files you will see an `annotations` directory containing a CSV file with your annotations.\n",
    "This file is auto-saved and updated each time you add new annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotator.annotations_file # see the path to the annotations file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your model\n",
    "\n",
    "https://mapreader.readthedocs.io/en/latest/using-mapreader/step-by-step-guide/4-classify/train.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no definite answer to how many annotations you need to train a model.\n",
    "However, the more annotations you have, the better your model will likely be - e.g. for our railspace/building models we had 62K annotations!\n",
    "\n",
    "> __*NOTE:*__ It is very unlikely you will have time to do enough annotations in the workshop to create a great model, but you should have enough to see some results.\n",
    "\n",
    "Here are some tips for getting the most out of your annotations:\n",
    "\n",
    "- Start by annotating a representative sample of your data.\n",
    "- In most experiments, patch-level classification is quick and so we can iteratively check the performance of our models after training. i.e. Do some annotations, train your model, visually inspect results to identify systematic errors, do some targeted annotations to fix these errors, then repeat the training process and so on.\n",
    "- Use contextual information - If you expect a certain pattern in neighboring patches you can use that information to identify and locate possible errors.\n",
    "- Use external datasets (e.g., StopsGB and railway patches)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mapreader import AnnotationsLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations = AnnotationsLoader() # initialise\n",
    "\n",
    "annotations.load(\n",
    "    annotator.annotations_file, # path to the annotations file\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During model training labels must be integers instead of strings so the `AnnotationsLoader` will create a mapping between labels and label indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations.labels_map # the mapping between the labels and label indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fine-tune our model, we will split our annotations into train (70%), validation (15%) and test (15%) datasets. \n",
    "\n",
    "- The train dataset is used to train the model (i.e. to update the model parameters).\n",
    "- The validation set is used to evaluate the model's performance during training (but not to update the model parameters).\n",
    "- The test set is unseen data reserved for us to use to evaluate the model's performance after training.\n",
    "\n",
    "e.g. if you have 100 annotations, the train dataset will have 70 annotations, the validation dataset will have 15 annotations and the test dataset will have 15 annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations.create_datasets() # create the datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to have __*at least*__ one instance of each label in each dataset (ideally you'd want a lot more than this, but for the workshop it should be fine).\n",
    "\n",
    "You can check this using the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for set_name, dataset in annotations.datasets.items():\n",
    "    print(f\"Number of instances of each label in '{set_name}':\\n{dataset.patch_df['label'].value_counts().to_dict()}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Continue to training the model if you feel you have enough annotations. If not, go back and add some more using the cells above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataloaders = annotations.create_dataloaders() # create the dataloaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up and train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mapreader import ClassifierContainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier = ClassifierContainer(\n",
    "    \"resnet18\", # the model architecture, choose from https://pytorch.org/vision/0.8/models.html\n",
    "    labels_map=annotations.labels_map,\n",
    "    dataloaders=dataloaders,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.add_loss_fn(\"cross-entropy\") # add the loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.initialize_optimizer(\"adam\") # add the optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.initialize_scheduler() # add the scheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can actually train the model, we will start with 10 epochs (1 epoch = 1 full pass through the training data):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the model\n",
    "my_classifier.train(num_epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you now look in your files you will see a `models` directory containing your model files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize progress\n",
    "\n",
    "MapReader logs a number of common metrics during model training/evaluation and saves them in a dictionary ``my_classifier.metrics``.\n",
    "For example:\n",
    "- loss, calculated using the loss function we defined earlier (i.e. cross-entropy)\n",
    "- f-scores\n",
    "- precision scores\n",
    "- recall scores\n",
    "\n",
    "[This page](https://cohere.com/blog/classification-eval-metrics) provides a good overview of what each of these scores mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = \"loss\" # choose the metric to plot\n",
    "\n",
    "my_classifier.plot_metric(\n",
    "    y_axis=[f\"epoch_{metric}_train\", f\"epoch_{metric}_val\"], # plot for training and validation sets\n",
    "    y_label=metric,\n",
    "    legends=[f\"epoch_{metric}_train\", f\"epoch_{metric}_val\"],\n",
    "    ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Infer\n",
    "\n",
    "https://mapreader.readthedocs.io/en/latest/using-mapreader/step-by-step-guide/4-classify/infer.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to create a new dataset containing all our patches, including the ones from the map we didn't annotate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mapreader import PatchDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_dataset = PatchDataset(\n",
    "    patch_df,\n",
    "    transform=\"test\", # apply the test transform on the patches\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.load_dataset(patch_dataset, set_name=\"all_patches\") # load the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use our fine-tuned model to predict the labels on the rest of our patches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.inference(set_name=\"all_patches\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_classifier.save_predictions(\"all_patches\") # save the predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> If you now look in your files you will see a file called ``all_patches_predictions_patch_df.csv`` which contains predictions for each patch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load the predictions as metadata in our `my_maps` object. \n",
    "This makes it easy to visualize our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps.add_metadata(\n",
    "    \"./all_patches_predictions_patch_df.csv\", \n",
    "    tree_level=\"patch\" # add the predictions as patch metadata\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualise for the map we partly annotated\n",
    "my_maps.show_parent(\n",
    "    \"map_74427695.png\",\n",
    "    column_to_plot=\"pred\",\n",
    "    vmin=0,\n",
    "    vmax=1,\n",
    "    alpha=0.5, # set the transparency\n",
    "    patch_border=False, # don't show borders around patches\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualise for a completely new map\n",
    "my_maps.show_parent(\n",
    "    \"map_74427696.png\",\n",
    "    column_to_plot=\"pred\",\n",
    "    vmin=0,\n",
    "    vmax=1,\n",
    "    alpha=0.5, # set the transparency\n",
    "    patch_border=False, # don't show borders around patches\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save our results as CSVs, or in GEOJSON format for use in GIS software:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_df, patch_df = my_maps.convert_images(save=True, save_format=\"csv\") # as CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_maps.save_patches_to_geojson(\"predicted_outputs.geojson\", rewrite=True) # as GeoJSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mr_py312",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}