diff --git a/README.md b/README.md index d888e5e7..efcae9a6 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,88 @@ **DELTA** (Deep Earth Learning, Tools, and Analysis) is a framework for deep learning on satellite imagery, -based on Tensorflow. Use DELTA to train and run neural networks to classify large satellite images. DELTA -provides pre-trained autoencoders for a variety of satellites to reduce required training data -and time. +based on Tensorflow. DELTA classifies large satellite images with neural networks, automatically handling +tiling large imagery. DELTA is currently under active development by the -[NASA Ames Intelligent Robotics Group](https://ti.arc.nasa.gov/tech/asr/groups/intelligent-robotics/). Expect -frequent changes. It is initially being used to map floods for disaster response, in collaboration with the +[NASA Ames Intelligent Robotics Group](https://ti.arc.nasa.gov/tech/asr/groups/intelligent-robotics/). +Initially, it is mapping floods for disaster response, in collaboration with the [U.S. Geological Survey](http://www.usgs.gov), [National Geospatial Intelligence Agency](https://www.nga.mil/), [National Center for Supercomputing Applications](http://www.ncsa.illinois.edu/), and -[University of Alabama](https://www.ua.edu/). DELTA is a component of the -[Crisis Mapping Toolkit](https://github.com/nasa/CrisisMappingToolkit), in addition -to our previous software for mapping floods with Google Earth Engine. +[University of Alabama](https://www.ua.edu/). Installation ============ -1. Install [python3](https://www.python.org/downloads/), [GDAL](https://gdal.org/download.html), and the [GDAL python bindings](https://pypi.org/project/GDAL/). - For Ubuntu Linux, you can run `scripts/setup.sh` from the DELTA repository to install these dependencies. +1. Install [python3](https://www.python.org/downloads/), [GDAL](https://gdal.org/download.html), + and the [GDAL python bindings](https://pypi.org/project/GDAL/). For Ubuntu Linux, you can run + `scripts/setup.sh` from the DELTA repository to install these dependencies. -2. Install Tensorflow with pip following the [instructions](https://www.tensorflow.org/install). For +2. Install Tensorflow following the [instructions](https://www.tensorflow.org/install). For GPU support in DELTA (highly recommended) follow the directions in the [GPU guide](https://www.tensorflow.org/install/gpu). 3. Checkout the delta repository and install with pip: - ``` - git clone http://github.com/nasa/delta - python3 -m pip install delta - ``` +```bash +git clone http://github.com/nasa/delta +python3 -m pip install delta +``` + +DELTA is now installed and ready to use! + +Documentation +============= +DELTA can be used either as a command line tool or as a python library. +See the python documentation for the master branch [here](https://nasa.github.io/delta/), +or generate the documentation with `scripts/docs.sh`. + +Example +======= + +As a simple example, consider training a neural network to map clouds with Landsat-8 images. +The script `scripts/example/l8_cloud.sh` trains such a network using DELTA from the +[USGS SPARCS dataset](https://www.usgs.gov/core-science-systems/nli/landsat/spatial-procedures-automated-removal-cloud-and-shadow-sparcs), +and shows how DELTA can be used. The steps involved in this, and other, classification processes are: + +1. **Collect** training data. The SPARCS dataset contains Landsat-8 imagery with and without clouds. - This installs DELTA and all dependencies (except for GDAL which must be installed manually in step 1). +2. **Label** training data. The SPARCS labels classify each pixel according to cloud, land, water and other classes. -Usage -===== +3. **Train** the neural network. The script `scripts/example/l8_cloud.sh` invokes the command -As a simple example, consider training a neural network to map water in Worldview imagery. -You would: + ``` + delta train --config l8_cloud.yaml l8_clouds.h5 + ``` -1. **Collect** training data. Find and save Worldview images with and without water. For a robust - classifier, the training data should be as representative as possible of the evaluation data. + where `scripts/example/l8_cloud.yaml` is a configuration file specifying the labeled training data and + training parameters (learn more about configuration files below). A neural network file + `l8_clouds.h5` is output. -2. **Label** training data. Create images matching the training images pixel for pixel, where each pixel - in the label is 0 if it is not water and 1 if it is. +4. **Classify** with the trained network. The script runs -3. **Train** the neural network. Run - ``` - delta train --config wv_water.yaml wv_water.h5 - ``` - where `wv_water.yaml` is a configuration file specifying the labeled training data and any - training parameters (learn more about configuration files below). The command will output a - neural network file `wv_water.h5` which can be - used for classification. The neural network operates on the level of *chunks*, inputting - and output smaller blocks of the image at a time. + ``` + delta classify --config l8_cloud.yaml --image-dir ./validate --overlap 32 l8_clouds.h5 + ``` -4. **Classify** with the trained network. Run - ``` - delta classify --image image.tiff wv_water.h5 - ``` - to classify `image.tiff` using the network `wv_water.h5` learned previously. - The file `image_predicted.tiff` will be written to the current directory showing the resulting labels. + to classify the images in the `validate` folder using the network `l8_clouds.h5` learned previously. + The overlap tiles to ignore border regions when possible to make a more aesthetically pleasing classified + image. The command outputs a predicted image and confusion matrix. -Configuration Files -------------------- +The results could be improved--- with more training, more data, an improved network, or more--- but this +example shows the basic usage of DETLA. -DELTA is configured with YAML files. Some options can be overwritten with command line options (use -`delta --help` to see which). [Learn more about DELTA configuration files](./delta/config/README.md). +Configuration and Extensions +============================ -All available configuration options and their default values are shown [here](./delta/config/delta.yaml). -We suggest that users create one reusable configuration file to describe the parameters specific -to each dataset, and separate configuration files to train on or classify that dataset. +DELTA provides many options for customizing data inputs and training. All options are configured via +YAML files. Some options can be overwritten with command line options (use +`delta --help` to see which). See the `delta.config` README to learn about available configuration +options. -Supported Image Formats ------------------------ -DELTA supports tiff files and a few other formats. -Users can extend DELTA with their own custom formats. We are looking to expand DELTA to support other -useful file formats. +DELTA can be extended to support custom neural network layers, image types, preprocessing operations, metrics, losses, +and training callbacks. Learn about DELTA extensions in the `delta.config.extensions` documentation. -MLFlow ------- +Data Management +============= DELTA integrates with [MLFlow](http://mlflow.org) to track training. MLFlow options can be specified in the corresponding area of the configuration file. By default, training and @@ -93,18 +98,6 @@ View all the logged training information through mlflow by running:: and navigating to the printed URL in a browser. This makes it easier to keep track when running experiments and adjusting parameters. -Using DELTA from Code -===================== -You can also call DELTA as a python library and customize it with your own extensions, for example, -custom image types. The python API documentation can be generated as HTML. To do so: - -``` - pip install pdoc3 - ./scripts/docs.sh -``` - -Then open `html/delta/index.html` in a web browser. - Contributors ============ We welcome pull requests to contribute to DELTA. However, due to NASA legal restrictions, we must require diff --git a/delta/config/README.md b/delta/config/README.md index 598d78fa..db2b7e08 100644 --- a/delta/config/README.md +++ b/delta/config/README.md @@ -5,9 +5,33 @@ all options, showing all parameters DELTA and their default values, see [delta.y `delta` accepts multiple config files on the command line. For example, run - delta train --config dataset.yaml --config train.yaml +```bash +delta train --config dataset.yaml --config train.yaml +``` + +to train on a dataset specified by `dataset.yaml`: + +```yaml +dataset: + images: + type: tiff + directory: train/ + labels: + type: tiff + directory: labels/ + classes: 2 +``` + +with training parameters given in `train.yaml`: + +```yaml +train: + network: + model: + yaml_file: networks/convpool.yaml + epochs: 10 +``` -to train on a dataset specified by `dataset.yaml` with training parameters given in `train.yaml`. Parameters can be overriden globally for all runs of `delta` as well, by placing options in `$HOME/.config/delta/delta.yaml` on Linux. This is only recommended for global parameters such as the cache directory. @@ -17,8 +41,7 @@ only setting the necessary options. Note that some configuration options can be overwritten on the command line: run `delta --help` to see which. -The remainder of this document details the available configuration parameters. Note that -DELTA is still under active development and parts are likely to change in the future. +The remainder of this document details the available configuration parameters. Dataset ----------------- @@ -26,70 +49,129 @@ Images and labels are specified with the `images` and `labels` fields respective within `dataset`. Both share the same underlying options. - * `type`: Indicates which loader to use, e.g., `tiff` for geotiff. + * `type`: Indicates which `delta.imagery.delta_image.DeltaImage` image reader to use, e.g., `tiff` for geotiff. + The reader should previously be registered with `delta.config.extensions.register_image_reader`. * Files to load must be specified in one of three ways: - * `directory` and `extension`: Use all images in the directory ending with the given extension. - * `file_list`: Provide a text file with one image file name per line. - * `files`: Provide a list of file names in yaml. - * `preprocess`: Supports limited image preprocessing. We recommend + * `directory` and `extension`: Use all images in the directory ending with the given extension. + * `file_list`: Provide a text file with one image file name per line. + * `files`: Provide a list of file names in yaml. + * `preprocess`: Specify a preprocessing chain. We recommend scaling input imagery in the range 0.0 to 1.0 for best results with most of our networks. DELTA also supports custom preprocessing commands. Default actions include: - * `scale` with `factor` argument: Divide all values by amount. - * `offset` with `factor` argument: Add `factor` to pixel values. - * `clip` with `bounds` argument: clip all pixels to bounds. + * `scale` with `factor` argument: Divide all values by amount. + * `offset` with `factor` argument: Add `factor` to pixel values. + * `clip` with `bounds` argument: clip all pixels to bounds. + Preprocessing commands are registered with `delta.config.extensions.register_preprocess`. + A full list of defaults (and examples of how to create new ones) can be found in `delta.extensions.preprocess`. * `nodata_value`: A pixel value to ignore in the images. + * `classes`: Either an integer number of classes or a list of individual classes. If individual classes are specified, + each list item should be the pixel value of the class in the label images, and a dictionary with the + following potential attributes (see example below): + * `name`: Name of the class. + * `color`: Integer to use as the RGB representation for some classification options. + * `weight`: How much to weight the class during training (useful for underrepresented classes). As an example: - ``` - dataset: - images: - type: worldview - directory: images/ - labels: - type: tiff - directory: labels/ - extension: _label.tiff - ``` - -This configuration will load worldview files ending in `.zip` from the `images/` directory. +```yaml +dataset: + images: + type: tiff + directory: images/ + preprocess: + - scale: + factor: 256.0 + nodata_value: 0 + labels: + type: tiff + directory: labels/ + extension: _label.tiff + nodata_value: 0 + classes: + - 1: + name: Cloud + color: 0x0000FF + weight: 2.0 + - 2: + name: Not Cloud + color: 0xFFFFFF + weight: 1.0 +``` + +This configuration will load tiff files ending in `.tiff` from the `images/` directory. It will then find matching tiff files ending in `_label.tiff` from the `labels` directory -to use as labels. +to use as labels. The image values will be divied by a factor of 256 before they are used. +(It is often helpful to scale images to a range of 0-1 before training.) The labels represent two classes: +clouds and non-clouds. Since there are fewer clouds, these are weighted more havily. The label +images should contain 0 for nodata, 1 for cloud pixels, and 2 for non-cloud pixels. Train ----- These options are used in the `delta train` command. - * `network`: The nueral network to train. See the next section for details. + * `network`: The nueral network to train. One of `yaml_file` or `layers` must be specified. + * `yaml_file`: A path to a yaml file with only the params and layers fields. See `delta/config/networks` + for examples. + * `params`: A dictionary of parameters to substitute in the `layers` field. + * `layers`: A list of layers which compose the network. See the following section for details. * `stride`: When collecting training samples, skip every `n` pixels between adjacent blocks. Keep the - default of 1 to use all available training data. - * `batch_size`: The number of chunks to train on in a group. May affect convergence speed. Larger - batches allow higher training data throughput, but may encounter memory limitations. + default of ~ or 1 to use all available training data. Not used for fully convolutional networks. + * `batch_size`: The number of patches to train on at a time. If running out of memory, reducing + batch size may be helpful. * `steps`: If specified, stop training for each epoch after the given number of batches. * `epochs`: the number of times to iterate through all training data during training. * `loss`: [Keras loss function](https://keras.io/losses/). For integer classes, use - `sparse_categorical_cross_entropy`. - * `metrics`: A list of [Keras metrics](https://keras.io/metrics/) to evaluate. - * `optimizer`: The [Keras optimizer](https://keras.io/optimizers/) to use. + `sparse_categorical_cross_entropy`. May be specified either as a string, or as a dictionary + with arguments to pass to the loss function constructor. Custom losses registered with + `delta.config.extensions.register_loss` may be used. + * `metrics`: A list of [Keras metrics](https://keras.io/metrics/) to evaluate. Either the string + name or a dictionary with the constructor arguments may be used. Custom metrics registered with + `delta.config.extensions.register_metric` or loss functions may also be used. + * `optimizer`: The [Keras optimizer](https://keras.io/optimizers/) to use. May be specified as a string or + as a dictionary with constructor parameters. + * `callbacks`: A list of [Keras callbacks)(https://keras.io/api/callbacks/) to use during training, specified as + either a string or as a dictionary with constructor parameters. Custom callbacks registered with + `delta.config.extensions.register_metric` may also be used. * `validation`: Specify validation data. The validation data is tested after each epoch to evaluate the classifier performance. Always use separate training and validation data! * `from_training` and `steps`: If `from_training` is true, take the `steps` training batches and do not use it for training but for validation instead. * `images` and `labels`: Specified using the same format as the input data. Use this imagery as testing data if `from_training` is false. + * `log_folder` and `resume_cutoff`: If log_folder is specified, store read records of how much of each image + has been trained on in this folder. If the number of reads exceeds resume_cutoff, skip the tile when resuming + training. This allows resuming training skipping part of an epoch. You should generally not bother using this except + on very large training sets (thousands of large images). ### Network -These options configure the neural network to train with the `delta train` command. +For the `layers` attribute, any [Keras Layer](https://keras.io/api/layers/) can +be used, including custom layers registered with `delta.config.extensions.register_layer`. - * `classes`: The number of classes in the input data. The classes must currently have values - 0 - n in the label images. - * `model`: The network structure specification. - folder. You can either point to another `yaml_file`, such as the ones in the delta/config/networks - directory, or specify one under the `model` field in the same format as these files. The network - layers are specified using the [Keras functional layers API](https://keras.io/layers/core/) - converted to YAML files. +Sub-fields of the layer are argument names and values which are passed to the layer's constructor. +A special sub-field, `inputs`, is a list of the names of layers to pass as inputs to this layer. +If `inputs` is not specified, the previous layer is used by default. Layer names can be specified `name`. + +```yaml +layers: + Input: + shape: [~, ~, num_bands] + name: input + Add: + inputs: [input, input] +``` + +This simple example takes an input and adds it to itself. + +Since this network takes inputs of variable size ((~, ~, `num_bands`) is the input shape) it is a **fully +convolutional network**. This means that during training and classification, it will be evaluated on entire +tiles rather than smaller chunks. + +A few special parameters are available by default: + + * `num_bands`: The number of bands / channels in an image. + * `num_classes`: The number of classes provided in dataset.classes. MLFlow ------ @@ -119,10 +201,18 @@ General ------- * `gpus`: The number of GPUs to use, or `-1` for all. + * `verbose`: Trigger verbose printing. + * `extensions`: List of extensions to load. Add custom modules here and they will be loaded when + delta starts. + +I/O +------- * `threads`: The number of threads to use for loading images into tensorflow. - * `tile_size`: The size of a tile to load from an image at a time. For convolutional networks (input size is [~, ~, X], - an entire tile is one training sample. For fixed size networks the tile is split into chunks. This parameter affects - performance: larger tiles will be faster but take more memory (quadratic with chunk size for fixed size networks). - * `cache`: Configure cacheing options. The subfield `dir` specifies a directory on disk to store cached files, - and `limit` is the number of files to retain in the cache. Used mainly for image types - which much be extracted from archive files. + * `tile_size`: The size of a tile to load into memory at a time. For fully convolutional networks, the + entire tile will be processed at a time, for others it will be chunked. + * `interleave_images`: The number of images to interleave between. If this value is three, three images will + be opened at a time. Chunks / tiles will be interleaved from the first three tiles until one is completed, then + a new image will be opened. Larger interleaves can aid training (but comes at a cost in memory). + * `cache`: Options for a cache, which is used by a few image types (currently worldview and landsat). + * `dir`: Directory to store the cache. `default` gives a reasonable OS-specific default. + * `limit`: Maximum number of items to store in the cache before deleting old entries. diff --git a/delta/config/__init__.py b/delta/config/__init__.py index ee9bcbab..36ae09e3 100644 --- a/delta/config/__init__.py +++ b/delta/config/__init__.py @@ -18,6 +18,8 @@ """ Configuration via YAML files and command line options. +.. include:: README.md + Access the singleton `delta.config.config` to get configuration values, specified either in YAML files or on the command line, and to load additional YAML files. diff --git a/delta/config/config.py b/delta/config/config.py index acb9bfed..e3d977d6 100644 --- a/delta/config/config.py +++ b/delta/config/config.py @@ -14,14 +14,36 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +""" +Loading configuration from command line arguments and yaml files. + +Most users will want to use the global object `delta.config.config.config` +to access configuration parameters. +""" import os.path +from typing import Any, Callable, List, Optional, Tuple, Union import yaml import pkg_resources import appdirs -def validate_path(path, base_dir): +def validate_path(path: str, base_dir: str) -> str: + """ + Normalizes a path. + + Parameters + ---------- + path: str + Input path + base_dir: str + The base directory for relative paths. + + Returns + ------- + str + The normalized path. + """ if path == 'default': return path path = os.path.expanduser(path) @@ -30,12 +52,42 @@ def validate_path(path, base_dir): path = os.path.normpath(os.path.join(base_dir, path)) return path -def validate_positive(num, _): +def validate_positive(num: Union[int, float], _: str) -> Union[int, float]: + """ + Checks that a number is positive. + + Parameters + ---------- + num: Union[int, float] + Input number + _: str + Unused base path. + + Raises + ------ + ValueError + If number is not positive. + """ if num <= 0: raise ValueError('%d is not positive' % (num)) return num -def validate_non_negative(num, _): +def validate_non_negative(num: Union[int, float], _: str) -> Union[int, float]: + """ + Checks that a number is not negative. + + Parameters + ---------- + num: Union[int, float] + Input number + _: str + Unused base path. + + Raises + ------ + ValueError + If number is negative. + """ if num < 0: raise ValueError('%d is negative' % (num)) return num @@ -49,15 +101,15 @@ class DeltaConfigComponent: Handles one subsection of a config file. Generally subclasses will want to register fields and components in the constructor, - and possibly override setup_arg_parser and parse_args to handle + and possibly override `setup_arg_parser` and `parse_args` to handle command line options. - - section_header is the title of the section for command line - arguments in the help. """ - def __init__(self, section_header = None): + def __init__(self, section_header: Optional[str] = None): """ - Constructs the component. + Parameters + ---------- + section_header: Optional[str] + The title of the section for command line arguments in the help. """ self._config_dict = {} self._components = {} @@ -76,9 +128,18 @@ def reset(self): for c in self._components.values(): c.reset() - def register_component(self, component, name : str, attr_name = None): + def register_component(self, component: 'DeltaConfigComponent', name : str, attr_name: Optional[str] = None): """ - Register a subcomponent with a name and attribute name (access as self.attr_name) + Register a subcomponent. + + Parameters + ---------- + component: DeltaConfigComponent + The subcomponent to add. + name: str + Name of the subcomponent. Must be unique. + attr_name: Optional[str] + If specified, can access the component as self.attr_name. """ assert name not in self._components self._components[name] = component @@ -86,16 +147,25 @@ def register_component(self, component, name : str, attr_name = None): attr_name = name setattr(self, attr_name, component) - def register_field(self, name : str, types, accessor = None, validate_fn = None, desc = None): + def register_field(self, name: str, types: Union[type, Tuple[type, ...]], accessor: Optional[str] = None, + validate_fn: Optional[Callable[[Any, str], Any]] = None, desc = None): """ Register a field in this component of the configuration. - types is a single type or a tuple of valid types - - validate_fn (optional) should take two strings as input, the field's value and - the base directory, and return what to save to the config dictionary. - It should raise an exception if the field is invalid. - accessor is an optional name to create an accessor function with + Parameters + ---------- + name: str + Name of the field (must be unique). + types: type or tuple of types + Valid type or types for the field. + accessor: Optional[str] + If set, defines a function self.accessor() which retrieves the field. + validate_fn: Optional[Callable[[Any, str], Any]] + If specified, sets input = validate_fn(input, base_path) before using it, where + base_path is the current directory. The validate function should raise an error + if the input is invalid. + desc: Optional[str] + A description to use in help messages. """ self._fields.append(name) self._validate[name] = validate_fn @@ -108,16 +178,25 @@ def access(self) -> types: access.__doc__ = desc setattr(self.__class__, accessor, access) - def register_arg(self, field, argname, **kwargs): + def register_arg(self, field: str, argname: str, options_name: Optional[str] =None, **kwargs): """ - Registers a command line argument in this component. - - field is the (registered) field this argument modifies. - argname is the name of the flag on the command line (i.e., '--flag') - **kwargs are arguments to ArgumentParser.add_argument. - - If help and type are not specified, will use the ones for the field. - If default is not specified, will use the value from the config files. + Registers a command line argument in this component. Command line arguments override the + values in the config files when specified. + + Parameters + ---------- + field: str + The previously registered field this argument modifies. + argname: str + The name of the flag on the command line (i.e., '--flag') + options_name: Optional[str] + Name stored in the options object. It defaults to the + field if not specified. Only needed for duplicates, such as for multiple image + specifications. + **kwargs: + Further arguments are passed to ArgumentParser.add_argument. + If `help` and `type` are not specified, will use the values from field registration. + If `default` is not specified, will use the value from the config files. """ assert field in self._fields, 'Field %s not registered.' % (field) if 'help' not in kwargs: @@ -128,7 +207,7 @@ def register_arg(self, field, argname, **kwargs): del kwargs['type'] if 'default' not in kwargs: kwargs['default'] = _NotSpecified - self._cmd_args[argname] = (field, kwargs) + self._cmd_args[argname] = (field, field if options_name is None else options_name, kwargs) def to_dict(self) -> dict: """ @@ -171,33 +250,46 @@ def _load_dict(self, d : dict, base_dir): else: self._set_field(k, v, base_dir) - def setup_arg_parser(self, parser, components = None) -> None: + def setup_arg_parser(self, parser : 'argparse.ArgumentParser', components: Optional[List[str]] = None) -> None: """ - Adds arguments to the parser. Must overridden by child classes. + Adds arguments to the parser. May be overridden by child classes. + + Parameters + ---------- + parser: argparse.ArgumentParser + The praser to set up arguments with and later pass the command line flags to. + components: Optional[List[str]] + If specified, only parse arguments from the given components, specified by name. """ if self._section_header is not None: parser = parser.add_argument_group(self._section_header) for (arg, value) in self._cmd_args.items(): - (field, kwargs) = value - parser.add_argument(arg, dest=field, **kwargs) + (_, options_name, kwargs) = value + parser.add_argument(arg, dest=options_name, **kwargs) for (name, c) in self._components.items(): if components is None or name in components: c.setup_arg_parser(parser) - def parse_args(self, options): + def parse_args(self, options: 'argparse.Namespace'): """ - Parse options extracted from an ArgParser configured with + Parse options extracted from an `argparse.ArgumentParser` configured with `setup_arg_parser` and override the appropriate configuration values. + + Parameters + ---------- + options: argparse.Namespace + Options returned from a call to parse_args on a parser initialized with + setup_arg_parser. """ d = {} - for (field, _) in self._cmd_args.values(): - if not hasattr(options, field) or getattr(options, field) is None: + for (field, options_name, _) in self._cmd_args.values(): + if not hasattr(options, options_name) or getattr(options, options_name) is None: continue - if getattr(options, field) is _NotSpecified: + if getattr(options, options_name) is _NotSpecified: continue - d[field] = getattr(options, field) + d[field] = getattr(options, options_name) self._load_dict(d, None) for c in self._components.values(): @@ -205,14 +297,20 @@ def parse_args(self, options): class DeltaConfig(DeltaConfigComponent): """ - DELTA configuration manager. - - Access and control all configuration parameters. + DELTA configuration manager. Access and control all configuration parameters. """ - def load(self, yaml_file: str = None, yaml_str: str = None): + def load(self, yaml_file: Optional[str] = None, yaml_str: Optional[str] = None): """ Loads a config file, then updates the default configuration with the loaded values. + + Parameters + ---------- + yaml_file: Optional[str] + Filename of a yaml file to load. + yaml_str: Optional[str] + Load yaml directly from a str. Exactly one of `yaml_file` and `yaml_str` + must be specified. """ base_path = None if yaml_file: @@ -239,10 +337,16 @@ def reset(self): super().reset() self.load(pkg_resources.resource_filename('delta', 'config/delta.yaml')) - def initialize(self, options, config_files = None): + def initialize(self, options: 'argparse.Namespace', config_files: Optional[List[str]] = None): """ - Loads the default files unless config_files is specified, in which case it - loads them. Then loads options (from argparse). + Loads all config files, then parses all command line arguments. + Parameters + ---------- + options: argparse.Namespace + Command line options from `setup_arg_parser` to parse. + config_files: Optional[List[str]] + If specified, loads only the listed files. Otherwise, loads the default config + files. """ self.reset() @@ -259,3 +363,4 @@ def initialize(self, options, config_files = None): config.parse_args(options) config = DeltaConfig() +"""Global config object. Use this to access all configuration.""" diff --git a/delta/config/delta.yaml b/delta/config/delta.yaml index d2c46d80..ec55319c 100644 --- a/delta/config/delta.yaml +++ b/delta/config/delta.yaml @@ -12,16 +12,12 @@ io: tile_size: [256, 1024] # number of different images to interleave at a time when loading interleave_images: 5 - # When resuming training with a log_folder, skip input image where we have - # already loaded this many tiles. - resume_cutoff: 5000 cache: # default is OS-specific, in Linux, ~/.cache/delta dir: default limit: 8 dataset: - log_folder: ~ # Storage location for any record keeping files about the input dataset images: type: tiff # preprocess the images when loading (i.e., scaling) @@ -67,12 +63,10 @@ dataset: train: network: - model: - yaml_file: networks/convpool.yaml - params: ~ - layers: ~ + yaml_file: networks/convpool.yaml + params: ~ + layers: ~ stride: ~ - max_tile_offset: ~ # if set to integer, offset tiles by +1 each epoch modulus max_tile_offset batch_size: 500 steps: ~ # number of batches to train on (or ~ for all) epochs: 5 @@ -105,6 +99,10 @@ train: extension: default file_list: ~ files: ~ + log_folder: ~ # Storage location for any record keeping files about the input dataset + # When resuming training with a log_folder, skip input image where we have + # already loaded this many tiles. + resume_cutoff: 10 mlflow: # default to ~/.local/share/delta/mlflow diff --git a/delta/config/extensions.py b/delta/config/extensions.py index 09ecf39d..ed009530 100644 --- a/delta/config/extensions.py +++ b/delta/config/extensions.py @@ -18,9 +18,12 @@ Manage extensions to DELTA. To extend delta, add the name for your extension to the `extensions` field -in the DELTA config file. It will then be imported when DELTA loads. -The named python module should then call the appropriate register_* -functions and the extensions can be used like existing DELTA options. +in a DELTA config file. It will then be imported when DELTA loads. +The named python module should then call the appropriate registration +function (e.g., `register_layer` to register a custom Keras layer) and +the extensions can be used like existing DELTA options. + +All extensions can take keyword arguments that can be specified in the config file. """ #pylint:disable=global-statement @@ -50,6 +53,11 @@ def register_extension(name : str): """ Register an extension python module. For internal use --- users should use the config files. + + Parameters + ---------- + name: str + Name of the extension to load. """ global __extensions_to_load __extensions_to_load.add(name) @@ -57,51 +65,109 @@ def register_extension(name : str): def register_layer(layer_type : str, layer_constructor): """ Register a custom layer for use by DELTA. + + Parameters + ---------- + layer_type: str + Name of the layer. + layer_constructor + Either a class extending + [tensorflow.keras.layers.Layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LayerFunction), + or a function that returns a function that inputs and outputs tensors. + + See Also + -------- + delta.ml.train.DeltaLayer : Layer wrapper with Delta extensions """ global __layers __layers[layer_type] = layer_constructor -def register_image_reader(image_type : str, image_constructor): +def register_image_reader(image_type : str, image_class): """ Register a custom image type for reading by DELTA. + + Parameters + ---------- + image_type: str + Name of the image type. + image_class: Type[`delta.imagery.delta_image.DeltaImage`] + A class that extends `delta.imagery.delta_image.DeltaImage`. """ global __readers - __readers[image_type] = image_constructor + __readers[image_type] = image_class -def register_image_writer(image_type : str, image_constructor): +def register_image_writer(image_type : str, writer_class): """ Register a custom image type for writing by DELTA. + + Parameters + ---------- + image_type: str + Name of the image type. + writer_class: Type[`delta.imagery.delta_image.DeltaImageWriter`] + A class that extends `delta.imagery.delta_image.DeltaImageWriter`. """ global __writers - __writers[image_type] = image_constructor + __writers[image_type] = writer_class -def register_loss(loss_type : str, loss_constructor): +def register_loss(loss_type : str, custom_loss): """ - Register a custom loss function for use by DELTA. Loss - functions can also be used as metrics. + Register a custom loss function for use by DELTA. + + Note that loss functions can also be used as metrics. + + Parameters + ---------- + loss_type: str + Name of the loss function. + custom_loss: + Either a loss extending [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses/Loss) or a + function of the form loss(y_true, y_pred) which returns a tensor of the loss. """ global __losses - __losses[loss_type] = loss_constructor + __losses[loss_type] = custom_loss -def register_metric(metric_type : str, metric_constructor): +def register_metric(metric_type : str, custom_metric): """ Register a custom metric for use by DELTA. + + Parameters + ---------- + metric_type: str + Name of the metric. + custom_metric: Type[`tensorflow.keras.metrics.Metric`] + A class extending [Metric](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric). """ global __metrics - __metrics[metric_type] = metric_constructor + __metrics[metric_type] = custom_metric -def register_callback(cb_type : str, cb_constructor): +def register_callback(cb_type : str, cb): """ - Register a custom callback for use by DELTA. + Register a custom training callback for use by DELTA. + + Parameters + ---------- + cb_type: str + Name of the callback. + cb: Type[`tensorflow.keras.callbacks.Callback`] + A class extending [Callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback) + or a function that returns one. """ global __callbacks - __callbacks[cb_type] = cb_constructor + __callbacks[cb_type] = cb def register_preprocess(function_name : str, prep_function): """ Register a preprocessing function for use in delta. - preprocess_function(data, rectangle, bands_list) should return a numpy array. + Parameters + ---------- + function_name: str + Name of the preprocessing function. + prep_function: + A function of the form prep_function(data, rectangle, bands_list), where data is an input + numpy array, rectangle a `delta.imagery.rectangle.Rectangle` specifying the region covered by data, + and bands_list is an integer list of bands loaded. The function must return a numpy array. """ global __prep_funcs __prep_funcs[function_name] = prep_function @@ -109,6 +175,16 @@ def register_preprocess(function_name : str, prep_function): def layer(layer_type : str): """ Retrieve a custom layer by name. + + Parameters + ---------- + layer_type: str + Name of the layer. + + Returns + ------- + Layer + The previously registered layer. """ __initialize() return __layers.get(layer_type) @@ -116,6 +192,16 @@ def layer(layer_type : str): def loss(loss_type : str): """ Retrieve a custom loss by name. + + Parameters + ---------- + loss_type: str + Name of the loss function. + + Returns + ------- + Loss + The previously registered loss function. """ __initialize() return __losses.get(loss_type) @@ -123,6 +209,16 @@ def loss(loss_type : str): def metric(metric_type : str): """ Retrieve a custom metric by name. + + Parameters + ---------- + metric_type: str + Name of the metric. + + Returns + ------- + Metric + The previously registered metric. """ __initialize() return __metrics.get(metric_type) @@ -130,30 +226,50 @@ def metric(metric_type : str): def callback(cb_type : str): """ Retrieve a custom callback by name. + + Parameters + ---------- + cb_type: str + Name of the callback function. + + Returns + ------- + Callback + The previously registered callback. """ __initialize() return __callbacks.get(cb_type) def preprocess_function(prep_type : str): """ - Retrieve a custom callback by name. - """ - __initialize() - return __prep_funcs.get(prep_type) + Retrieve a custom preprocessing function by name. -def custom_objects(): - """ - Returns a dictionary of all supported custom objects for use - by tensorflow. + Parameters + ---------- + prep_type: str + Name of the preprocessing function. + + Returns + ------- + Preprocessing Function + The previously registered preprocessing function. """ __initialize() - d = __layers.copy() - d.update(__losses.copy()) - return d + return __prep_funcs.get(prep_type) def image_reader(reader_type : str): """ Get the reader of the given type. + + Parameters + ---------- + reader_type: str + Name of the image reader. + + Returns + ------- + Type[`delta.imagery.delta_image.DeltaImage`] + The previously registered image reader. """ __initialize() return __readers.get(reader_type) @@ -161,6 +277,31 @@ def image_reader(reader_type : str): def image_writer(writer_type : str): """ Get the writer of the given type. + + Parameters + ---------- + writer_type: str + Name of the image writer. + + Returns + ------- + Type[`delta.imagery.delta_image.DeltaImageWriter`] + The previously registered image writer. """ __initialize() return __writers.get(writer_type) + +def custom_objects(): + """ + Returns a dictionary of all supported custom objects for use + by tensorflow. Passed as an argument to load_model. + + Returns + ------- + dict + A dictionary of registered custom tensorflow objects. + """ + __initialize() + d = __layers.copy() + d.update(__losses.copy()) + return d diff --git a/delta/config/modules.py b/delta/config/modules.py index aa2526ca..d50316e7 100644 --- a/delta/config/modules.py +++ b/delta/config/modules.py @@ -25,6 +25,9 @@ from .extensions import register_extension class ExtensionsConfig(DeltaConfigComponent): + """ + Configuration component for extensions. + """ def __init__(self): super().__init__() @@ -42,6 +45,9 @@ def _load_dict(self, d : dict, base_dir): _config_initialized = False def register_all(): + """ + Register all default config modules. + """ global _config_initialized #pylint: disable=global-statement # needed to call twice when testing subcommands and when not if _config_initialized: diff --git a/delta/extensions/__init__.py b/delta/extensions/__init__.py index a4ad29cc..6b63af4d 100644 --- a/delta/extensions/__init__.py +++ b/delta/extensions/__init__.py @@ -17,6 +17,9 @@ """ Module for extensions to DELTA. + +This is a collection of default extensions that come with DELTA. If you +are interested in making your own extensions, see `delta.config.extensions`. """ from .defaults import initialize diff --git a/delta/extensions/callbacks.py b/delta/extensions/callbacks.py index 6c273a34..7bd8cc71 100644 --- a/delta/extensions/callbacks.py +++ b/delta/extensions/callbacks.py @@ -26,7 +26,24 @@ from delta.ml.train import ContinueTrainingException class SetTrainable(tensorflow.keras.callbacks.Callback): - def __init__(self, layer_name, epoch, trainable=True, learning_rate=None): + """ + Changes whether a given layer is trainable during training. + + This is useful for transfer learning, to do an initial training and then allow fine-tuning. + """ + def __init__(self, layer_name: str, epoch: int, trainable: bool=True, learning_rate: float=None): + """ + Parameters + ---------- + layer_name: str + The layer to modify. + epoch: int + The change will take place at the start of this epoch (the first epoch is 1). + trainable: bool + Whether the layer will be made trainable or not trainable. + learning_rate: float + Optionally change the learning rate as well. + """ super().__init__() self._layer_name = layer_name self._epoch = epoch - 1 @@ -44,7 +61,17 @@ def on_epoch_begin(self, epoch, logs=None): # have to abort, recompile changed model, and continue training raise ContinueTrainingException(completed_epochs=epoch, recompile_model=True, learning_rate=self._lr) -def ExponentialLRScheduler(start_epoch=10, multiplier=0.95): +def ExponentialLRScheduler(start_epoch: int=10, multiplier: float=0.95): + """ + Schedule the learning rate exponentially. + + Parameters + ---------- + start_epoch: int + The epoch to begin. + multiplier: float + After `start_epoch`, multiply the learning rate by this amount each epoch. + """ def schedule(epoch, lr): if epoch < start_epoch: return lr diff --git a/delta/extensions/defaults.py b/delta/extensions/defaults.py index ef25595a..d67a8691 100644 --- a/delta/extensions/defaults.py +++ b/delta/extensions/defaults.py @@ -15,7 +15,7 @@ # See the License for the specific language governing permissions and # limitations under the License. """ -Module to install extensions that come with DELTA. +Module to install all extensions that come with DELTA by default. """ from delta.config.extensions import register_extension, register_image_reader, register_image_writer @@ -27,6 +27,9 @@ from .sources import sentinel1 def initialize(): + """ + Register all default extensions. + """ register_extension('delta.extensions.callbacks') register_extension('delta.extensions.layers.pretrained') @@ -39,7 +42,6 @@ def initialize(): register_extension('delta.extensions.preprocess') register_image_reader('tiff', tiff.TiffImage) - register_image_reader('rgba', tiff.RGBAImage) register_image_reader('npy', npy.NumpyImage) register_image_reader('landsat', landsat.LandsatImage) register_image_reader('worldview', worldview.WorldviewImage) diff --git a/delta/extensions/layers/__init__.py b/delta/extensions/layers/__init__.py index 6448ef38..d51fe988 100644 --- a/delta/extensions/layers/__init__.py +++ b/delta/extensions/layers/__init__.py @@ -16,5 +16,5 @@ # limitations under the License. """ -Extra layers provided by DELTA. +Custom layers provided by DELTA. """ diff --git a/delta/extensions/layers/efficientnet.py b/delta/extensions/layers/efficientnet.py index 189dd77b..85182935 100644 --- a/delta/extensions/layers/efficientnet.py +++ b/delta/extensions/layers/efficientnet.py @@ -16,9 +16,12 @@ # limitations under the License. #pylint:disable=dangerous-default-value, too-many-arguments - -# taken from tensorflow and modified to remove initial layers +""" +An implementation of EfficientNet. This is +taken from tensorflow but modified to remove initial layers. +""" # https://github.com/keras-team/keras-applications/blob/master/keras_applications/efficientnet.py + from __future__ import absolute_import from __future__ import division from __future__ import print_function diff --git a/delta/extensions/layers/gaussian_sample.py b/delta/extensions/layers/gaussian_sample.py index 6bb13df9..5f7d99c6 100644 --- a/delta/extensions/layers/gaussian_sample.py +++ b/delta/extensions/layers/gaussian_sample.py @@ -16,7 +16,7 @@ # limitations under the License. """ -DELTA specific network layers. +Gaussian sampling layer, used in variational autoencoders. """ import tensorflow.keras.backend as K @@ -28,6 +28,16 @@ # If layers inherit from callback as well we add them automatically on fit class GaussianSample(DeltaLayer): def __init__(self, kl_loss=True, **kwargs): + """ + A layer that takes two inputs, a mean and a log variance, both of the same + dimensions. This layer returns a tensor of the same dimensions, sample + according to the provided mean and variance. + + Parameters + ---------- + kl_loss: bool + Add a kl loss term for the layer if true, to encourage a Normal(0, 1) distribution. + """ super().__init__(**kwargs) self._use_kl_loss = kl_loss self._kl_enabled = K.variable(0.0, name=self.name + ':kl_enabled') diff --git a/delta/extensions/layers/pretrained.py b/delta/extensions/layers/pretrained.py index 4b616ff6..49233cca 100644 --- a/delta/extensions/layers/pretrained.py +++ b/delta/extensions/layers/pretrained.py @@ -16,15 +16,25 @@ # limitations under the License. """ -DELTA specific network layers. +Use a pretrained model inside another network. """ +from typing import List, Optional import tensorflow import tensorflow.keras.models from delta.config.extensions import register_layer class InputSelectLayer(tensorflow.keras.layers.Layer): + """ + A layer that takes any number of inputs, and returns a given one. + """ def __init__(self, arg_number, **kwargs): + """ + Parameters + ---------- + arg_number: int + The index of the input to select. + """ super().__init__(**kwargs) self._arg = arg_number def call(self, inputs, **kwargs): @@ -45,13 +55,26 @@ def _model_to_output_layers(model, break_point, trainable): break return output_layers -def pretrained(filename, encoding_layer, outputs=None, trainable=True, training=True, **kwargs): +def pretrained(filename, encoding_layer, outputs: Optional[List[str]]=None, trainable: bool=True, + training: bool=True, **kwargs): """ Creates pre-trained layer from an existing model file. - Only works with sequential models. + Only works with sequential models. This was quite tricky to get right with tensorflow. - training is for batch norm layers. Since this model is not saved (it - only saves the contents) + Parameters + ---------- + filename: str + Model file to load. + encoding_layer: str + Name of the layer to stop at. + outputs: Optional[List[str]] + List of names of output layers that may be used later in the model. + Only layers listed here will be accessible as inputs to other layers, in the form + this_layer_name/internal_name. (internal_name must be included in outputs to do so) + trainable: bool + Whether to update weights during training for this layer. + training: bool + Standard tensorflow option, used for batch norm layers. """ model = tensorflow.keras.models.load_model(filename, compile=False) diff --git a/delta/extensions/layers/simple.py b/delta/extensions/layers/simple.py index 512aa53b..b3b9aa70 100644 --- a/delta/extensions/layers/simple.py +++ b/delta/extensions/layers/simple.py @@ -16,7 +16,7 @@ # limitations under the License. """ -DELTA specific network layers. +Simple helpful layers. """ import tensorflow as tf @@ -25,8 +25,13 @@ from delta.config.extensions import register_layer -# If layers inherit from callback as well we add them automatically on fit class RepeatedGlobalAveragePooling2D(tensorflow.keras.layers.Layer): + """ + Global average pooling in 2D for fully convolutional networks. + + Takes the global average over the entire input, and repeats + it to return a tensor the same size as the input. + """ def compute_output_shape(self, input_shape): return input_shape @@ -39,6 +44,9 @@ def call(self, inputs, **_): return mean * ones class ReflectionPadding2D(tensorflow.keras.layers.Layer): + """ + Add reflected padding of the given size surrounding the input. + """ def __init__(self, padding=(1, 1), **kwargs): super().__init__(**kwargs) self.padding = tuple(padding) diff --git a/delta/extensions/losses.py b/delta/extensions/losses.py index 3ab3312b..b176953e 100644 --- a/delta/extensions/losses.py +++ b/delta/extensions/losses.py @@ -29,9 +29,15 @@ from delta.config.extensions import register_loss def ms_ssim(y_true, y_pred): + """ + `tf.image.ssim_multiscale` as a loss function. + """ return 1.0 - tf.image.ssim_multiscale(y_true, y_pred, 4.0) def ms_ssim_mse(y_true, y_pred): + """ + Sum of MS-SSIM and Mean Squared Error. + """ return ms_ssim(y_true, y_pred) + K.mean(K.mean(tensorflow.keras.losses.MSE(y_true, y_pred), -1), -1) # from https://gist.github.com/wassname/7793e2058c5c9dacb5212c0ac0b18a8a @@ -45,13 +51,32 @@ def dice_coef(y_true, y_pred, smooth=1): return (2. * intersection + smooth) / (K.sum(K.square(y_true),-1) + K.sum(K.square(y_pred),-1) + smooth) def dice_loss(y_true, y_pred): + """ + Dice coefficient as a loss function. + """ return 1 - dice_coef(y_true, y_pred) class MappedLoss(tf.keras.losses.Loss): #pylint: disable=abstract-method def __init__(self, mapping, name=None): """ - Pass as argument either a list with probabilities for labels in order, - or a dictionary with classes mapped to their probabilities. + This is a base class for losses when the labels of the input images do not match the labels + output by the network. For example, if one class in the labels should be ignored, or two + classes in the label should map to the same output, or one label should be treated as a probability + between two classes. It applies a transform to the output labels and then applies the loss function. + + Note that the transform is applied after preprocessing (labels in the config will be transformed to 0-n + in order, and nodata will be n+1). + + Parameters + ---------- + mapping + One of: + * A list with transforms, where the first entry is what to transform the first label, to etc., i.e., + [1, 0] will swap the order of two labels. + * A dictionary with classes mapped to transformed values. Classes can be referenced by name or by + number (see `delta.imagery.imagery_config.ClassesConfig.class_id` for class formats). + name: Optional[str] + Optional name for the loss function. """ super().__init__(name=name) if isinstance(mapping, list): @@ -74,31 +99,42 @@ def __init__(self, mapping, name=None): self._lookup = tf.constant(map_list, dtype=tf.float32) class MappedCategoricalCrossentropy(MappedLoss): - # this is cross entropy, but first replaces the labels with - # a probability distribution from a lookup table + """ + `MappedLoss` for categorical_crossentropy. + """ def call(self, y_true, y_pred): y_true = tf.squeeze(y_true) true_convert = tf.gather(self._lookup, tf.cast(y_true, tf.int32), axis=None) return tensorflow.keras.losses.categorical_crossentropy(true_convert, y_pred) class MappedBinaryCrossentropy(MappedLoss): - # this is cross entropy, but first replaces the labels with - # a probability distribution from a lookup table + """ + `MappedLoss` for binary_crossentropy. + """ def call(self, y_true, y_pred): true_convert = tf.gather(self._lookup, tf.cast(y_true, tf.int32), axis=None) return tensorflow.keras.losses.binary_crossentropy(true_convert, y_pred) class MappedDiceLoss(MappedLoss): + """ + `MappedLoss` for `dice_loss`. + """ def call(self, y_true, y_pred): true_convert = tf.gather(self._lookup, tf.cast(y_true, tf.int32), axis=None) return dice_loss(true_convert, y_pred) class MappedMsssim(MappedLoss): + """ + `MappedLoss` for `ms_ssim`. + """ def call(self, y_true, y_pred): true_convert = tf.gather(self._lookup, tf.cast(y_true, tf.int32), axis=None) return ms_ssim(true_convert, y_pred) class MappedDiceBceMsssim(MappedLoss): + """ + `MappedLoss` for sum of `ms_ssim`, `dice_loss`, and `binary_crossentropy`. + """ def call(self, y_true, y_pred): true_convert = tf.gather(self._lookup, tf.cast(y_true, tf.int32), axis=None) dice = dice_loss(true_convert, y_pred) diff --git a/delta/extensions/metrics.py b/delta/extensions/metrics.py index 7bc1249f..30a237e5 100644 --- a/delta/extensions/metrics.py +++ b/delta/extensions/metrics.py @@ -14,7 +14,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +# pylint: disable=too-many-ancestors """ Various helpful loss functions. """ @@ -25,21 +25,44 @@ from delta.config import config from delta.config.extensions import register_metric -class SparseRecall(tensorflow.keras.metrics.Metric): # pragma: no cover - # this is cross entropy, but first replaces the labels with - # a probability distribution from a lookup table - def __init__(self, label, class_id=None, name=None, binary=False): +class SparseMetric(tensorflow.keras.metrics.Metric): # pylint:disable=abstract-method # pragma: no cover + """ + An abstract class for metrics applied to integer class labels, + with networks that output one-hot encoding. + """ + def __init__(self, label, class_id: int=None, name: str=None, binary: int=False): + """ + Parameters + ---------- + label + A class identifier accepted by `delta.imagery.imagery_config.ClassesConfig.class_id`. + Compared to valuse in the label image. + class_id: Optional[int] + For multi-class one-hot outputs, used if the output class ID is different than the + one in the label image. + name: str + Metric name. + binary: bool + Use binary threshold (0.5) or argmax on one-hot encoding. + """ super().__init__(name=name) self._binary = binary self._label_id = config.dataset.classes.class_id(label) self._class_id = class_id if class_id is not None else self._label_id - self._total_class = self.add_weight('total_class', initializer='zeros') - self._true_positives = self.add_weight('true_positives', initializer='zeros') def reset_state(self): for s in self.variables: s.assign(tf.zeros(shape=s.shape)) +class SparseRecall(SparseMetric): # pragma: no cover + """ + Recall. + """ + def __init__(self, label, class_id: int=None, name: str=None, binary: int=False): + super().__init__(label, class_id, name, binary) + self._total_class = self.add_weight('total_class', initializer='zeros') + self._true_positives = self.add_weight('true_positives', initializer='zeros') + def update_state(self, y_true, y_pred, sample_weight=None): #pylint: disable=unused-argument, arguments-differ y_true = tf.squeeze(y_true) right_class = tf.math.equal(y_true, self._label_id) @@ -58,21 +81,15 @@ def update_state(self, y_true, y_pred, sample_weight=None): #pylint: disable=unu def result(self): return tf.math.divide_no_nan(self._true_positives, self._total_class) -class SparsePrecision(tensorflow.keras.metrics.Metric): # pragma: no cover - # this is cross entropy, but first replaces the labels with - # a probability distribution from a lookup table - def __init__(self, label, class_id=None, name=None, binary=False): - super().__init__(name=name) - self._binary = binary - self._label_id = config.dataset.classes.class_id(label) - self._class_id = class_id if class_id is not None else self._label_id +class SparsePrecision(SparseMetric): # pragma: no cover + """ + Precision. + """ + def __init__(self, label, class_id: int=None, name: str=None, binary: int=False): + super().__init__(label, class_id, name, binary) self._total_class = self.add_weight('total_class', initializer='zeros') self._true_positives = self.add_weight('true_positives', initializer='zeros') - def reset_state(self): - for s in self.variables: - s.assign(tf.zeros(shape=s.shape)) - def update_state(self, y_true, y_pred, sample_weight=None): #pylint: disable=unused-argument, arguments-differ y_true = tf.squeeze(y_true) right_class = tf.math.equal(y_true, self._label_id) @@ -92,19 +109,16 @@ def update_state(self, y_true, y_pred, sample_weight=None): #pylint: disable=unu def result(self): return tf.math.divide_no_nan(self._true_positives, self._total_class) -class SparseBinaryAccuracy(tensorflow.keras.metrics.Metric): # pragma: no cover - # Binary accuracy but where labels are transformed - def __init__(self, label, name=None): - super().__init__(name=name) - self._label_id = config.dataset.classes.class_id(label) +class SparseBinaryAccuracy(SparseMetric): # pragma: no cover + """ + Accuracy. + """ + def __init__(self, label, name: str=None): + super().__init__(label, label, name, False) self._nodata_id = config.dataset.classes.class_id('nodata') self._total = self.add_weight('total', initializer='zeros') self._correct = self.add_weight('correct', initializer='zeros') - def reset_state(self): - for s in self.variables: - s.assign(tf.zeros(shape=s.shape)) - def update_state(self, y_true, y_pred, sample_weight=None): #pylint: disable=unused-argument, arguments-differ y_true = tf.squeeze(y_true) y_pred = tf.squeeze(y_pred) diff --git a/delta/extensions/preprocess.py b/delta/extensions/preprocess.py index 4c9d7fa1..1ea57635 100644 --- a/delta/extensions/preprocess.py +++ b/delta/extensions/preprocess.py @@ -17,6 +17,11 @@ #pylint:disable=unused-argument """ Various helpful preprocessing functions. + +These are intended to be included in image: preprocess in a yaml file. +See the `delta.config` documentation for details. Note that for all +functions, the image_type will be specified automatically: other +parameters must be specified in the config file. """ import numpy as np @@ -29,16 +34,41 @@ 'sentinel1' : None} def scale(image_type, factor='default'): + """ + Divides by a given scale factor. + + Parameters + ---------- + factor: Union[str, float] + Scale factor to divide by. 'default' will scale by an image type specific + default amount. + """ if factor == 'default': factor = __DEFAULT_SCALE_FACTORS[image_type] factor = np.float32(factor) return (lambda data, _, dummy: data / factor) def offset(image_type, factor): + """ + Add an amount to all pixels. + + Parameters + ---------- + factor: float + Number to add. + """ factor = np.float32(factor) return lambda data, _, dummy: data + factor def clip(image_type, bounds): + """ + Clips all pixel values within a range. + + Parameters + ---------- + bounds: List[float] + List of two floats to clip all values between. + """ if isinstance(bounds, list): assert len(bounds) == 2, 'Bounds must have two items.' else: @@ -47,14 +77,38 @@ def clip(image_type, bounds): return lambda data, _, dummy: np.clip(data, bounds[0], bounds[1]) def cbrt(image_type): + """ + Cubic root. + """ return lambda data, _, dummy: np.cbrt(data) def sqrt(image_type): + """ + Square root. + """ return lambda data, _, dummy: np.sqrt(data) def gauss_mult_noise(image_type, stddev): + """ + Multiplies each pixel by p ~ Normal(1, stddev) + + Parameters + ---------- + stddev: float + Standard deviation of distribution to sample from. + """ return lambda data, _, dummy: data * np.random.normal(1.0, stddev, data.shape) def substitute(image_type, mapping): + """ + Replaces pixels in image with the listed values. + + Parameters + ---------- + mapping: List[Any] + For example, to change a binary image to a one-hot representation, + use [[1, 0], [0, 1]]. This replaces all 0 pixels with [1, 0] and all + 1 pixels with [0, 1]. + """ return lambda data, _, dummy: np.take(mapping, data) register_preprocess('scale', scale) diff --git a/delta/extensions/sources/__init__.py b/delta/extensions/sources/__init__.py index 57524841..9b3fff39 100644 --- a/delta/extensions/sources/__init__.py +++ b/delta/extensions/sources/__init__.py @@ -17,4 +17,9 @@ """ Imagery types for DELTA. + +These are specified in the "type" field in the configuration yaml file. + +Note that while DELTA supports compressed images for some satellites, we +recommend extracting these images to tiffs beforehand as it will speed up training. """ diff --git a/delta/extensions/sources/landsat.py b/delta/extensions/sources/landsat.py index 87f2772e..f9550360 100644 --- a/delta/extensions/sources/landsat.py +++ b/delta/extensions/sources/landsat.py @@ -146,7 +146,7 @@ def _find_mtl_file(folder): class LandsatImage(tiff.TiffImage): - """Compressed Landsat image tensorflow dataset wrapper (see imagery_dataset.py)""" + """Compressed Landsat image. Loads a compressed zip or tar file with a .mtl file.""" def __init__(self, paths, nodata_value=None, bands=None): self._bands = bands diff --git a/delta/extensions/sources/npy.py b/delta/extensions/sources/npy.py index 5894e2de..bb3a7b49 100644 --- a/delta/extensions/sources/npy.py +++ b/delta/extensions/sources/npy.py @@ -20,16 +20,27 @@ """ import os +from typing import Optional + import numpy as np from delta.imagery import delta_image class NumpyImage(delta_image.DeltaImage): """ - Numpy image data tensorflow dataset wrapper (see imagery_dataset.py). - Can set either path to load a file, or data to load a numpy array directly. + Load a numpy array as an image. """ - def __init__(self, data=None, path=None, nodata_value=None): + def __init__(self, data: Optional[np.ndarray]=None, path: Optional[str]=None, nodata_value=None): + """ + Parameters + ---------- + data: Optional[numpy.ndarray] + Loads a numpy array directly. + path: Optional[str] + Load a numpy array from a file with `numpy.load`. Only one of data or path should be given. + nodata_value + The pixel value representing no data. + """ super().__init__(nodata_value) if path: @@ -43,11 +54,6 @@ def __init__(self, data=None, path=None, nodata_value=None): self._data = data def _read(self, roi, bands, buf=None): - """ - Read the image of the given data type. An optional roi specifies the boundaries. - - This function is intended to be overwritten by subclasses. - """ if buf is None: buf = np.zeros(shape=(roi.width(), roi.height(), self.num_bands() ), dtype=self._data.dtype) (min_x, max_x, min_y, max_y) = roi.bounds() @@ -55,13 +61,14 @@ def _read(self, roi, bands, buf=None): return buf def size(self): - """Return the size of this image in pixels, as (width, height).""" return (self._data.shape[1], self._data.shape[0]) def num_bands(self): - """Return the number of bands in the image.""" return self._data.shape[2] + def dtype(self): + return self._data.dtype + class NumpyWriter(delta_image.DeltaImageWriter): def __init__(self): self._buffer = None diff --git a/delta/extensions/sources/tiff.py b/delta/extensions/sources/tiff.py index f8513634..da36b5df 100644 --- a/delta/extensions/sources/tiff.py +++ b/delta/extensions/sources/tiff.py @@ -24,7 +24,6 @@ import numpy as np from osgeo import gdal -from delta.config import config from delta.imagery import delta_image, rectangle @@ -42,13 +41,20 @@ _NUMPY_TO_GDAL_TYPES = {v: k for k, v in _GDAL_TO_NUMPY_TYPES.items()} class TiffImage(delta_image.DeltaImage): - """For geotiffs.""" + """Images supported by GDAL.""" def __init__(self, path, nodata_value=None): - ''' - Opens a geotiff for reading. paths can be either a single filename or a list. - For a list, the images are opened in order as a multi-band image, assumed to overlap. - ''' + """ + Opens a geotiff for reading. + + Parameters + ---------- + paths: str or List[str] + Either a single filename or a list. + For a list, the images are opened in order as a multi-band image, assumed to overlap. + nodata_value: dtype of image + Value representing no data. + """ super().__init__(nodata_value) paths = self._prep(path) @@ -76,8 +82,17 @@ def _prep(self, paths): #pylint:disable=no-self-use """ Prepare the file to be opened by other tools (unpack, etc). - Returns a list of underlying files to load instead of the original path. - This is intended to be overwritten by subclasses. + This can be overwritten by subclasses to, for example, + unpack a zip file to a cache directory. + + Parameters + ---------- + paths: str or List[str] + Paths passed to constructor + + Returns + ------- + Returns a list of underlying files to load instead of the original paths. """ if isinstance(paths, str): return [paths] @@ -88,11 +103,17 @@ def __asert_open(self): raise IOError('Operating on an image that has been closed.') def close(self): + """ + Close the image. + """ self._handles = None # gdal doesn't have a close function for some reason self._band_map = None self._paths = None def path(self): + """ + Returns the paths returned by `_prep`. + """ return self._path def num_bands(self): @@ -108,7 +129,7 @@ def _read(self, roi, bands, buf=None): num_bands = len(bands) if bands else self.num_bands() if buf is None: - buf = np.zeros(shape=(num_bands, roi.width(), roi.height()), dtype=self.numpy_type()) + buf = np.zeros(shape=(num_bands, roi.width(), roi.height()), dtype=self.dtype()) for i, b in enumerate(bands): band_handle = self._gdal_band(b) s = buf[i, :, :].shape @@ -124,58 +145,43 @@ def _gdal_band(self, band): assert ret return ret - # using custom nodata from config TODO: use both - #def nodata_value(self, band=0): - # ''' - # Returns the value that indicates no data is present in a pixel for the specified band. - # ''' - # self.__asert_open() - # return self._gdal_band(band).GetNoDataValue() - - def data_type(self, band=0): - ''' + def _gdal_type(self, band=0): + """ Returns the GDAL data type of the image. - ''' + """ self.__asert_open() return self._gdal_band(band).DataType - def numpy_type(self, band=0): + def dtype(self): self.__asert_open() - dtype = self.data_type(band) + dtype = self._gdal_type(0) if dtype in _GDAL_TO_NUMPY_TYPES: return _GDAL_TO_NUMPY_TYPES[dtype] raise Exception('Unrecognized gdal data type: ' + str(dtype)) def bytes_per_pixel(self, band=0): - ''' - Returns the number of bytes per pixel - ''' + """ + Returns + ------- + int: + the number of bytes per pixel + """ self.__asert_open() - return gdal.GetDataTypeSize(self.data_type(band)) // 8 + return gdal.GetDataTypeSize(self._gdal_type(band)) // 8 def block_size(self): - """Returns (block height, block width)""" - (bs, _) = self.block_info() - return bs - - def block_info(self, band=0): - """Returns ((block height, block width), (num blocks x, num blocks y))""" + """ + Returns + ------- + (int, int): + block height, block width + """ self.__asert_open() - band_handle = self._gdal_band(band) + band_handle = self._gdal_band(0) block_size = band_handle.GetBlockSize() - - num_blocks_x = int(math.ceil(self.width() / block_size[0])) - num_blocks_y = int(math.ceil(self.height() / block_size[1])) - - # we are backwards from gdal I think - return ((block_size[1], block_size[0]), (num_blocks_x, num_blocks_y)) + return (block_size[1], block_size[0]) def metadata(self): - ''' - Returns all useful image metadata. - - If multiple images were specified, returns the information from the first. - ''' self.__asert_open() data = dict() h = self._handles[0] @@ -187,17 +193,13 @@ def metadata(self): return data def block_aligned_roi(self, desired_roi): - ''' - Returns the block aligned pixel region to read in a Rectangle format - to get the requested data region while respecting block boundaries. - ''' self.__asert_open() bounds = rectangle.Rectangle(0, 0, width=self.width(), height=self.height()) if not bounds.contains_rect(desired_roi): raise Exception('desired_roi ' + str(desired_roi) + ' is outside the bounds of image with size' + str(self.size())) - (block_size, unused_num_blocks) = self.block_info(0) + block_size = self.block_size() start_block_x = int(math.floor(desired_roi.min_x / block_size[0])) start_block_y = int(math.floor(desired_roi.min_y / block_size[1])) # Rect max is exclusive @@ -216,24 +218,33 @@ def block_aligned_roi(self, desired_roi): bounds = rectangle.Rectangle(0, 0, width=self.width(), height=self.height()) return ans.get_intersection(bounds) - def save(self, path, tile_size=(0,0), nodata_value=None, show_progress=False): + def save(self, path, tile_size=None, nodata_value=None, show_progress=False): """ - Save a TiffImage to the file output_path, optionally overwriting the tile_size. - Input tile size is (width, height) + Save to file, with preprocessing applied. + + Parameters + ---------- + path: str + Filename to save to. + tile_size: (int, int) + If specified, overwrite block size + nodata_value: image dtype + If specified, overwrite nodata value + show_progress: bool + Write progress bar to stdout """ if nodata_value is None: nodata_value = self.nodata_value() # Use the input tile size for the block size unless the user specified one. block_size_y, block_size_x = self.block_size() - if tile_size[0] > 0: + if tile_size is not None: block_size_x = tile_size[0] - if tile_size[1] > 0: block_size_y = tile_size[1] # Set up the output image with _TiffWriter(path, self.width(), self.height(), self.num_bands(), - self.data_type(), block_size_x, block_size_y, + self._gdal_type(), block_size_x, block_size_y, nodata_value, self.metadata()) as writer: input_bounds = rectangle.Rectangle(0, 0, width=self.width(), height=self.height()) output_rois = input_bounds.make_tile_rois((block_size_x, block_size_y), include_partials=True) @@ -251,35 +262,30 @@ def callback_function(output_roi, data): self.process_rois(output_rois, callback_function, show_progress=show_progress) -class RGBAImage(TiffImage): - """Basic RGBA images where the alpha channel needs to be stripped""" - - def _prep(self, paths): - """Converts RGBA images to RGB images""" - - # Get the path to the cached image - fname = os.path.basename(paths) - output_path = config.io.cache.manager().register_item(fname) - - if not os.path.exists(output_path): - # Just remove the alpha band from the original image - cmd = 'gdal_translate -b 1 -b 2 -b 3 ' + paths + ' ' + output_path - os.system(cmd) - return [output_path] - -def numpy_dtype_to_gdal_type(dtype): #pylint: disable=R0911 +def _numpy_dtype_to_gdal_type(dtype): #pylint: disable=R0911 if dtype in _NUMPY_TO_GDAL_TYPES: return _NUMPY_TO_GDAL_TYPES[dtype] raise Exception('Unrecognized numpy data type: ' + str(dtype)) -def write_tiff(output_path, data, metadata=None): - """Try to write a tiff file""" +def write_tiff(output_path: str, data: np.ndarray, metadata: dict=None): + """ + Write a numpy array to a file as a tiff. + + Parameters + ---------- + output_path: str + Filename to save tiff file to + data: numpy.ndarray + Image data to save. + metadata: dict + Optional metadata to include. + """ if len(data.shape) < 3: num_bands = 1 else: num_bands = data.shape[2] - data_type = numpy_dtype_to_gdal_type(data.dtype) + data_type = _numpy_dtype_to_gdal_type(data.dtype) TILE_SIZE=256 @@ -399,31 +405,25 @@ def write_region(self, data, x, y): gdal_band.WriteArray(data[:, :, band], y, x) class TiffWriter(delta_image.DeltaImageWriter): + """ + Write a geotiff to a file. + """ def __init__(self, filename): self._filename = filename self._tiff_w = None def initialize(self, size, numpy_dtype, metadata=None, nodata_value=None): - """ - Prepare for writing with the given size and dtype. - """ assert (len(size) == 3), ('Error: len(size) of '+str(size)+' != 3') TILE_SIZE = 256 self._tiff_w = _TiffWriter(self._filename, size[0], size[1], num_bands=size[2], - data_type=numpy_dtype_to_gdal_type(numpy_dtype), metadata=metadata, + data_type=_numpy_dtype_to_gdal_type(numpy_dtype), metadata=metadata, nodata_value=nodata_value, tile_width=min(TILE_SIZE, size[0]), tile_height=min(TILE_SIZE, size[1])) def write(self, data, x, y): - """ - Writes the data as a rectangular block starting at the given coordinates. - """ self._tiff_w.write_region(data, x, y) def close(self): - """ - Finish writing. - """ if self._tiff_w is not None: self._tiff_w.close() diff --git a/delta/extensions/sources/worldview.py b/delta/extensions/sources/worldview.py index 4879a5ff..db827669 100644 --- a/delta/extensions/sources/worldview.py +++ b/delta/extensions/sources/worldview.py @@ -83,7 +83,7 @@ def unpack_wv_to_folder(zip_path, unpack_folder): class WorldviewImage(tiff.TiffImage): - """Compressed WorldView image tensorflow dataset wrapper (see imagery_dataset.py)""" + """Compressed WorldView image. Loads an image from a zip file with a tiff and a .imd file.""" def __init__(self, paths, nodata_value=None): self._meta_path = None self._meta = None diff --git a/delta/imagery/delta_image.py b/delta/imagery/delta_image.py index 67eddc8c..f206da98 100644 --- a/delta/imagery/delta_image.py +++ b/delta/imagery/delta_image.py @@ -16,7 +16,7 @@ # limitations under the License. """ -Base class for loading images. +Base classes for reading and writing images. """ from abc import ABC, abstractmethod @@ -31,10 +31,17 @@ class DeltaImage(ABC): """ - Base class used for wrapping input images in a way that they can be passed - to Tensorflow dataset objects. + Base class used for wrapping input images in DELTA. Can be extended + to support new data types. A variety of image types are implemented in + `delta.extensions.sources`. """ def __init__(self, nodata_value=None): + """ + Parameters + ---------- + nodata_value: Optional[Any] + Nodata value for the image, if any. + """ self.__preprocess_function = None self.__nodata_value = nodata_value @@ -42,11 +49,22 @@ def read(self, roi: rectangle.Rectangle=None, bands: List[int]=None, buf: np.nda """ Reads the image in [row, col, band] indexing. - If `roi` is not specified, reads the entire image. - If `buf` is specified, writes the image to buf. - If `bands` is not specified, reads all bands, otherwise - only the listed bands are read. - If bands is a single integer, drops the band dimension. + Subclasses should generally not overwrite this method--- they will likely want to implement + `_read`. + + Parameters + ---------- + roi: `rectangle.Rectangle` + The bounding box to read from the image. If None, read the entire image. + bands: List[int] + Bands to load (zero-indexed). If None, read all bands. + buf: np.ndarray + If specified, reads the image into this buffer. Must be sufficiently large. + + Returns + ------- + np.ndarray: + A buffer containing the requested part of the image. """ if roi is None: roi = rectangle.Rectangle(0, 0, width=self.width(), height=self.height()) @@ -65,76 +83,185 @@ def read(self, roi: rectangle.Rectangle=None, bands: List[int]=None, buf: np.nda return self.__preprocess_function(result, roi, bands) return result - def set_preprocess(self, callback: Callable[[np.ndarray, rectangle.Rectangle, List[int]], np.ndarray]) -> None: + def set_preprocess(self, callback: Callable[[np.ndarray, rectangle.Rectangle, List[int]], np.ndarray]): """ - Set a preproprocessing function callback to be applied to the results of all reads on the image. - - The function takes the arguments callback(image, roi, bands), where image is the numpy array containing - the read data, roi is the region of interest read, and bands is a list of the bands being read. + Set a preproprocessing function callback to be applied to the results of + all reads on the image. + + Parameters + ---------- + callback: Callable[[np.ndarray, rectangle.Rectangle, List[in]], np.ndarray] + A function to be called on loading image data, callback(image, roi, bands), + where `image` is the numpy array containing the read data, `roi` is the region of interest read, + and `bands` is a list of the bands read. Must return a numpy array. """ self.__preprocess_function = callback - def get_preprocess(self): + def get_preprocess(self) -> Callable[[np.ndarray, rectangle.Rectangle, List[int]], np.ndarray]: """ - Returns the preprocess function. + Returns + ------- + Callable[[np.ndarray, rectangle.Rectangle, List[int]], np.ndarray] + The preprocess function currently set. """ return self.__preprocess_function def nodata_value(self): """ - Returns the value of pixels to treat as nodata. + Returns + ------- + The value of pixels to treat as nodata. """ return self.__nodata_value @abstractmethod - def _read(self, roi, bands, buf=None): + def _read(self, roi: rectangle.Rectangle, bands: List[int], buf: np.ndarray=None) -> np.ndarray: """ - Read the image of the given data type. An optional roi specifies the boundaries. - - This function is intended to be overwritten by subclasses. + Read the image. + + Abstract function to be implemented by subclasses. Users should call `read` instead. + + Parameters + ---------- + roi: rectangle.Rectangle + Segment of the image to read. + bands: List[int] + List of bands to read (zero-indexed). + buf: np.ndarray + Buffer to read into. If not specified, a new buffer should be allocated. + + Returns + ------- + np.ndarray: + The relevant part of the image as a numpy array. """ def metadata(self): #pylint:disable=no-self-use """ - Returns a dictionary of metadata, in the format used by GDAL. + Returns + ------- + A dictionary of metadata, if any is given for the image type. """ return {} @abstractmethod def size(self) -> Tuple[int, int]: - """Return the size of this image in pixels, as (width, height).""" + """ + Returns + ------- + Tuple[int, int]: + The size of this image in pixels, as (width, height). + """ @abstractmethod def num_bands(self) -> int: - """Return the number of bands in the image.""" + """ + Returns + ------- + int: + The number of bands in this image. + """ + + @abstractmethod + def dtype(self) -> np.dtype: + """ + Returns + ------- + numpy.dtype: + The underlying data type of the image. + """ def block_aligned_roi(self, desired_roi: rectangle.Rectangle) -> rectangle.Rectangle:#pylint:disable=no-self-use - """Return the block-aligned roi containing this image region, if applicable.""" + """ + Parameters + ---------- + desired_roi: rectangle.Rectangle + Original region of interest. + + Returns + ------- + rectangle.Rectangle: + The block-aligned roi containing the specified roi. + """ return desired_roi def block_size(self): #pylint: disable=no-self-use - """Return the preferred block size for efficient reading.""" + """ + Returns + ------- + (int, int): + The suggested block size for efficient reading. + """ return (256, 256) def width(self) -> int: - """Return the number of columns.""" + """ + Returns + ------- + int: + The number of image columns + """ return self.size()[0] def height(self) -> int: - """Return the number of rows.""" + """ + Returns + ------- + int: + The number of image rows + """ return self.size()[1] def tiles(self, shape, overlap_shape=(0, 0), partials: bool=True, min_shape=(0, 0), - partials_overlap: bool=False, by_block=False) -> Iterator[rectangle.Rectangle]: - """Generator to yield ROIs for the image.""" + partials_overlap: bool=False, by_block=False): + """ + Splits the image into tiles with the given properties. + + Parameters + ---------- + shape: (int, int) + Shape of each tile + overlap_shape: (int, int) + Amount to overlap tiles in x and y direction + partials: bool + If true, include partial tiles at the edge of the image. + min_shape: (int, int) + If true and `partials` is true, keep partial tiles of this minimum size. + partials_overlap: bool + If `partials` is false, and this is true, expand partial tiles + to the desired size. Tiles may overlap in some areas. + by_block: bool + If true, changes the returned generator to group tiles by block. + This is intended to optimize disk reads by reading the entire block at once. + + Returns + ------- + List[Rectangle] or List[(Rectangle, List[Rectangle])] + List of ROIs. If `by_block` is true, returns a list of (Rectangle, List[Rectangle]) + instead, where the first rectangle is a larger block containing multiple tiles in a list. + """ input_bounds = rectangle.Rectangle(0, 0, max_x=self.width(), max_y=self.height()) return input_bounds.make_tile_rois(shape, overlap_shape=overlap_shape, include_partials=partials, min_shape=min_shape, partials_overlap=partials_overlap, by_block=by_block) - def roi_generator(self, requested_rois: Iterator[rectangle.Rectangle]) -> Iterator[rectangle.Rectangle]: + def roi_generator(self, requested_rois: Iterator[rectangle.Rectangle]) -> \ + Iterator[Tuple[rectangle.Rectangle, np.ndarray, int, int]]: """ - Generator that yields ROIs of blocks in the requested region. + Generator that yields image blocks of the requested rois. + + Parameters + ---------- + requested_rois: Iterator[Rectangle] + Regions of interest to read. + + Returns + ------- + Iterator[Tuple[Rectangle, numpy.ndarray, int, int]] + A generator with read image regions. In each tuple, the first item + is the region of interest, the second is a numpy array of the image contents, + the third is the index of the current region of interest, and the fourth is the total + number of rois. """ block_rois = copy.copy(requested_rois) @@ -189,10 +316,17 @@ def process_rois(self, requested_rois: Iterator[rectangle.Rectangle], callback_function: Callable[[rectangle.Rectangle, np.ndarray], None], show_progress: bool=False) -> None: """ - Process the given region broken up into blocks using the callback function. - Each block will get the image data from each input image passed into the function. - Data reading takes place in a separate thread, but the callbacks are executed - in a consistent order on a single thread. + Apply a callback function to a list of ROIs. + + Parameters + ---------- + requested_rois: Iterator[Rectangle] + Regions of interest to evaluate + callback_function: Callable[[rectangle.Rectangle, np.ndarray], None] + A function to apply to each requested region. Pass the bounding box + of the current region and a numpy array of pixel values as inputs. + show_progress: bool + Print a progress bar on the command line if true. """ for (roi, buf, (i, total)) in self.roi_generator(requested_rois): callback_function(roi, buf) @@ -202,28 +336,50 @@ def process_rois(self, requested_rois: Iterator[rectangle.Rectangle], print() class DeltaImageWriter(ABC): + """ + Base class for writing images in DELTA. + """ @abstractmethod def initialize(self, size, numpy_dtype, metadata=None, nodata_value=None): """ - Prepare for writing with the given size and dtype. + Prepare for writing. + + Parameters + ---------- + size: tuple of ints + Dimensions of the image to write. + numpy_dtype: numpy.dtype + Type of the underling data. + metadata: dict + Dictionary of metadata to save with the image. + nodata_value: numpy_dtype + Value representing nodata in the image. """ @abstractmethod - def write(self, data, x, y): + def write(self, data: np.ndarray, x: int, y: int): """ - Writes the data as a rectangular block starting at the given coordinates. + Write a portion of the image. + + Parameters + ---------- + data: np.ndarray + A block of image data to write. + x: int + y: int + Top-left coordinates of the block of data to write. """ @abstractmethod def close(self): """ - Finish writing. + Finish writing, perform cleanup. """ @abstractmethod def abort(self): """ - Cancel writing before finished. + Cancel writing before finished, perform cleanup. """ def __del__(self): diff --git a/delta/imagery/disk_folder_cache.py b/delta/imagery/disk_folder_cache.py index 957e07da..f15fb343 100644 --- a/delta/imagery/disk_folder_cache.py +++ b/delta/imagery/disk_folder_cache.py @@ -26,10 +26,14 @@ class DiskCache: It is safe to mix different datasets in the cache folder, though all items in the folder will count towards the limit. """ - def __init__(self, top_folder, limit): + def __init__(self, top_folder: str, limit: int): """ - The top level folder to store cached items in and the number to store - are specified. + Parameters + ---------- + top_folder: str + Top level cache directory. + limit: int + Maximum number of items to keep in cache. """ if limit < 1: raise Exception('Illegal limit passed to Disk Cache: ' + str(limit)) @@ -48,25 +52,44 @@ def __init__(self, top_folder, limit): def limit(self): """ - The number of items to store in the cache. + Returns + ------- + int: + The maximum number of items to cache. """ return self._limit def folder(self): """ - The directory to store cached items in. + Returns + ------- + str: + The cache directory. """ return self._folder def num_cached(self): """ - The number of items currently cached. + Returns + ------- + int: + The number of items currently cached. """ return len(self._item_list) def register_item(self, name): """ - Register a new item with the cache manager and return the full path to it. + Register a new item with the cache manager. + + Parameters + ---------- + name: str + Filename of the item to add to the cache. + + Returns + ------- + str: + Full path to store the item in the cache. """ # If we already have the name just move it to the back of the list diff --git a/delta/imagery/imagery_config.py b/delta/imagery/imagery_config.py index 0e22b41a..d511153c 100644 --- a/delta/imagery/imagery_config.py +++ b/delta/imagery/imagery_config.py @@ -37,13 +37,18 @@ class ImageSet: """ def __init__(self, images, image_type, preprocess=None, nodata_value=None): """ - The parameters for the constructor are: - - * An iterable of image filenames `images` - * The image type (i.e., tiff, worldview, landsat) `image_type` - * An optional preprocessing function to apply to the image, - following the signature in `delta.imagery.sources.delta_image.DeltaImage.set_process`. - * A `nodata_value` for pixels to disregard + Parameters + ---------- + images: Iterator[str] + Image filenames + image_type: str + The image type as a string (i.e., tiff, worldview, landsat). Must have + been previously registered with `delta.config.extensions.register_image_reader`. + preprocess: Callable + Optional preprocessing function to apply to the image + following the signature in `delta.imagery.delta_image.DeltaImage.set_preprocess`. + nodata_value: image dtype + A no data value for pixels to disregard """ self._images = images self._image_type = image_type @@ -52,27 +57,54 @@ def __init__(self, images, image_type, preprocess=None, nodata_value=None): def type(self): """ - The type of the image, a string. + Returns + ------- + str: + The type of the image """ return self._image_type def preprocess(self): """ - Return the preprocessing function. + Returns + ------- + Callable: + The preprocessing function """ return self._preprocess def nodata_value(self): """ - Value of pixels to disregard. + Returns + ------- + image dtype: + Value of pixels to disregard. """ return self._nodata_value def set_nodata_value(self, nodata): """ Set the pixel value to disregard. + + Parameters + ---------- + nodata: image dtype + The pixel value to set as nodata """ self._nodata_value = nodata def load(self, index): + """ + Loads the image of the given index. + + Parameters + ---------- + index: int + Index of the image to load. + + Returns + ------- + `delta.imagery.delta_image.DeltaImage`: + The image + """ img = image_reader(self.type())(self[index], self.nodata_value()) if self._preprocess: img.set_preprocess(self._preprocess) @@ -132,7 +164,12 @@ def __find_images(conf, matching_images=None, matching_conf=None): for m in matching_images: rel_path = os.path.relpath(m, matching_conf['directory']) label_path = os.path.join(conf['directory'], rel_path) - images.append(os.path.splitext(label_path)[0] + extension) + if matching_conf['directory'] is None: + images.append(os.path.splitext(label_path)[0] + extension) + else: + # if custom extension, remove it + label_path = label_path[:-len(__extension(matching_conf))] + images.append(label_path + extension) for img in images: if not os.path.exists(img): @@ -187,6 +224,12 @@ def class_shift(data, _, dummy): class_shift, len(classes_comp) if labels_nodata is not None else None)) class ImagePreprocessConfig(DeltaConfigComponent): + """ + Configuration for image preprocessing. + + Expects a list of preprocessing functions registered + with `delta.config.extensions.register_preprocess`. + """ def __init__(self): super().__init__() self._functions = [] @@ -209,6 +252,16 @@ def _load_dict(self, d, base_dir): self._functions.append((name, func[name])) def function(self, image_type): + """ + Parameters + ---------- + image_type: str + Type of the image + Returns + ------- + Callable: + The specified preprocessing function to apply to the image. + """ prep = lambda data, _, dummy: data for (name, args) in self._functions: t = preprocess_function(name) @@ -226,6 +279,11 @@ def _validate_paths(paths, base_dir): return out class ImageSetConfig(DeltaConfigComponent): + """ + Configuration for a set of images. + + Used for images, labels, and validation images and labels. + """ def __init__(self, name=None): super().__init__() self.register_field('type', str, 'type', None, 'Image type.') @@ -236,14 +294,20 @@ def __init__(self, name=None): self.register_field('nodata_value', (float, int), None, None, 'Value of pixels to ignore.') if name: - self.register_arg('type', '--' + name + '-type') - self.register_arg('file_list', '--' + name + '-file-list') - self.register_arg('directory', '--' + name + '-dir') - self.register_arg('extension', '--' + name + '-extension') + self.register_arg('type', '--' + name + '-type', name + '_type') + self.register_arg('file_list', '--' + name + '-file-list', name + '_file_list') + self.register_arg('directory', '--' + name + '-dir', name + '_directory') + self.register_arg('extension', '--' + name + '-extension', name + '_extension') self.register_component(ImagePreprocessConfig(), 'preprocess') self._name = name def preprocess_function(self): + """ + Returns + ------- + Callable: + Preprocessing function for the set of images. + """ return self._components['preprocess'].function(self._config_dict['type']) def setup_arg_parser(self, parser, components = None) -> None: @@ -263,7 +327,22 @@ def parse_args(self, options): self._config_dict['file_list'] = None class LabelClass: + """ + Label configuration. + """ def __init__(self, value, name=None, color=None, weight=None): + """ + Parameters + ---------- + value: int + Pixel of the label + name: str + Name of the class to display + color: int + In visualizations, set the class to this RGB color. + weight: float + During training weight this class by this amount. + """ color_order = [0x1f77b4, 0xff7f0e, 0x2ca02c, 0xd62728, 0x9467bd, 0x8c564b, \ 0xe377c2, 0x7f7f7f, 0xbcbd22, 0x17becf] if name is None: @@ -280,6 +359,11 @@ def __repr__(self): return 'Color: ' + self.name class ClassesConfig(DeltaConfigComponent): + """ + Configuration for classes. + + Specify either a number of classes or list of classes with details. + """ def __init__(self): super().__init__() self._classes = [] @@ -332,8 +416,17 @@ def _load_dict(self, d : dict, base_dir): def class_id(self, class_name): """ - class_name can either be an int (original pixel value in images) or the name of a class. - Returns the ID of the class in the labels after transformation in image preprocessing. + Parameters + ---------- + class_name: int or str + Either the original pixel value in images (int) or the name (str) of a class. + The special value 'nodata' will give the nodata class, if any. + + Returns + ------- + int: + the ID of the class in the labels after default image preprocessing (labels are arranged + to a canonical order, with nodata always coming after them.) """ if class_name == len(self._classes) or class_name == 'nodata': return len(self._classes) @@ -343,6 +436,12 @@ def class_id(self, class_name): raise ValueError('Class ' + class_name + ' not found.') def weights(self): + """ + Returns + ------- + List[float] + List of class weights for use in training, if specified. + """ weights = [] for c in self._classes: if c.weight is not None: @@ -353,6 +452,12 @@ def weights(self): return weights def classes_to_indices_func(self): + """ + Returns + ------- + Callable[[numpy.ndarray], numpy.ndarray]: + Function to convert label image to canonical form + """ if not self._conversions: return None def convert(data): @@ -363,6 +468,12 @@ def convert(data): return convert def indices_to_classes_func(self): + """ + Returns + ------- + Callable[[numpy.ndarray], numpy.ndarray]: + Reverse of `classes_to_indices_func`. + """ if not self._conversions: return None def convert(data): @@ -373,14 +484,15 @@ def convert(data): return convert class DatasetConfig(DeltaConfigComponent): + """ + Configuration for a dataset. + """ def __init__(self): super().__init__('Dataset') self.register_component(ImageSetConfig('image'), 'images', '__image_comp') self.register_component(ImageSetConfig('label'), 'labels', '__label_comp') self.__images = None self.__labels = None - self.register_field('log_folder', str, 'log_folder', validate_path, - 'Directory where dataset progress is recorded.') self.register_component(ClassesConfig(), 'classes') def reset(self): @@ -390,7 +502,10 @@ def reset(self): def images(self) -> ImageSet: """ - Returns the training images. + Returns + ------- + ImageSet: + the training images """ if self.__images is None: (self.__images, self.__labels) = load_images_labels(self._components['images'], @@ -400,7 +515,10 @@ def images(self) -> ImageSet: def labels(self) -> ImageSet: """ - Returns the label images. + Returns + ------- + ImageSet: + the label images """ if self.__labels is None: (self.__images, self.__labels) = load_images_labels(self._components['images'], @@ -409,6 +527,9 @@ def labels(self) -> ImageSet: return self.__labels class CacheConfig(DeltaConfigComponent): + """ + Configuration for cache. + """ def __init__(self): super().__init__() self.register_field('dir', str, None, validate_path, 'Cache directory.') @@ -422,7 +543,10 @@ def reset(self): def manager(self) -> disk_folder_cache.DiskCache: """ - Returns the disk cache object to manage the cache. + Returns + ------- + `disk_folder_cache.DiskCache`: + the object to manage the cache """ if self._cache_manager is None: # Auto-populating defaults here is a workaround so small tools can skip the full @@ -444,6 +568,9 @@ def _validate_tile_size(size, _): return size class IOConfig(DeltaConfigComponent): + """ + Configuration for I/O. + """ def __init__(self): super().__init__('IO') self.register_field('threads', int, None, None, 'Number of threads to use.') @@ -451,13 +578,18 @@ def __init__(self): 'Size of an image tile to load in memory at once.') self.register_field('interleave_images', int, 'interleave_images', validate_positive, 'Number of images to interleave at a time when training.') - self.register_field('resume_cutoff', int, 'resume_cutoff', None, - 'When resuming a dataset, skip images where we have read this many tiles.') self.register_arg('threads', '--threads') self.register_component(CacheConfig(), 'cache') + def threads(self): + """ + Returns + ------- + int: + number of threads to use for I/O + """ if 'threads' in self._config_dict and self._config_dict['threads']: return self._config_dict['threads'] return min(1, os.cpu_count() // 2) diff --git a/delta/imagery/imagery_dataset.py b/delta/imagery/imagery_dataset.py index 12dd5579..103ab31d 100644 --- a/delta/imagery/imagery_dataset.py +++ b/delta/imagery/imagery_dataset.py @@ -30,13 +30,29 @@ from delta.config import config class ImageryDataset: # pylint: disable=too-many-instance-attributes - """Create dataset with all files as described in the provided config file. + """ + A dataset for tiling very large imagery for training with tensorflow. """ def __init__(self, images, labels, output_shape, chunk_shape, stride=None, tile_shape=(256, 256), tile_overlap=None): """ - Initialize the dataset based on the specified image and label ImageSets + Parameters + ---------- + images: ImageSet + Images to train on + labels: ImageSet + Corresponding labels to train on + output_shape: (int, int) + Shape of the corresponding labels for a given chunk or tile size. + chunk_shape: (int, int) + If specified, divide tiles into individual chunks of this shape. + stride: (int, int) + Skip this stride between chunks. Only valid with chunk_shape. + tile_shape: (int, int) + Size of tiles to load from the images at a time. + tile_overlap: (int, int) + If specified, overlap tiles by this amount. """ self._resume_mode = False @@ -56,7 +72,6 @@ def __init__(self, images, labels, output_shape, chunk_shape, stride=None, if tile_overlap is None: tile_overlap = (0, 0) self._tile_overlap = tile_overlap - self._tile_offset = None if labels: assert len(images) == len(labels) @@ -72,7 +87,15 @@ def __init__(self, images, labels, output_shape, chunk_shape, stride=None, # I think we should probably get rid of it at some point. def set_resume_mode(self, resume_mode, log_folder): """ - Enable / disable resume mode and set a folder to store read log files. + Enable / disable resume mode and configure it. + + Parameters + ---------- + resume_mode: bool + If true, log and check access counts for if imagery can be skipped + this epoch. + log_folder: str + Folder to log access counts to """ self._resume_mode = resume_mode self._log_folder = log_folder @@ -80,7 +103,16 @@ def set_resume_mode(self, resume_mode, log_folder): os.mkdir(self._log_folder) def _resume_log_path(self, image_id): - """Return the path to the read log for an input image""" + """ + Parameters + ---------- + image_id: int + + Returns + ------- + str: + the path to the read log for an input image + """ if not self._log_folder: return None image_path = self._images[image_id] @@ -90,8 +122,20 @@ def _resume_log_path(self, image_id): return log_path def resume_log_read(self, image_id): #pylint: disable=R0201 - """Reads an access count file containing a boolean and a count. - The boolean is set to true if we need to check the count.""" + """ + Reads an access count file containing a boolean and a count. + + Parameters + ---------- + image_id: int + Image id to check logs for + + Returns + ------- + (bool, int): + need_to_check, access count + The boolean is set to true if we need to check the count. + """ path = self._resume_log_path(image_id) try: with portalocker.Lock(path, 'r', timeout=300) as f: @@ -110,6 +154,18 @@ def resume_log_read(self, image_id): #pylint: disable=R0201 return (False, 0) def resume_log_update(self, image_id, count=None, need_check=False): #pylint: disable=R0201 + """ + Update logs of when images are read. Should only be needed internally. + + Parameters + ---------- + image_id: int + The image to update + count: int + Number of tiles that have been read + need_check: bool + Set flag for if a check is needed + """ log_path = self._resume_log_path(image_id) if not log_path: return @@ -120,11 +176,15 @@ def resume_log_update(self, image_id, count=None, need_check=False): #pylint: d f.write('%d %d' % (int(need_check), count)) def reset_access_counts(self, set_need_check=False): - """Go through all the access files and reset one of the values. - This is needed for resume mode to work. - Call with default value to reset the counts to zero. Call with - "set_need_check" to keep the count and mark that it needs to be checked. - Call with default after each epoch, call (True) at start of training.""" + """ + Go through all the access files and reset the counts. Should be done at the end of each epoch. + + Parameters + ---------- + set_need_check: bool + if true, keep the count and mark that it needs to be checked. (should be + set at the start of training) + """ if not self._log_folder: return if config.general.verbose(): @@ -133,6 +193,17 @@ def reset_access_counts(self, set_need_check=False): self.resume_log_update(i, count=0, need_check=set_need_check) def _list_tiles(self, i): # pragma: no cover + """ + Parameters + ---------- + i: int + Image to list tiles for. + + Returns + ------- + List[Rectangle]: + List of tiles to read from the given image + """ # If we need to skip this file because of the read count, no need to look up tiles. if self._resume_mode: file_path = self._images[i] @@ -141,7 +212,7 @@ def _list_tiles(self, i): # pragma: no cover if config.general.verbose(): print('get_image_tile_list for index ' + str(i) + ' -> ' + file_path) (need_to_check, count) = self.resume_log_read(i) - if need_to_check and (count > config.io.resume_cutoff()): + if need_to_check and (count > config.train.resume_cutoff()): if config.general.verbose(): print('Skipping index ' + str(i) + ' tile gen with count ' + str(count) + ' -> ' + file_path) @@ -173,6 +244,18 @@ def _list_tiles(self, i): # pragma: no cover def _tile_generator(self, i, is_labels): # pragma: no cover """ A generator that yields image tiles from the given image. + + Parameters + ---------- + i: int + Image id + is_labels: bool + Load the label if true, image if false + + Returns + ------- + Iterator[numpy.ndarray]: + Iterator over iamge tiles. """ i = int(i) tiles = self._list_tiles(i) @@ -190,27 +273,6 @@ def _tile_generator(self, i, is_labels): # pragma: no cover image.set_preprocess(None) # parallelize the preprocessing, not in disk i/o threadpool bands = range(image.num_bands()) - # apply tile offset. do here so we always have same number of tiles (causes problems with tf) - if self._tile_offset: - def shift_tile(t): - t.shift(self._tile_offset[0], self._tile_offset[1]) - t.max_x = min(t.max_x, image.width()) - t.max_y = min(t.max_y, image.height()) - if t.width() < self._tile_shape[0]: - t.min_x = t.max_x - self._tile_shape[0] - if t.height() < self._tile_shape[1]: - t.min_y = t.max_y - self._tile_shape[1] - for (rect, subtiles) in tiles: - shift_tile(rect) - for t in subtiles: - # just use last tile that fits - if t.max_x > rect.width(): - t.max_x = rect.width() - t.min_x = rect.width() - self._tile_shape[0] - if t.max_y > rect.height(): - t.max_y = rect.height() - t.min_y = rect.height() - self._tile_shape[1] - # read one row ahead of what we process now next_buf = self._iopool.submit(lambda: image.read(tiles[0][0])) for (c, (rect, sub_tiles)) in enumerate(tiles): @@ -236,7 +298,18 @@ def shift_tile(t): def _load_images(self, is_labels, data_type): """ Loads a list of images as tensors. - If label_list is specified, load labels instead. The corresponding image files are still required however. + + Parameters + ---------- + is_labels: bool + Load labels if true, images if not + data_type: numpy.dtype + Data type that will be returned. + + Returns + ------- + Dataset: + Dataset of image tiles """ r = tf.data.Dataset.range(len(self._images)) r = r.shuffle(1000, seed=0, reshuffle_each_iteration=True) # shuffle same way for labels and non-labels @@ -285,7 +358,10 @@ def _reshape_labels(self, labels): # pragma: no cover def data(self): """ - Unbatched dataset of image chunks. + Returns + ------- + Dataset: + image chunks / tiles. """ ret = self._load_images(False, self._data_type) if self._chunk_shape: @@ -295,7 +371,10 @@ def data(self): def labels(self): """ - Unbatched dataset of labels. + Returns + ------- + Dataset: + Unbatched dataset of labels corresponding to `data()`. """ label_set = self._load_images(True, self._label_type) if self._chunk_shape or self._output_shape: @@ -306,16 +385,23 @@ def labels(self): def dataset(self, class_weights=None): """ - Return the underlying TensorFlow dataset object that this class creates. + Returns a tensorflow dataset as configured by the class. + + Parameters + ---------- + class_weights: list + list of weights for the classes. - class_weights: list of weights in the classes. - If class_weights is specified, returns a dataset of (data, labels, weights) instead. + Returns + ------- + tensorflow Dataset: + With (data, labels, optionally weights) """ # Pair the data and labels in our dataset ds = tf.data.Dataset.zip((self.data(), self.labels())) # ignore chunks which are all nodata (nodata is re-indexed to be after the classes) - if self._labels.nodata_value() is not None and self._tile_offset is None: + if self._labels.nodata_value() is not None: ds = ds.filter(lambda x, y: tf.math.reduce_any(tf.math.not_equal(y, self._labels.nodata_value()))) if class_weights is not None: class_weights.append(0.0) @@ -325,10 +411,25 @@ def dataset(self, class_weights=None): return ds def num_bands(self): - """Return the number of bands in each image of the data set""" + """ + Returns + ------- + int: + number of bands in each image + """ return self._num_bands def set_chunk_output_shapes(self, chunk_shape, output_shape): + """ + Parameters + ---------- + chunk_shape: (int, int) + Size of chunks to read at a time. Set to None to + use on a per tile basis (i.e., for FCNs). + output_shape: (int, int) + Shape output by the network. May differ from the input size + (dervied from chunk_shape or tile_shape) + """ if chunk_shape: assert len(chunk_shape) == 2, 'Chunk must be two dimensional.' assert (chunk_shape[0] % 2) == (chunk_shape[1] % 2) == \ @@ -342,59 +443,97 @@ def set_chunk_output_shapes(self, chunk_shape, output_shape): def chunk_shape(self): """ - Size of chunks used for inputs. + Returns + ------- + (int, int): + Size of chunks used for inputs. """ return self._chunk_shape def input_shape(self): - """Input size for the network.""" + """ + Returns + ------- + Tuple[int, ...]: + Input size for the network. + """ if self._chunk_shape: return (self._chunk_shape[0], self._chunk_shape[1], self._num_bands) return (None, None, self._num_bands) def output_shape(self): - """Output size of blocks of labels""" + """ + Returns + ------- + Tuple[int, ...]: + Output size, size of blocks of labels + """ if self._output_shape: return (self._output_shape[0], self._output_shape[1], self._output_dims) return (None, None, self._output_dims) def image_set(self): - """Returns set of images""" + """ + Returns + ------- + ImageSet: + set of images + """ return self._images def label_set(self): - """Returns set of label images""" + """ + Returns + ------- + ImageSet: + set of labels + """ return self._labels def set_tile_shape(self, tile_shape): - """Set the tile size.""" + """ + Set the tile size. + + Parameters + ---------- + tile_shape: (int, int) + New tile shape""" self._tile_shape = tile_shape def tile_shape(self): - """Returns tile size.""" + """ + Returns + ------- + Tuple[int, ...]: + tile shape to load at a time + """ return self._tile_shape def tile_overlap(self): - """Returns the amount tiles overlap, for FCNS.""" + """ + Returns + ------- + Tuple[int, ...]: + the amount tiles overlap + """ return self._tile_overlap - def tile_offset(self): - """Offset for start of tiles when tiling (for FCNs).""" - return self._tile_offset - - def set_tile_offset(self, offset): - """Set offset for start of tiles when tiling (for FCNs).""" - self._tile_offset = offset - def stride(self): + """ + Returns + ------- + Tuple[int, ...]: + Stride between chunks (only when chunk_shape is set). + """ return self._stride class AutoencoderDataset(ImageryDataset): - """Slightly modified dataset class for the Autoencoder which does not use separate label files""" + """ + Slightly modified dataset class for the autoencoder. + + Instead of specifying labels, the inputs are used as labels. + """ def __init__(self, images, chunk_shape, stride=(1, 1), tile_shape=(256, 256), tile_overlap=None): - """ - The images are used as labels as well. - """ super().__init__(images, None, chunk_shape, chunk_shape, tile_shape=tile_shape, stride=stride, tile_overlap=tile_overlap) self._labels = self._images diff --git a/delta/imagery/rectangle.py b/delta/imagery/rectangle.py index e151b22e..a17aa52a 100644 --- a/delta/imagery/rectangle.py +++ b/delta/imagery/rectangle.py @@ -21,12 +21,24 @@ import math class Rectangle: - """Simple rectangle class for ROIs. Max values are NON-INCLUSIVE. - When using it, stay consistent with float or integer values. + """ + Simple rectangle class for ROIs. Max values are NON-INCLUSIVE. + When using it, stay consistent with float or integer values. """ def __init__(self, min_x, min_y, max_x=0, max_y=0, width=0, height=0): - """Specify width/height by name to use those instead of max_x/max_y.""" + """ + Parameters + ---------- + min_x: int + min_y: int + max_x: int + max_y: int + Rectangle bounds. + width: int + height: int + Specify width / height to use these instead of max_x/max_y. + """ self.min_x = min_x self.min_y = min_y if width > 0: @@ -56,7 +68,12 @@ def __repr__(self): # yield(TileIndex(row,col)) def bounds(self): - '''Returns (min_x, max_x, min_y, max_y)''' + """ + Returns + ------- + (int, int, int, int): + (min_x, max_x, min_y, max_y) + """ return (self.min_x, self.max_x, self.min_y, self.max_y) def width(self): @@ -65,14 +82,18 @@ def height(self): return self.max_y - self.min_y def has_area(self): - '''Returns true if the rectangle contains any area.''' + """ + Returns + ------- + bool: + true if the rectangle contains any area. + """ return (self.width() > 0) and (self.height() > 0) def perimeter(self): return 2*self.width() + 2*self.height() def area(self): - '''Returns the valid area''' if not self.has_area(): return 0 return self.height() * self.width() @@ -159,17 +180,32 @@ def overlaps(self, other_rect): def make_tile_rois(self, tile_shape, overlap_shape=(0, 0), include_partials=True, min_shape=(0, 0), partials_overlap=False, by_block=False): - ''' + """ Return a list of tiles encompassing the entire area of this Rectangle. - tile_shape: (width, height) of tiles - overlap_shape: (x, y) overlap tiles by this many pixels in respective dimension - include_partials: include tiles that don't tile evenly - min_shape: minimum size of partial tiles to include with include_partials - partials_overlap: if not include_partials, if there are border regions not part of a tile, - make a full size tile including them, and nearby area may be in two tiles - by_block: Returns a list of (block_rect, [r1, r2, ..., rn]) tuples instead, where block_rect - is the bounding box of a row and rn are the subrectangles in that row - ''' + + Parameters + ---------- + tile_shape: (int, int) + Shape of each tile + overlap_shape: (int, int) + Amount to overlap tiles in x and y direction + include_partials: bool + If true, include partial tiles at the edge of the image. + min_shape: (int, int) + If true and `partials` is true, keep partial tiles of this minimum size. + partials_overlap: bool + If `partials` is false, and this is true, expand partial tiles + to the desired size. Tiles may overlap in some areas. + by_block: bool + If true, changes the returned generator to group tiles by block. + This is intended to optimize disk reads by reading the entire block at once. + + Returns + ------- + List[Rectangle]: + Generator yielding ROIs. If `by_block` is true, returns a generator of (Rectangle, List[Rectangle]) + instead, where the first rectangle is a larger block containing multiple tiles in a list. + """ tile_width, tile_height = tile_shape min_width, min_height = min_shape diff --git a/delta/imagery/utilities.py b/delta/imagery/utilities.py index 79a46475..9ad5d6dc 100644 --- a/delta/imagery/utilities.py +++ b/delta/imagery/utilities.py @@ -27,6 +27,13 @@ def unpack_to_folder(compressed_path, unpack_folder): """ Unpack a file into the given folder. + + Parameters + ---------- + compressed_path: str + Zip or tar file path + unpack_folder: str + Folder to unpack to """ tmpdir = os.path.normpath(unpack_folder) + '_working' @@ -49,6 +56,17 @@ def progress_bar(text, fill_amount, prefix = '', length = 80): #pylint: disable= """ Prints a progress bar. Call multiple times with increasing progress to overwrite the printed line. + + Parameters + ---------- + text: str + Text to print after progress bar + fill_amount: float + Percent to fill bar, from 0.0 - 1.0 + prefix: str + Text to print before progress bar + length: int + Number of characters to fill as bar """ filled_length = int(length * fill_amount) fill_char = '█' if sys.stdout.encoding.lower() == 'utf-8' else 'X' diff --git a/delta/ml/config_parser.py b/delta/ml/config_parser.py index d0277127..0c0b0611 100644 --- a/delta/ml/config_parser.py +++ b/delta/ml/config_parser.py @@ -22,7 +22,7 @@ from collections.abc import Mapping import copy import functools -from typing import Callable, List +from typing import Callable, List, Union import tensorflow import tensorflow.keras.layers @@ -192,9 +192,21 @@ def apply_params(s): return model_dict_copy -def model_from_dict(model_dict, exposed_params) -> Callable[[], tensorflow.keras.models.Sequential]: +def model_from_dict(model_dict: dict, exposed_params: dict) -> Callable[[], tensorflow.keras.models.Model]: """ - Creates a function that returns a sequential model from a dictionary. + Construct a model. + + Parameters + ---------- + model_dict: dict + Config dictionary describing the model + exposed_params: dict + Dictionary of parameter names and values to substitute. + + Returns + ------- + Callable[[], tensorflow.keras.models.Model]: + Model constructor function. """ model_dict = _apply_params(model_dict, exposed_params) return functools.partial(_make_model, model_dict['layers']) @@ -208,14 +220,22 @@ def _parse_str_or_dict(spec, type_name): return (name, spec[name]) raise ValueError('Unexpected entry for %s.' % (type_name)) -def loss_from_dict(loss_spec): - ''' - Creates a loss function object from a dictionary. +def loss_from_dict(loss_spec: Union[dict, str]) -> tensorflow.keras.losses.Loss: + """ + Construct a loss function. + + Parameters + ---------- + loss_spec: Union[dict, str] + Specification of the loss function. Either a string that is compatible + with the keras interface (e.g. 'categorical_crossentropy') or an object defined by a dict + of the form {'LossFunctionName': {'arg1':arg1_val, ...,'argN',argN_val}} - :param: loss_spec Specification of the loss function. Either a string that is compatible - with the keras interface (e.g. 'categorical_crossentropy') or an object defined by a dict - of the form {'LossFunctionName': {'arg1':arg1_val, ...,'argN',argN_val}} - ''' + Returns + ------- + tensorflow.keras.losses.Loss + The loss object. + """ (name, params) = _parse_str_or_dict(loss_spec, 'loss function') lc = extensions.loss(name) if lc is None: @@ -226,9 +246,19 @@ def loss_from_dict(loss_spec): lc = lc(**params) return lc -def metric_from_dict(metric_spec): +def metric_from_dict(metric_spec: Union[dict, str]) -> tensorflow.keras.metrics.Metric: """ - Creates a metric object from a dictionary or string. + Construct a metric. + + Parameters + ---------- + metric_spec: Union[dict, str] + Config dictionary or string defining the metric + + Returns + ------- + tensorflow.keras.metrics.Metric + The metric object. """ (name, params) = _parse_str_or_dict(metric_spec, 'metric') mc = extensions.metric(name) @@ -243,9 +273,19 @@ def metric_from_dict(metric_spec): mc = mc(**params) return mc -def optimizer_from_dict(spec): +def optimizer_from_dict(spec: Union[dict, str]) -> tensorflow.keras.optimizers.Optimizer: """ - Creates an optimizer from a dictionary or string. + Construct an optimizer from a dictionary or string. + + Parameters + ---------- + spec: Union[dict, str] + Config dictionary or string defining an optimizer + + Returns + ------- + tensorflow.keras.optimizers.Optimizer + The optimizer object. """ (name, params) = _parse_str_or_dict(spec, 'optimizer') mc = getattr(tensorflow.keras.optimizers, name, None) @@ -253,10 +293,20 @@ def optimizer_from_dict(spec): raise ValueError('Unknown optimizer %s.' % (name)) return mc(**params) -def callback_from_dict(callback_dict) -> tensorflow.keras.callbacks.Callback: - ''' - Constructs a callback object from a dictionary. - ''' +def callback_from_dict(callback_dict: Union[dict, str]) -> tensorflow.keras.callbacks.Callback: + """ + Construct a callback from a dictionary. + + Parameters + ---------- + callback_dict: Union[dict, str] + Config dictionary defining a callback. + + Returns + ------- + tensorflow.keras.callbacks.Callback + The callback object. + """ assert len(callback_dict.keys()) == 1, f'Error: Callback has more than one type {callback_dict.keys()}' cb_type = next(iter(callback_dict.keys())) @@ -270,18 +320,24 @@ def callback_from_dict(callback_dict) -> tensorflow.keras.callbacks.Callback: return callback_class(**callback_dict[cb_type]) def config_callbacks() -> List[tensorflow.keras.callbacks.Callback]: - ''' - Iterates over the list of callbacks specified in the config file, which is part of the training specification. - ''' + """ + Returns + ------- + List[tensorflow.keras.callbacks.Callback] + List of callbacks specified in the config file. + """ if not config.train.callbacks() is None: return [callback_from_dict(callback) for callback in config.train.callbacks()] return [] -def config_model(num_bands: int) -> Callable[[], tensorflow.keras.models.Sequential]: +def config_model(num_bands: int) -> Callable[[], tensorflow.keras.models.Model]: """ - Creates the model specified in the configuration. + Returns + ------- + Callable[[], tensorflow.keras.models.Model] + A function to construct the model given in the config file. """ params_exposed = {'num_classes' : len(config.dataset.classes), 'num_bands' : num_bands} - return model_from_dict(config.train.network.model.to_dict(), params_exposed) + return model_from_dict(config.train.network.to_dict(), params_exposed) diff --git a/delta/ml/io.py b/delta/ml/io.py index a88d8bbe..14d7fbfb 100644 --- a/delta/ml/io.py +++ b/delta/ml/io.py @@ -28,12 +28,25 @@ def save_model(model, filename): """ Save a model. Includes DELTA configuration. + + Parameters + ---------- + model: tensorflow.keras.models.Model + The model to save. + filename: str + Output filename. """ model.save(filename, save_format='h5') with h5py.File(filename, 'r+') as f: f.attrs['delta'] = config.export() def print_layer(l): + """ + Print a layer to stdout. + + l: tensorflow.keras.layers.Layer + The layer to print. + """ s = "{:<25}".format(l.name) + ' ' + '{:<20}'.format(str(l.input_shape)) + \ ' -> ' + '{:<20}'.format(str(l.output_shape)) c = l.get_config() @@ -44,6 +57,14 @@ def print_layer(l): print(s) def print_network(a, tile_shape=None): + """ + Print a model to stdout. + + a: tensorflow.keras.models.Model + The model to print. + tile_shape: Optional[Tuple[int, int]] + If specified, print layer output sizes (necessary for FCN only). + """ for l in a.layers: print_layer(l) in_shape = a.layers[0].input_shape[0] diff --git a/delta/ml/ml_config.py b/delta/ml/ml_config.py index 0b89f875..dad06c44 100644 --- a/delta/ml/ml_config.py +++ b/delta/ml/ml_config.py @@ -22,6 +22,8 @@ # when tensorflow isn't needed import os.path +from typing import Optional + import appdirs import pkg_resources import yaml @@ -33,14 +35,20 @@ class ValidationSet:#pylint:disable=too-few-public-methods """ Specifies the images and labels in a validation set. """ - def __init__(self, images=None, labels=None, from_training=False, steps=1000): + def __init__(self, images: Optional[ImageSet]=None, labels: Optional[ImageSet]=None, + from_training: bool=False, steps: int=1000): """ - Uses the specified `delta.imagery.imagery_config.ImageSet`s images and labels. - - If `from_training` is `True`, instead takes samples from the training set - before they are used for training. - - The number of samples to use for validation is set by `steps`. + Parameters + ---------- + images: ImageSet + Validation images. + labels: ImageSet + Optional, validation labels. + from_training: bool + If true, ignore images and labels arguments and take data from the training imagery. + The validation data will not be used for training. + steps: int + If from_training is true, take this many batches for validation. """ self.images = images self.labels = labels @@ -52,7 +60,7 @@ class TrainingSpec:#pylint:disable=too-few-public-methods,too-many-arguments Options used in training by `delta.ml.train.train`. """ def __init__(self, batch_size, epochs, loss, metrics, validation=None, steps=None, - stride=None, optimizer='Adam', max_tile_offset=None): + stride=None, optimizer='Adam'): self.batch_size = batch_size self.epochs = epochs self.loss = loss @@ -61,9 +69,11 @@ def __init__(self, batch_size, epochs, loss, metrics, validation=None, steps=Non self.metrics = metrics self.stride = stride self.optimizer = optimizer - self.max_tile_offset = max_tile_offset -class NetworkModelConfig(config.DeltaConfigComponent): +class NetworkConfig(config.DeltaConfigComponent): + """ + Configuration for a neural network. + """ def __init__(self): super().__init__() self.register_field('yaml_file', str, 'yaml_file', config.validate_path, @@ -90,6 +100,9 @@ def _load_dict(self, d : dict, base_dir): self._config_dict.update(yaml.safe_load(f)) def validate_size(size, _): + """ + Validate an image region size. + """ if size is None: return size assert len(size) == 2, 'Size must be tuple.' @@ -97,16 +110,10 @@ def validate_size(size, _): assert size[0] > 0 and size[1] > 0, 'Size must be positive.' return size -class NetworkConfig(config.DeltaConfigComponent): - def __init__(self): - super().__init__() - self.register_component(NetworkModelConfig(), 'model') - - def setup_arg_parser(self, parser, components = None) -> None: - group = parser.add_argument_group('Network') - super().setup_arg_parser(group, components) - class ValidationConfig(config.DeltaConfigComponent): + """ + Configuration for training validation. + """ def __init__(self): super().__init__() self.register_field('steps', int, 'steps', config.validate_positive, @@ -154,8 +161,11 @@ def _validate_stride(stride, _): return stride class TrainingConfig(config.DeltaConfigComponent): + """ + Configuration for training. + """ def __init__(self): - super().__init__() + super().__init__(section_header='Training') self.register_field('stride', (list, int, None), None, _validate_stride, 'Pixels to skip when iterating over chunks. A value of 1 means to take every chunk.') self.register_field('epochs', int, None, config.validate_positive, @@ -165,21 +175,19 @@ def __init__(self): self.register_field('loss', (str, dict), None, None, 'Keras loss function.') self.register_field('metrics', list, None, None, 'List of metrics to apply.') self.register_field('steps', int, None, config.validate_non_negative, 'Batches to train per epoch.') - self.register_field('max_tile_offset', int, None, config.validate_non_negative, - 'Each epoch, offset tiles by +1 up to max.') self.register_field('optimizer', (str, dict), None, None, 'Keras optimizer to use.') self.register_field('callbacks', list, 'callbacks', None, 'Callbacks used to modify training') self.register_arg('epochs', '--epochs') self.register_arg('batch_size', '--batch-size') self.register_arg('steps', '--steps') + self.register_field('log_folder', str, 'log_folder', config.validate_path, + 'Directory where dataset progress is recorded.') + self.register_field('resume_cutoff', int, 'resume_cutoff', None, + 'When resuming a dataset, skip images where we have read this many tiles.') self.register_component(ValidationConfig(), 'validation') self.register_component(NetworkConfig(), 'network') self.__training = None - def setup_arg_parser(self, parser, components = None) -> None: - group = parser.add_argument_group('Training') - super().setup_arg_parser(group, components) - def spec(self) -> TrainingSpec: """ Returns the options configuring training. @@ -198,8 +206,7 @@ def spec(self) -> TrainingSpec: validation=validation, steps=self._config_dict['steps'], stride=self._config_dict['stride'], - optimizer=self._config_dict['optimizer'], - max_tile_offset=self._config_dict['max_tile_offset']) + optimizer=self._config_dict['optimizer']) return self.__training def _load_dict(self, d : dict, base_dir): @@ -208,6 +215,9 @@ def _load_dict(self, d : dict, base_dir): class MLFlowCheckpointsConfig(config.DeltaConfigComponent): + """ + Configure MLFlow checkpoints. + """ def __init__(self): super().__init__() self.register_field('frequency', int, 'frequency', None, @@ -216,6 +226,9 @@ def __init__(self): 'If true, only keep the most recent checkpoint.') class MLFlowConfig(config.DeltaConfigComponent): + """ + Configure MLFlow. + """ def __init__(self): super().__init__() self.register_field('enabled', bool, 'enabled', None, 'Enable MLFlow.') @@ -238,6 +251,9 @@ def uri(self) -> str: return uri class TensorboardConfig(config.DeltaConfigComponent): + """ + Tensorboard configuration. + """ def __init__(self): super().__init__() self.register_field('enabled', bool, 'enabled', None, 'Enable Tensorboard.') diff --git a/delta/ml/predict.py b/delta/ml/predict.py index d52351f1..50ac2981 100644 --- a/delta/ml/predict.py +++ b/delta/ml/predict.py @@ -28,10 +28,6 @@ from delta.imagery import rectangle -#pylint: disable=unsubscriptable-object -# Pylint was barfing lines 32 and 76. See relevant bug report -# https://github.com/PyCQA/pylint/issues/1498 - class Predictor(ABC): """ Abstract class to run prediction for an image given a model. @@ -42,10 +38,18 @@ def __init__(self, model, tile_shape=None, show_progress=False): self._tile_shape = tile_shape @abstractmethod - def _initialize(self, shape, label, image): + def _initialize(self, shape, image, label=None): """ Called at the start of a new prediction. - The output shape, label image, and image being read are passed as inputs. + + Parameters + ---------- + shape: (int, int) + The final output shape from the network. + image: delta.imagery.delta_image.DeltaImage + The image to classify. + label: delta.imagery.delta_image.DeltaImage + The label image, if provided (otherwise None). """ def _complete(self): @@ -55,14 +59,41 @@ def _abort(self): """Cancel the operation and cleanup neatly.""" @abstractmethod - def _process_block(self, pred_image, x, y, labels, label_nodata): + def _process_block(self, pred_image: np.ndarray, x: int, y: int, labels: np.ndarray, label_nodata): """ - Processes a predicted block. The predictions are in pred_image, - (sx, sy) is the starting coordinates of the block, and the corresponding labels - if available are passed as labels. + Processes a predicted block. Must be overriden in subclasses. + + Parameters + ---------- + pred_image: numpy.ndarray + Output of model for a block of the image. + x: int + Top-left x coordinate of block. + y: int + Top-left y coordinate of block. + labels: numpy.ndarray + Labels (or None if not available) for same block as `pred_image`. + label_nodata: dtype of labels + Pixel value for nodata (or None). """ - def _predict_array(self, data, image_nodata_value): + def _predict_array(self, data: np.ndarray, image_nodata_value): + """ + Runs model on data. + + Parameters + ---------- + data: np.ndarray + Block of image to apply the model to. + image_nodata_value: dtype of data + Nodata value in image. If given, nodata values are + replaced with nans in output. + + Returns + ------- + np.ndarray: + Result of applying model to data. + """ net_input_shape = self._model.input_shape[1:] net_output_shape = self._model.output_shape[1:] @@ -117,9 +148,25 @@ def _predict_array(self, data, image_nodata_value): def predict(self, image, label=None, input_bounds=None, overlap=(0, 0)): """ - Runs the model on `image`, comparing the results to `label` if specified. - Results are limited to `input_bounds`. Returns output, the meaning of which - depends on the subclass. + Runs the model on an image. The behavior is specific to the subclass. + + Parameters + ---------- + image: delta.imagery.delta_image.DeltaImage + Image to evalute. + label: delta.imagery.delta_image.DeltaImage + Optional label to compare to. + input_bounds: delta.imagery.rectangle.Rectangle + If specified, only evaluate the given portion of the image. + overlap: (int, int) + `predict` evaluates the image by selecting tiles, dependent on the tile_shape + provided in the subclass. If an overlap is specified, the tiles will be overlapped + by the given amounts in the x and y directions. Subclasses may select or interpolate + to favor tile interior pixels for improved classification. + + Returns + ------- + The result of the `_complete` function, which depends on the sublcass. """ net_input_shape = self._model.input_shape[1:] net_output_shape = self._model.output_shape[1:] @@ -147,7 +194,7 @@ def predict(self, image, label=None, input_bounds=None, overlap=(0, 0)): tiles = input_bounds.make_tile_rois((block_size_x - offset_r, block_size_y - offset_c), include_partials=False, overlap_shape=(-offset_r, -offset_c)) - self._initialize(output_shape, label, image) + self._initialize(output_shape, image, label) label_nodata = label.nodata_value() if label else None @@ -193,8 +240,25 @@ class LabelPredictor(Predictor): def __init__(self, model, tile_shape=None, output_image=None, show_progress=False, # pylint:disable=too-many-arguments colormap=None, prob_image=None, error_image=None, error_colors=None): """ - output_image, prob_image, and error_image are all DeltaImageWriter's. - colormap and error_colors are all numpy arrays mapping classes to colors. + Parameters + ---------- + model: tensorflow.keras.models.Model + Model to evaluate. + tile_shape: (int, int) + Shape of tiles to process. + output_image: str + If specified, output the results to this image. + show_progress: bool + Print progress to command line. + colormap: List[Any] + Map classes to colors given in the colormap. + prob_image: str + If given, output a probability image to this file. Probabilities are scaled as bytes + 1-255, with 0 as nodata. + error_image: str + If given, output an image showing where the classification is incorrect. + error_colors: List[Any] + Colormap for the error_image. """ super().__init__(model, tile_shape, show_progress) self._confusion_matrix = None @@ -219,7 +283,7 @@ def __init__(self, model, tile_shape=None, output_image=None, show_progress=Fals self._prob_o = None self._errors = None - def _initialize(self, shape, label, image): + def _initialize(self, shape, image, label=None): net_output_shape = self._model.output_shape[1:] self._num_classes = net_output_shape[-1] if self._prob_image: @@ -332,17 +396,27 @@ class ImagePredictor(Predictor): """ def __init__(self, model, tile_shape=None, output_image=None, show_progress=False, transform=None): """ - Trains on model, outputs to output_image, which is a DeltaImageWriter. - - transform is a tuple (function, output numpy type, number of bands) applied - to the output image. + Parameters + ---------- + model: tensorflow.keras.models.Model + Model to evaluate. + tile_shape: (int, int) + Shape of tiles to process at a time. + output_image: str + File to output results to. + show_progress: bool + Print progress to screen. + transform: (Callable[[numpy.ndarray], numpy.ndarray], output_type, num_bands) + The callable will be applied to the results from the network before saving + to a file. The results should be of type output_type and the third dimension + should be size num_bands. """ super().__init__(model, tile_shape, show_progress) self._output_image = output_image self._output = None self._transform = transform - def _initialize(self, shape, label, image): + def _initialize(self, shape, image, label=None): net_output_shape = self._model.output_shape[1:] if self._output_image is not None: dtype = np.float32 if self._transform is None else np.dtype(self._transform[1]) diff --git a/delta/ml/train.py b/delta/ml/train.py index 491b2366..ce556d0b 100644 --- a/delta/ml/train.py +++ b/delta/ml/train.py @@ -44,7 +44,13 @@ class DeltaLayer(Layer): """ def callback(self): # pylint:disable=no-self-use """ - Returns a Keras callback to be added, or None. + Override this method to make a layer automatically register + a training callback. + + Returns + ------- + tensorflow.keras.callbacks.Callback: + The callback to register (or None). """ return None @@ -148,26 +154,6 @@ def on_epoch_end(self, epoch, _=None): if epoch != self.last_epoch: self.ids.reset_access_counts() -class _TileOffsetCallback(tf.keras.callbacks.Callback): - """ - Reset imagery_dataset file counts on epoch end - """ - def __init__(self, ids, max_tile_offset): - super().__init__() - self.ids = ids - self.max_tile_offset = max_tile_offset - self.ids.set_tile_offset((0, 0)) - - def on_epoch_end(self, epoch, _=None): #pylint: disable=W0613 - (tox, toy) = self.ids.tile_offset() - tox += 1 - if tox == self.max_tile_offset: - tox = 0 - toy += 1 - if toy == self.max_tile_offset: - toy = 0 - self.ids.set_tile_offset((tox, toy)) - class _MLFlowCallback(tf.keras.callbacks.Callback): """ Callback to log everything for MLFlow. @@ -254,8 +240,6 @@ def _build_callbacks(model, dataset, training_spec): callbacks.append(tcb) callbacks.append(_EpochResetCallback(dataset, training_spec.epochs)) - if training_spec.max_tile_offset: - callbacks.append(_TileOffsetCallback(dataset, training_spec.max_tile_offset)) callbacks.extend(config_callbacks()) @@ -271,7 +255,20 @@ class ContinueTrainingException(Exception): Callbacks can raise this exception to modify the model, recompile, and continue training. """ - def __init__(self, msg=None, completed_epochs=0, recompile_model=False, learning_rate=None): + def __init__(self, msg: str=None, completed_epochs: int=0, + recompile_model: bool=False, learning_rate: float=None): + """ + Parameters + ---------- + msg: str + Optional error message. + completed_epochs: int + The number of epochs that have been finished. (resumes from the next epoch) + recompile_model: bool + If True, recompile the model. This is necessary if the model has been changed. + learning_rate: float + Optionally set the learning rate to the given value. + """ super().__init__(msg) self.completed_epochs = completed_epochs self.recompile_model = recompile_model @@ -280,6 +277,20 @@ def __init__(self, msg=None, completed_epochs=0, recompile_model=False, learning def compile_model(model_fn, training_spec, resume_path=None): """ Compile and check that the model is valid. + + Parameters + ---------- + model_fn: Callable[[], tensorflow.keras.model.Model] + Function to construct a keras Model. + training_spec: delta.ml.ml_config.TrainingSpec + Trainnig parameters. + resume_path: str + File name to load initial model weights from. + + Returns + ------- + tensorflow.keras.models.Model: + The compiled model, ready for training. """ if not hasattr(training_spec, 'strategy'): training_spec.strategy = _strategy(_devices(config.general.gpus())) @@ -314,6 +325,22 @@ def train(model_fn, dataset : ImageryDataset, training_spec, resume_path=None): """ Trains the specified model on a dataset according to a training specification. + + Parameters + ---------- + model_fn: Callable[[], tensorflow.keras.model.Model] + Function that constructs a model. + dataset: delta.imagery.imagery_dataset.ImageryDataset + Dataset to train on. + training_spec: delta.ml.ml_config.TrainingSpec + Training parameters. + resume_path: str + Optional file to load initial model weights from. + + Returns + ------- + (tensorflow.keras.models.Model, History): + The trained model and the training history. """ model = compile_model(model_fn, training_spec, resume_path) assert model.input_shape[3] == dataset.num_bands(), 'Number of bands in model does not match data.' diff --git a/delta/subcommands/commands.py b/delta/subcommands/commands.py index 93a3107d..ad111e4f 100644 --- a/delta/subcommands/commands.py +++ b/delta/subcommands/commands.py @@ -48,6 +48,7 @@ def setup_classify(subparsers): sub.add_argument('--no-colormap', dest='noColormap', action='store_true', help='Save raw classification values instead of colormapped values.') sub.add_argument('--overlap', dest='overlap', type=int, default=0, help='Classify with the autoencoder.') + sub.add_argument('--validation', dest='validation', help='Classify validation images instead.') sub.add_argument('model', help='File to save the network to.') sub.set_defaults(function=main_classify) diff --git a/delta/subcommands/main.py b/delta/subcommands/main.py index 15bca996..e88934c6 100644 --- a/delta/subcommands/main.py +++ b/delta/subcommands/main.py @@ -14,6 +14,9 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +""" +Main delta command, calls subcommands. +""" import sys import argparse @@ -23,6 +26,9 @@ from delta.subcommands import commands def main(args): + """ + DELTA main function. + """ delta.config.modules.register_all() parser = argparse.ArgumentParser(description='DELTA Machine Learning Toolkit') subparsers = parser.add_subparsers() diff --git a/delta/subcommands/train.py b/delta/subcommands/train.py index 2a4828b6..34203886 100644 --- a/delta/subcommands/train.py +++ b/delta/subcommands/train.py @@ -39,7 +39,7 @@ def main(options): - log_folder = config.dataset.log_folder() + log_folder = config.train.log_folder() if log_folder: if not options.resume: # Start fresh and clear the read logs os.system('rm -f ' + log_folder + '/*') diff --git a/scripts/example/l8_cloud.sh b/scripts/example/l8_cloud.sh new file mode 100755 index 00000000..392a559b --- /dev/null +++ b/scripts/example/l8_cloud.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +# This example trains a Landsat 8 cloud classifier. This classification is +# based on the SPARCS validation data: +# https://www.usgs.gov/core-science-systems/nli/landsat/spatial-procedures-automated-removal-cloud-and-shadow-sparcs + +SCRIPT=$(readlink -f "$0") +SCRIPTPATH=$(dirname "$SCRIPT") + +if [ ! -f l8cloudmasks.zip ]; then + echo "Downloading dataset." + wget https://landsat.usgs.gov/cloud-validation/sparcs/l8cloudmasks.zip +fi + +if [ ! -d sending ]; then + echo "Extracting dataset." + unzip -q l8cloudmasks.zip + mkdir validate + mv sending/LC82290562014157LGN00_24_data.tif sending/LC82210662014229LGN00_18_data.tif validate/ + mkdir train + mv sending/*_data.tif train/ + mkdir labels + mv sending/*_mask.png labels/ +fi + +if [ ! -f l8_clouds.h5 ]; then + cp $SCRIPTPATH/l8_cloud.yaml . + delta train --config l8_cloud.yaml l8_clouds.h5 +fi + +delta classify --config l8_cloud.yaml --image-dir ./validate --overlap 32 l8_clouds.h5 diff --git a/scripts/example/l8_cloud.yaml b/scripts/example/l8_cloud.yaml new file mode 100644 index 00000000..565f9dda --- /dev/null +++ b/scripts/example/l8_cloud.yaml @@ -0,0 +1,154 @@ +dataset: + images: + type: tiff + extension: _data.tif + directory: train + labels: + extension: _mask.png + type: tiff + directory: labels + classes: + - 0: + name: Shadow + color: 0x000000 + - 1: + name: Shadow over Water + color: 0x000080 + - 2: + name: Water + color: 0x0000FF + - 3: + name: Snow + color: 0x00FFFF + - 4: + name: Land + color: 0x808080 + - 5: + name: Cloud + color: 0xFFFFFF + - 6: + name: Flooded + color: 0x808000 + +io: + tile_size: [512, 512] + +train: + loss: sparse_categorical_crossentropy + metrics: + - sparse_categorical_accuracy + network: + model: + layers: + - Input: + shape: [~, ~, num_bands] + - Conv2D: + filters: 16 + kernel_size: [3, 3] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: c1 + - Dropout: + rate: 0.2 + - MaxPool2D: + - Conv2D: + filters: 32 + kernel_size: [3, 3] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: c2 + - Dropout: + rate: 0.2 + - MaxPool2D: + - Conv2D: + filters: 64 + kernel_size: [3, 3] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: c3 + - Dropout: + rate: 0.2 + - MaxPool2D: + - Conv2D: + filters: 128 + kernel_size: [3, 3] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: c4 + - UpSampling2D: + - Conv2D: + filters: 64 + kernel_size: [2, 2] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: u3 + - Concatenate: + inputs: [c3, u3] + - Dropout: + rate: 0.2 + - Conv2D: + filters: 64 + kernel_size: [3, 3] + padding: same + - UpSampling2D: + - Conv2D: + filters: 32 + kernel_size: [2, 2] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: u2 + - Concatenate: + inputs: [c2, u2] + - Dropout: + rate: 0.2 + - Conv2D: + filters: 32 + kernel_size: [3, 3] + padding: same + - UpSampling2D: + - Conv2D: + filters: 16 + kernel_size: [2, 2] + padding: same + - BatchNormalization: + - Activation: + activation: relu + name: u1 + - Concatenate: + inputs: [c1, u1] + - Dropout: + rate: 0.2 + - Conv2D: + filters: 7 + kernel_size: [3, 3] + activation: linear + padding: same + - Softmax: + axis: 3 + batch_size: 10 + epochs: 10 + validation: + from_training: false + images: + type: tiff + extension: _data.tif + directory: validate + labels: + extension: _mask.png + type: tiff + directory: labels + +mlflow: + experiment_name: Landsat8 Clouds Example diff --git a/scripts/label-img-info b/scripts/label-img-info deleted file mode 100755 index 5bc326d3..00000000 --- a/scripts/label-img-info +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env python - -import sys -import pathlib -import numpy as np -from osgeo import gdal - -if __name__=='__main__': - assert len(sys.argv) > 1, 'Need to supply a file' - filename = pathlib.Path(sys.argv[1]) - - tif_file = gdal.Open(str(filename)) - assert tif_file is not None, f'Could not open file {filename}' - tif_data = tif_file.ReadAsArray() - unique_labels = np.unique(tif_data) - print(np.any(np.isnan(tif_data)), tif_data.min(), tif_data.max(), tif_data.shape, unique_labels) - print(np.histogram(tif_data, bins=len(unique_labels))) - - diff --git a/setup.py b/setup.py index 7dcdc756..348f0a17 100644 --- a/setup.py +++ b/setup.py @@ -47,7 +47,7 @@ ], install_requires=[ 'tensorflow>=2.1', - 'usgs', + 'usgs<0.3', 'scipy', 'matplotlib', 'mlflow', diff --git a/tests/test_commands.py b/tests/test_commands.py index c2e93233..769bd07c 100644 --- a/tests/test_commands.py +++ b/tests/test_commands.py @@ -83,15 +83,14 @@ def test_train_main(identity_config, tmp_path): steps: 5 epochs: 3 network: - model: - layers: - - Input: - shape: [1, 1, num_bands] - - Conv2D: - filters: 2 - kernel_size: [1, 1] - activation: relu - padding: same + layers: + - Input: + shape: [1, 1, num_bands] + - Conv2D: + filters: 2 + kernel_size: [1, 1] + activation: relu + padding: same batch_size: 1 validation: steps: 2 @@ -110,17 +109,15 @@ def test_train_validate(identity_config, binary_identity_tiff_filenames, tmp_pat train: steps: 5 epochs: 3 - max_tile_offset: 2 network: - model: - layers: - - Input: - shape: [~, ~, num_bands] - - Conv2D: - filters: 2 - kernel_size: [1, 1] - activation: relu - padding: same + layers: + - Input: + shape: [~, ~, num_bands] + - Conv2D: + filters: 2 + kernel_size: [1, 1] + activation: relu + padding: same batch_size: 1 validation: from_training: false diff --git a/tests/test_config.py b/tests/test_config.py index 06cf917f..32fbb62c 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -307,8 +307,7 @@ def test_network_file(): classes: 3 train: network: - model: - yaml_file: networks/convpool.yaml + yaml_file: networks/convpool.yaml ''' config.load(yaml_str=test_str) model = config_parser.config_model(2)() @@ -338,19 +337,18 @@ def test_network_inline(): classes: 3 train: network: - model: - params: - v1 : 10 - layers: - - Input: - shape: [5, 5, num_bands] - - Flatten: - - Dense: - units: v1 - activation : relu - - Dense: - units: 3 - activation : softmax + params: + v1 : 10 + layers: + - Input: + shape: [5, 5, num_bands] + - Flatten: + - Dense: + units: v1 + activation : relu + - Dense: + units: 3 + activation : softmax ''' config.load(yaml_str=test_str) assert len(config.dataset.classes) == 3 diff --git a/tests/test_tiff.py b/tests/test_tiff.py index c9402f57..e55534fc 100644 --- a/tests/test_tiff.py +++ b/tests/test_tiff.py @@ -32,12 +32,8 @@ def check_landsat_tiff(filename): input_reader = TiffImage(filename) assert input_reader.size() == (37, 37) assert input_reader.num_bands() == 8 - for i in range(0, input_reader.num_bands()): - (bsize, (blocks_x, blocks_y)) = input_reader.block_info(i) - assert bsize == (6, 37) - assert blocks_x == 1 - assert blocks_y == 7 - assert input_reader.numpy_type(i) == np.float32 + assert input_reader.dtype() == np.float32 + assert input_reader.block_size() == (6, 37) meta = input_reader.metadata() geo = meta['geotransform'] @@ -65,11 +61,10 @@ def check_same(filename1, filename2, data_only=False): in2 = TiffImage(filename2) assert in1.size() == in2.size() assert in1.num_bands() == in2.num_bands() - for i in range(in1.num_bands()): - if not data_only: - assert in1.block_info(i) == in2.block_info(i) - assert in1.data_type(i) == in2.data_type(i) - assert in1.nodata_value() == in2.nodata_value() + assert in1.dtype() == in2.dtype() + if not data_only: + assert in1.block_size() == in2.block_size() + assert in1.nodata_value() == in2.nodata_value() if not data_only: m_1 = in1.metadata()