Merge pull request #30 from SKA-ScienceDataProcessor/feature-SIM-407

Stable docker release

.gitlab-ci.yml

@@ -9,11 +9,9 @@ docs:
     - apt-get update
     - apt-get -y install pandoc rsync
     - pip install -r requirements.txt -r requirements-docs.txt
-    - python -c "import sys;print(sys.path)"
     - mkdir -p docs/build/html
     - mkdir -p test_results
     - rsync -avz rsync://casa-rsync.nrao.edu/casa-data/geodetic /build/
-    - ls -l /build/geodetic
   script:
     - PYTHONPATH=`pwd` HOME=`pwd` make -k -C docs html
   after_script:
@@ -29,9 +27,8 @@ unittests:
     - apt-get update
     - apt-get -y install rsync
     - pip install -r requirements.txt
-    - python -c "import sys;print(sys.path)"
+    - mkdir -p test_results
     - rsync -avz rsync://casa-rsync.nrao.edu/casa-data/geodetic /build/
-    - ls -l /build/geodetic
   script:
     - pip install pytest pytest-xdist pytest-cov
     - HOME=`pwd` py.test tests/workflows/test*rsexecute.py --verbose --cov=rascil --cov-report=html:coverage --durations=30 --forked

CHANGELOG.md

@@ -1,124 +0,0 @@
-**May 1, 2019** [Tim] New workflow to map a function across an image: image_rsexecute_map_workflow
-
-**April 28, 2019** [Tim] Fill in all __init__.py files so that tab completion works.
-
-**April 16, 2019** [Tim] Added examples/comparisons directory hold code for comparing ARL results with those from 
-other packages.
-
-**April 04, 2019** [Tim] Pointing simulation capabilities put in place. New data model PointingTable similar to 
-GainTable. testing_support now has a function to simulate pointing errors. See workflows/scripts/SKA_SIM for examples.
-
-**March 25, 2019** [Tim] Performance measurement capabilites and improvements. See workflows/scripts/performance for 
-tests.
-
-**January 26, 2019** [Feng] Added create_blockvisibility_from_uvfits.
-
-**January 23, 2019** [Tim] Work on MPC. Documented in SDP memo 97.
-Branch feature_mpc merged into master. The old modelpartition code has
-been removed.
-1. SkyModel now has image, mask, and gaintable.
-2. Gaintable now has phasecentre, as needed for non-isoplanatic 
-processing.
-3. Improved speed of scalar calibration.
-4. Improved HDF5 for SkyModel. Old ones not compatible.
-5. Functions for working with skycomponents e.g. filtering, 
-neighbour filtering.
-6. Voronoi partitioning.
-7. Predict and invert workflows for skymodel.
-8. Ensure imaging_weights retained in BlockVisibility.
-9. Ionospheric phase screen simulation using 
-ARatmospy: create_gaintable_from_screen. Gaintable phases can be 
-back-propagated to screen: grid_gaintable_to_screen.
-10. Pipeline for MPCCAL: mpccal_skymodel_list_rsexecute_workflow.
-
-**November 26, 2018** [Tim] Many changes in run up to demo:
-1. Parallel weighting, imaging weight added to BlockVisibility
-2. GLEAM lookup optimised, can now generate a skymodel and predict from it.
-3. Added skycomponent filtering
-4. Default visibility format is Visibility not BlockVisibility. This means fewer conversions.
-5. arl_demo notebook in place
-
-**September 25, 2018** [Tim] Gridding/degridding code has been refactored to a cleaner design. It also allows more 
-control of AW projection.
-
-**September 18, 2018** [Rodrigo] Fix for daliuge port execution of list, to match dask.delayed capabilities. Need to
-update version of daliuge:
-
-    pip install "daliuge>=0.5.3" pytest
-
-**September 16, 2018** [Tim] Deleted imaging-pipelines-sip.ipynb. Use imaging-pipelines-serial.ipynb instead.
-
-**September 4, 2018** [Rodrigo]  Added add support for daliuge as an (experimental) backend of the rsexecute module. 
-Support uses daliuge's delayed function, which accepts the same parameters as dask's; therefore the change is simple, 
-and transparent to the rest of the ARL code. All these changes are within the context of [SDP ticket 
-TSK-2569](https://jira.ska-sdp.org/browse/TSK-2569). See also the 
-[Integration of ARL with the DALiuGE Execution Framework, part 2](https://confluence.ska-sdp.org/display/WBS/Integration+of+ARL+with+the+DALiuGE+Execution+Framework%2C+part+2)
-
-**August 16, 2018** [Tim] More refactoring to being closer alignment with SDP architecture.
-1. There are now wrappers for all processing components, both serial and 
-rsexecute. At the moment, these are just pass-throughs but the point is that they can
-be expanded as appropriate. The non-python wrappers will be more substantial.
-2. There are only workflows for calibration, imaging, and pipelines.
-3. To distinguish the nature of the workflows, these are now all called something like
-predict_list_rsexecute_workflow since they all work on lists of data models rather 
-than just data models.
-4. The workflows for serial and rsexecute should work alike. For example, all now 
-expect lists of Data Models. This is compared to processing_components
-where only single Data Models are accepted. A necessary consequence is
-that the full range of imaging algorithms are only available via 
-workflows, either as serial or rsexecute versions (and soon other
-types of wrappers).
-5. libs has been renamed to processing_library.
-
-All Dask/rsexecute code now lives in either wrappers or workflows.
-
-![ARL Module View](./docs/ARL_Module_View.png)
-
-
-**July 26, 2018** [Tim], Extracted pure-serial uses of processing components 
-into workflows/serial (in analogy with workflows/rsexecute). This means that
-all functions remaining in processing components are suitable for use in
-workflows. The split between processing components and workflows is clearer.
-As a consequence nearly all notebooks have moved to workflows/notebooks.
-
-**July 24, 2018** [Tim], Renamed calskymodel to modelpartition to 
-be in line with the SDP model views. Also documentation cleanup.
-
-**June 15, 2018** [Tim], Some more moves and renaming:
-* processing_components/component_support->libs/execution support
-* processing_components/util->processing_components/simulation
-
-generic functions moved to image_components and visibility_components
-
-**June 15 2018** [Tim], the capabilities for reading measurement sets have been improved.
-* Both BlockVisibility's and Visibility's can be created. The former is preferred.
-* A channel range e.g. range(17,32) can be specified.
-* See tests/processing_components/test_visibility_ms for various ways to use this capability.
-
-**June 14, 2018 [Tim]**, BufferDataModel has been introduced as the root of e.g. BufferImage, BufferSkyModel. All of 
-these, except for BufferImage use ad hoc HDF5 files. Image can use fits.
-
-**June 12, 2018 [Tim]**, To fill out the architecture, there is now a ProcessingComponentInterface function for executing 
-some components. Components have to be wrapped by hand, and the interface defined via a JSON file.
-
-**May 25, 2018** [Piers], Kubernetes support added.
-
-**April 30 2018** [Tim], the ARL has been updated to be consistent with the SDP Processing Architecture. This required 
-very substantial changes throughout. The code is consistent internally but ARL code kept outside the code tree will 
-need to be updated manually.
-
-* The top level directory arl has been split into three: libs, processing_components, and workflows
-    - libs contains functions that are not accessed directly by the Execution Framework
-    - processing_components contains functions that may be accessed by the EF. 
-    - workflows contains high level workflows using the processing_components. This eventually will migrate to the EF
-     but some are kept here as scripts or notebooks.
-* The tests and notebooks have been moved to be inside the appropriate directory.
-* The data definitions formerly in rascil/data have been moved to a top level directory data_models. 
-* The top level Makefile has been updated
-* The docs have been updated
-* The use of the term 'graph' has been replaced in many places by 'list' to reflect the wrapping of dask in 
-rsexecute.
-
-**April 18, 2018** [Tim], Deconvolution can now be done using overlapped, tapered sub-images (aka facets).
-Look for deconvolve_facets, deconvolve_overlap, and deconvolve_taper.
-

docker/docker-compose.yml

@@ -12,7 +12,7 @@ services:
     ports:
       - "8786:8786"
       - "8787:8787"
-    command: ["dask-scheduler", "--local-directory", "/tmp"]
+    command: ["dask-scheduler"]
 
   worker:
     build:

docker/rascil-no-data/Dockerfile

@@ -9,9 +9,10 @@ ENV DEBIAN_FRONTEND=noninteractive
 
 RUN apt-get update -y && apt-get install -y \
     ca-certificates \
-    gosu git wget curl \
+    gosu git wget curl rsync \
     libxml2-dev liblapack-dev libcfitsio-dev libgtkmm-3.0-dev \
-    cmake g++ build-essential gfortran libpng-dev casacore-dev \
+    cmake g++ build-essential gfortran libpng-dev casacore-dev python-numpy \
+    python-setuptools libboost-python-dev wcslib-dev \
     python3 python3-dev python3-pip python3-setuptools && \
     apt-get clean -y && \
     rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*
@@ -34,6 +35,7 @@ RUN mkdir -p /rascil/dask-worker-space && \
     chmod 777 /rascil/dask-worker-space
 
 RUN pip3 install Cython setuptools && \
+    pip3 install python-casacore==3.1.1 --upgrade && \
     pip3 install aotools \
     astropy \
     dask \
@@ -44,17 +46,19 @@ RUN pip3 install Cython setuptools && \
     jupyter_contrib_nbextensions \
     jupyter-server-proxy \
     jupyterlab \
-    "matplotlib<3.1.3" \
+    matplotlib \
     photutils\
-    python-casacore \
     reproject \
     scikit-image \
     scipy \
-    seqfile && \
+    seqfile \
+    tabulate && \
     pip3 install "numpy>=1.16.0" && \
     python3 setup.py build && python3 setup.py install && \
     rm -rf /root/.cache
 
+RUN rsync -avz rsync://casa-rsync.nrao.edu/casa-data/geodetic /var/lib/casacore/data
+
 # Use entrypoint script to create a user on the fly and avoid running as root.
 RUN chmod +x /rascil/entrypoint.sh
 ENTRYPOINT ["/rascil/entrypoint.sh"]

docker/rascil-notebook/Dockerfile

@@ -16,5 +16,5 @@ RUN curl -sL https://deb.nodesource.com/setup_13.x | bash - && \
     apt-get install -y nodejs
 
 RUN pip3 install dask-labextension \
-    && jupyter labextension install @jupyter-widgets/jupyterlab-manager dask-labextension@1.0.1
+    && jupyter labextension install @jupyter-widgets/jupyterlab-manager dask-labextension
 

docs/installation/RASCIL_docker.rst

@@ -113,6 +113,16 @@ Click on the 127.0.0.1 URL. We have used the jupyter lab interface instead of ju
 because the former allows control of Dask from the interface. This can be changed in the docker-compose.yml
 file. Note also that the classic notebook interface can be selected at the lab interface.
 
+CASA Measures Tables
+--------------------
+
+We use the CASA measures system for TAI/UTC corrections. These rely upon tables downloaded from NRAO.
+It may happen that the tables become out ofdate. If so do the following at the command prompt inside a
+docker image::
+
+    rsync -avz rsync://casa-rsync.nrao.edu/casa-data/geodetic /var/lib/casacore/data
+
+
 Customisability
 ---------------