DALiuGE is a workflow graph development, management and execution framework, specifically designed to support very large scale processing graphs for the reduction of interferometric radio astronomy data sets. DALiuGE has already been used for processing large astronomical datasets in existing radio astronomy projects. It originated from a prototyping activity as part of the SDP Consortium called Data Flow Management System (DFMS). DFMS aimed to prototype the execution framework of the proposed SDP architecture.
Development and maintenance of DALiuGE is currently hosted at ICRAR and is performed by the DIA team.
Most users will need to use DALiuGE locally only when prototyping their workflow as it is designed in EAGLE. This repository provides a Makefile that simplifies the setup options for running DALiuGE locally, without needing to take into account the distributed and flexible nature of the ecosystem.
Note
It is recommended to have Docker installed when running DALiuGE as a workflow
prototyping tool. Please review the installation material for Docker here <https://docs.docker.com/engine/install/>.
The following steps are recommended for quickstarting your DALiuGE install:
- Create and enter the virtual environment
- Build the docker images
- Run the docker images
- Confirm the images are running and accessible from EAGLE using a web browser.
Creating the virtual environment is as simple as running:
make virtualenv source .venv/bin/activate
Now, in the .venv environment, run:
make docker-install
This will install a development version of the DALiuGE images and is appropriate for local installation (not a production environment).
Running both the Engine and the Translator is as a simple as:
make docker-run
It is possible to confirm the running of both by accessing the following links in a browser (Opera, Firefox, or Chrome are recommended):
http://dlg-trans.local:8084/ # Translator http://dlg-engine.local:8000/ # Node Manager http://dlg-engine.local:8001/ # Data Island Manager
With this setup running, it is now possible to translate and deploy a prototype EAGLE workflow on your local machine.
For more information about the installation and usage of the system please refer to the documentation
See the docs/ directory for more information, or visit our online
documentation
| [1] | 流 (pronounced Liu) is the Chinese character for "flow". |