advanced-tutorials.tex

\section{Advanced Tutorials}

This section outlines some of the more advanced features of cylc along with
links to tutorials on these features on the rose documentation website.

The final example from each tutorial has been provided as a pre-built suite on
the Metomi VM for convenience. These suites are referenced here in the ``Demo
Suite'' sections. If you don't want to go work though the full tutorials these
suites will give you working examples of the tutorials topic.


\subsection{Advanced Cycling}
\label{advanced-cycling}

So far we have defined cycling using graph section headings of the form
\lstinline{[[[TXX]]]} as in the following example:

\begin{lstlisting}
[scheduling]
    initial cycle point = 2000-01-01T00
    [[dependencies]]
        [[[T00]]]  # Run every day at 00:00
            graph = foo => bar
\end{lstlisting}

This is the simplest form of graph section heading. More powerful forms exist
to help define complex recurrences.

They all follow abbreviations of a basic format - \lstinline{Rn/DATE-TIME/REPEAT_INTERVAL}, an ISO 8601\footnote{\url{https://en.wikipedia.org/wiki/ISO_8601}} recurrence format, where \lstinline{n} is an optional limit on repetitions, \lstinline{DATE-TIME} is a starting date-time, and \lstinline{REPEAT_INTERVAL} is the duration between successive recurrences.

A date-time without a repeat interval must have a missing unit of date or time to extrapolate a repeat interval from. \lstinline{T00} above is short for \lstinline{2000-01-01T00}, and the absence of higher-level day information is taken to mean 'daily'. It is thereforce ultimately short for \lstinline{R/2000-01-01T00/P1D}. This means: starting at \lstinline{2000-01-01T00:00:00}, repeat every day (\lstinline{P1D}) indefinitely (\lstinline{R} with no number limit suffix).

\lstinline{T12} would mean the same thing but starting at \lstinline{2000-01-01T12:00:00}. Another example - \lstinline{01T00} - has missing month information, so it is taken to mean 'monthly' or \lstinline{R/2000-01-01T00/P1M}.

A repeat interval without a date-time is taken to use the initial cycle point as the start date-time. \lstinline{P2D} is taken to mean \lstinline{R/2000-01-01T00/P2D}.

A single \lstinline{R1} is taken to mean 'run once at the initial cycle point'.

\paragraph*{Examples} $ $

\begin{tabular}{l l}
\begin{lstlisting}
[[[ T06, T1845 ]]]
\end{lstlisting}
& Run every day at 06:00 and 18:45\\
\begin{lstlisting}
[[[ 01T00 ]]]
\end{lstlisting}
& Run every month on the first of the month \\
\begin{lstlisting}
[[[ PT15M ]]]
\end{lstlisting}
& Run every 5 minutes \\
\begin{lstlisting}
[[[ +P5D/P1M ]]]
\end{lstlisting}
& Run every month, starting 5 days after the initial cycle point \\
\begin{lstlisting}
[[[ R3/T06 ]]]
\end{lstlisting}
& Run three times, once every day at 06:00
\\
\end{tabular}

\vspace{5mm}

\begin{shaded*}
\textbf{Further Reading}:
\url{http://cylc.github.io/cylc/html/single/cug-html.html#9.3.4}
\end{shaded*}


\subsection{Advanced Dependencies}

The \& symbol can be used to condense multiple graph lines, for instance the
following example:

\begin{lstlisting}
graph = """
    foo => bar
    foo => baz
"""
\end{lstlisting}

... can be condensed to \lstinline{graph = foo => bar & baz}.
Graph lines can also be written with the \textbar \, symbol meaning or, for
example in the following example \lstinline{baz} will run as soon as either
\lstinline{foo} or \lstinline{bar} succeed.

\begin{lstlisting}
graph = foo | bar => baz
\end{lstlisting}

Up until now we have written dependencies in the form \lstinline{foo => bar}
which means \lstinline{bar} will trigger (run) as soon as \lstinline{foo}
succeeds. It is possible to trigger tasks off of other states e.g:

\begin{tabular}{ll}
\lstinline[mathescape]=foo:fail $=$> bar= &
\lstinline=bar= triggers if \lstinline=foo= fails \\
\lstinline[mathescape]=foo:submit $=$> bar= &
\lstinline=bar= triggers once \lstinline=foo= has submitted \\
\lstinline[mathescape]=foo:start $=$> bar= &
\lstinline=bar= triggers once \lstinline=foo= starts executing \\
\lstinline[mathescape]=foo:finish $=$> bar= &
\lstinline=bar= triggers once \lstinline=foo= succeeds or fails \\
\end{tabular}


\subsection{Families}
\label{Families}

Often a suite will contain a collection of similar tasks. With cylc these
tasks can be grouped together for convenience. This can be used to factor out a lot of repeated configuration to make suites more readable and maintainable.

\paragraph*{Example}
In the following example the
task \lstinline{hello_eris} inherits the \lstinline{script} setting and
\lstinline{IS_WORLD_A_PLANET} environment variable from the family
\lstinline{HELLO_FAIMILY}. The task \lstinline{hello_pluto} on the other hand
inherits the \lstinline{script} setting but overrides the
\lstinline{IS_WORLD_A_PLANET} environment variable.

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = HELLO_FAMILY  # Runs all tasks that inherit from HELLO_FAMILY.
[runtime]
    [[HELLO_FAMILY]]  # A family.
        script = echo $IS_WORLD_A_PLANET
        [[[environment]]]
            IS_WORLD_A_PLANET = false # Environment variable shared with tasks
                                      # that inherit from this family.
    [[hello_eris]]
        inherit = HELLO_FAMILY
    [[hello_pluto]]
        inherit = HELLO_FAMILY
        [[[environment]]]
            IS_WORLD_A_PLANET = true  # Overrides the inherited environment
                                      # variable.
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}:
\url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-family-trigs.html}

\textbf{Demo Suite}: \url{~/tutorial/suites/family-triggers}
\end{shaded*}

\subsection{Retries}
Sometimes tasks fail, you can tell cylc to automatically retry failed tasks.

\paragraph*{Example} $ $

\begin{lstlisting}
[runtime]
    [[task]]
        retry delays = 5*PT10S  # Retry up to 5 times waiting 10 seconds before
                                # each retry.
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}:
\url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-retries.html}

\textbf{Demo Suite}:
\url{~/tutorial/suites/retries}
\end{shaded*}


\subsection{Jinja2}

Jinja2 (\url{http://jinja.pocoo.org/}) is a {\em template processor} -
essentially a programming language that can be embedded in arbitrary text,
for the purpose of altering that text.  In cylc, you can think of
Jinja2 as providing programming constructs (variables, data structures,
loops, etc.) for {\em generating your suite definition}.

\paragraph{Example} $ $

\begin{lstlisting}
#!jinja2
{% set N_MEMS = 5 -%}
[scheduling]
    [[dependencies]]
        graph = """
{% for MEM in range(N_MEMS) -%}
     pre => sim_m{{MEM}} => post
{% endfor -%}
        sim_m2 => boo"""
[runtime]
{% for MEM in range(N_MEMS) -%}
    [[sim_m{{MEM}}]]
        script = echo "I'm member {{MEM}}"
{% endfor -%}
    [[boo]]
        script = echo "BOO!"
\end{lstlisting}

When this file is parsed Jinja2 generates the final suite definition, which
you can see with \lstinline=cylc view -j=:
\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
    pre => sim_m0 => post
    pre => sim_m1 => post
    pre => sim_m2 => post
    pre => sim_m3 => post
    pre => sim_m4 => post
        sim_m2 => boo"""
[runtime]
    [[sim_m0]]
        script = echo "I'm member 0"
    [[sim_m1]]
        script = echo "I'm member 1"
    [[sim_m2]]
        script = echo "I'm member 2"
    [[sim_m3]]
        script = echo "I'm member 3"
    [[sim_m4]]
        script = echo "I'm member 4"
    [[boo]]
        script = echo "BOO!"
\end{lstlisting}

The \lstinline=cylc graph= command shows the suite like this:
\begin{center}
\includegraphics[width=0.5\columnwidth]{resources/jinja2-param.png}
\end{center}

\begin{shaded*}
\textbf{Tutorial}: \url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-jinja2.html}

\textbf{Demo Suite}:
\url{~/tutorial/suites/jinja2}
\end{shaded*}

\subsection{Parameterized Tasks}

Since 6.11.0, parameter expansion can be used instead of messy Jinja2 loops
to generate tasks. Here is the parameterized task version of the
small Jinja2 example just above:

\begin{lstlisting}
[cylc]
    [[parameters]]
        m = 0..4
[scheduling]
    [[dependencies]]
        graph = """pre => sim<m> => post
                     sim<m=2> => boo"""
[runtime]
    [[sim<m>]]
        script = echo "I'm member $CYLC_TASK_PARAM_m"
    [[boo]]
        script = echo "BOO!"
\end{lstlisting}

This produces exactly the same result as the Jinja2 version, but it is much
clearer. Multiple parameters can be used at once, instead of nested Jinja2
loops.  For more on this topic see the Cylc User Guide.


\subsection{Cylc Broadcast}

\lstinline[language=bash]{cylc broadcast} is a command line utility that can
be used to change any setting contained within the \lstinline{[runtime]}
section of a suite.rc file whilst the suite is running.

\paragraph*{Example} The following line of bash script will set the remote
host for the task \lstinline{task_name}.

\begin{lstlisting}[language=bash]
$ cylc broadcast <suite_name> -n <task_name> -s [remote]host=<host_name>
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}: \url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-broadcast.html}

\textbf{Demo Suite}:
\url{~/tutorial/suites/broadcast}
\end{shaded*}

\subsection{Suicide Triggers}

Suicide triggers can be used to remove tasks from a suite's graph during
runtime.

\paragraph*{Example}

The task \lstinline{recover_from_failure} will be removed from the graph if
the task \lstinline{task} succeeds.

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        graph = """
            task:fail => recover_from_failure  # Run recover_from_failure if
                                               # task fails.
            task => !recover_from_failure      # Don't run if task succeeds.
        """
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}: \url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-suicide.html}

\textbf{Demo Suite}:
\url{~/tutorial/suites/suicide-triggers}
\end{shaded*}

\subsection{Queues}

Queues can be used to limit the number of certain tasks that are submitted or
run at any given time.

\paragraph*{Example}

In this example all \lstinline{foo}, \lstinline{bar} and \lstinline{baz} tasks
will go to the \lstinline{task_queue} queue. This queue will only allow two
tasks to run at a time.

\begin{lstlisting}
[scheduling]
    [[queues]]
        [[[task_queue]]]
            limit = 2
            members = foo, bar, baz
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}: \url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-queues.html}

\textbf{Demo Suite}:
\url{~/tutorial/suites/queues}
\end{shaded*}

\subsection{Clock Triggered Tasks}
\label{Clock Triggered Tasks}

Sometimes tasks should wait until a certain time before running, in cylc this
is possible using ''clock triggering``.

% TODO - describe as "real date-time dependencies"?

- see \ref{Clock Triggered Tasks}) 
\paragraph*{Example} $ $

\begin{lstlisting}
[scheduling]
    [[dependencies]]
        [[[T00]]]
            graph = daily_task
    [[special tasks]]
        clock-triggered = daily_task(PT0H)  # Run daily_task with 0 hours
                                            # offset from the cycle point.
\end{lstlisting}

\begin{shaded*}
\textbf{Tutorial}: \url{http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-clock-triggered.html}.

\textbf{Demo Suite}:
\url{~/tutorial/suites/clock-triggers}
\end{shaded*}