Skip to content

Commit

Permalink
Merge pull request #30 from Chymyst/feature/better-docs-6
Browse files Browse the repository at this point in the history
more improvements in the tutorial
  • Loading branch information
winitzki authored Dec 21, 2016
2 parents 0a6faef + b443222 commit b5a3f5d
Show file tree
Hide file tree
Showing 10 changed files with 258 additions and 234 deletions.
29 changes: 16 additions & 13 deletions docs/chymyst01.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,24 +48,27 @@ We will say that in a reaction such as
the **input molecules** are `a`, `b`, and `c`, and the **output molecules** are `d` and `e`.
A reaction can have one or more input molecules, and zero or more output molecules.

Once a reaction starts, the input molecules instantaneously disappear from the soup (they are consumed by the reaction), and then the output molecules are emitted into the soup.
Once a reaction starts, the input molecules instantaneously disappear from the soup (we say they are **consumed** by the reaction), and then the output molecules are **emitted** into the soup.

The simulator will start many reactions concurrently whenever their input molecules are available.

## Concurrent computations on the chemical machine

The chemical machine is implemented by the runtime engine of `JoinRun`.
Now, rather than merely watch as reactions happen, we are going to use this engine for running actual concurrent programs.
Rather than merely watch as reactions happen, we are going to use the chemical machine for running actual concurrent programs.

To this end, we are going to modify the chemical machine as follows:
To this end, `JoinRun` introduces three features:

1. Each molecule in the soup is required to _carry a value_. Molecule values are strongly typed: A molecule of a given sort (such as `a` or `b`) can only carry values of some fixed type (such as `Boolean` or `String`).
1. Each molecule in the soup is now required to _carry a value_.
Molecule values are strongly typed: a molecule of a given sort (such as `a` or `b`) can only carry values of some fixed type (such as `Boolean` or `String`).

2. Since molecules must carry values, we now need to specify a value of the correct type when we emit a new molecule into the soup.
2. Since molecules must carry values, we now need to specify a value of the correct type whenever we emit a new molecule into the soup.

3. For the same reason, reactions that produce new molecules will now need to put values on each of the output molecules. These output values must be _functions of the input values_, -- that is, of the values carried by the input molecules consumed by this reaction. Therefore, each reaction will now need to carry a Scala expression (called the **reaction body**) that will compute the new output values and emit the output molecules.
3. For the same reason, reactions that produce new molecules will now need to put values on each of the output molecules.
These output values must be _functions of the input values_, -- that is, of the values carried by the input molecules consumed by this reaction.
Therefore, each chemical reaction will now carry a Scala expression (called the **reaction body**) that will compute the new output values and emit the output molecules.

In this way, the chemical machine can be programmed to run arbitrary computations.
Let us see how the chemical machine can be programmed to run arbitrary computations.

We will use syntax such as `b(123)` to denote molecule values.
In a chemical reaction, the syntax `b(123)` means that the molecule `b` carries an integer value `123`.
Expand All @@ -82,25 +85,25 @@ where z = computeZ(x,y) // -- reaction body
In this example, the reaction's input molecules are `a(x)` and `b(y)`; that is, the input molecules have chemical designations `a` and `b` and carry values `x` and `y` respectively.

The reaction body is an expression that captures the values `x` and `y` from the input molecules.
The reaction body computes a value `z` out of `x` and `y` using the function `computeZ` (or any other code as needed).
The reaction body computes a value `z` out of `x` and `y` (in this example, this is done using the function `computeZ`).
The newly computed value `z` is placed onto the output molecule `a`, which is emitted back into the soup.

Another example of reaction is
Another example of a reaction is

```scala
a(x) + c(y) println(x+y) // -- reaction body with no output molecules

```

This reaction consumes the molecules `a` and `c` but does not emit any output molecules.
This reaction consumes the molecules `a` and `c` as its input, but does not emit any output molecules.
The only result of running the reaction is the side-effect of printing the number `x+y`.

![Reaction diagram a(x) + b(y) => a(z), a(x) + c(y) => ...](https://chymyst.github.io/joinrun-scala/reactions2.svg)

The computations performed by the chemical machine are _automatically concurrent_:
Whenever input molecules are available in the soup, the runtime engine will start a reaction that consumes these input molecules.
If many copies of input molecules are available, the runtime engine could start several reactions concurrently.
(The runtime engine can decide how many reactions to run depending on system load and the number of available cores.)
(The runtime engine can decide how many reactions to run depending on the system load and the number of available cores.)

The reaction body can be a _pure function_ that computes output values solely from the input values it receives from its input molecules.
If the reaction body is a pure function, it is completely safe (free of contention or race conditions) to execute concurrently several copies of the same reaction as different processes.
Expand All @@ -109,7 +112,7 @@ This is how the chemical machine achieves safe and automatic concurrency in a pu

## The syntax of `JoinRun`

So far, we have been using a kind of chemistry-resembling pseudocode to illustrate the structure of reactions in `JoinRun`.
So far, we have been using a kind of chemistry-resembling pseudocode to illustrate the structure of reactions.
This pseudocode was designed to prepare us for the actual syntax of `JoinRun`, which is only a little more verbose:

```scala
Expand All @@ -130,7 +133,7 @@ site(

```

The helper functions `m`, `site`, and `run` are defined in the `JoinRun` library.
The helper functions `m`, `site`, and `go` are defined in the `JoinRun` library.

The `site` call declares a **reaction site**, which can be visualized as a place where molecules gather and wait for their reaction partners.
We will talk later in more detail about reaction sites.
Expand Down
26 changes: 17 additions & 9 deletions docs/chymyst03.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@

## Molecule names

For debugging purposes, molecules in `JoinRun` can have names.
For debugging purposes, molecules in `JoinRun` have names.
These names have no effect on any concurrent computations.
For instance, the runtime engine will not check that each molecule is assigned a name, or that the names for different molecule sorts are different.
For instance, the runtime engine will not check that each molecule's name is not empty, or that the names for different molecule sorts are different.
Molecule names are used only for debugging: they are printed when logging reactions and reaction sites.

There are two ways of assigning a name to a molecule:

- specify the name explicitly, by using a class constructor;
- use the macros `m` and `b`.

Expand Down Expand Up @@ -54,7 +55,7 @@ y.toString // returns “fetch/B"
Emitted molecules cannot be, say, stored in a data structure or passed as arguments to functions.
The programmer has no direct access to the molecules in the soup, apart from being able to emit them.
But emitters _are_ ordinary, locally defined Scala values and can be manipulated as any other Scala values.
- Emitterss are local values of class `B` or `M`, which both extend the abstract class `Molecule`.
- Emitters are local values of class `B` or `M`, which both extend the abstract class `Molecule`.
Blocking molecule emitters are of class `B`, non-blocking of class `M`.
- Reactions are local values of class `Reaction`. Reactions are created using the function `go { case ... => ... }`.
- Only one `case` clause can be used in each reaction.
Expand All @@ -72,7 +73,7 @@ Reactions that share no input molecules can (and should) be defined in separate

## Molecules and molecule emitters

Molecules are emitted into the chemical soup using the syntax such as `c(123)`. Here, `c` is a value we define using a construction such as
Molecules are emitted into the chemical soup using the syntax such as `c(123)`. Here, `c` is a value we define using a construction such as

```scala
val c = m[Int]
Expand Down Expand Up @@ -105,7 +106,7 @@ val f = b[Int, String]

Now `f` is an emitter that takes an `Int` value and returns a `String`.

Emitterss for blocking molecules are essentially functions: their type is `B[T, R]`, which extends `Function1[T, R]`.
Emitters for blocking molecules are essentially functions: their type is `B[T, R]`, which extends `Function1[T, R]`.
The emitter `f` could be equivalently defined by

```scala
Expand All @@ -116,16 +117,23 @@ val f = new B[Int, String]("f")
Once `f` is defined like this, an emission call such as

```scala
val x = f(123)
val result = f(123)

```

will emit a molecule of sort `f` with value `123` into the soup.

The calling process in `f(123)` will wait until some reaction consumes this molecule and executes a “reply action” with a `String` value.
Only after the reaction body executes the “reply action”, the `x` will be assigned to that string value, and the calling process will become unblocked and will continue its computations.
The calling process in `f(123)` will wait until some reaction consumes this molecule and performs a **reply action** for the molecule `f`.
The reply action must pass a string value to the reply function:

```scala
go { case c(x) + f(y, r) => r((x+y).toString) }

```

Only after the reaction body executes the reply action, the `result` will be assigned to that string value, and the calling process will become unblocked and will continue its computations.

## The emission type matrix
## The type matrix of molecule emission

Let us consider what _could_ theoretically happen when we call an emitter function.
The emitter call can be either blocking or non-blocking, and it could return a value or return no value.
Expand Down
55 changes: 33 additions & 22 deletions docs/joinrun.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@

# `JoinRun` library documentation

`JoinRun` is an implementation of Join Calculus as an embedded DSL in Scala.
`JoinRun` is an embedded DSL for declarative concurrency in Scala.
It follows the **chemical machine** paradigm and provides high-level, purely functional concurrency primitives used by the `Chymyst` framework.

Currently, it compiles with Scala 2.11 and Scala 2.12 on Oracle JDK 8.

# Main structures

Join Calculus is implemented using molecule emitters, reactions, and reaction sites.
The main concepts of the chemical machine paradigm are molecule emitters, reactions, and reaction sites.

There are only two primitive operations:

Expand All @@ -18,14 +19,15 @@ There are only two primitive operations:
## Molecule emitters

Molecule emitters are instances of one of the two classes:

- `M[T]` for non-blocking molecules carrying a value of type `T`
- `B[T, R]` for blocking molecules carrying a value of type `T` and returning a value of type `R`

Molecule emitters should be defined as local values, before these molecules can be used in reactions.
Before molecules can be used in defining reactions, their molecule emitters must be defined as local values:

```scala
val x = new M[Int]("x")
val y = new B[Unit, String]("y")
val x = new M[Int]("x") // define a non-blocking emitter with name "x" and integer value type
val y = new B[Unit, String]("y") // define a blocking emitter with name "y", with empty value type, and String return type

```

Expand All @@ -41,20 +43,20 @@ val fetch = b[Unit, String] // same as new B[Unit, String]("fetch")

```

These macros will read the enclosing `val` definition at compile time and substitute the name of the variable into the class constructor.
These macros will read the enclosing `val` definition at compile time and substitute the name of the variable as a string into the class constructor.

## Emitting molecules

Molecule emitters inherit from `Function1` and can be used as functions with one argument.
Calling these functions will perform the side-effect of emitting the molecule into the soup that pertains to the reaction site to which the molecule is bound.

```scala
... M[T] extends Function1[T, Unit]
... B[T, R] extends Function1[T, R]
... M[T] extends (T => Unit) ...
... B[T, R] extends (T => R) ...

val x = new M[Int]("x") // define emitter using class constructor

// Need to define reactions - this is omitted here.
// Need to define some reactions with "x" - that code is omitted here.

x(123) // emit molecule with value 123

Expand Down Expand Up @@ -85,11 +87,11 @@ val result: Option[String] = f.timeout(100 millis)(10)

```

Emission with timeout results in an `Option` value.
Timed emission will result in an `Option` value.
The value will be `None` if timeout is reached.

Exceptions may be thrown as a result of emitting of a blocking molecule when it is unblocked:
For instance, this happens when the reaction code attempts to execute the reply action more than once.
For instance, this happens when the reaction body performs the reply action more than once, or the reaction body does not reply at all.

## Debugging

Expand Down Expand Up @@ -118,7 +120,7 @@ It is a runtime error to use `setLogLevel` or `logSoup` on molecules that are no
## Reactions

A reaction is an instance of class `Reaction`.
It is created using the `run` method with a partial function syntax that resembles pattern-matching on molecule values:
Reactions are declared using the `go` method with a partial function syntax that resembles pattern-matching on molecule values:

```scala
val reaction1 = go { case a(x) + b(y) => a(x+y) }
Expand Down Expand Up @@ -175,13 +177,12 @@ The reply action will unblock the calling process concurrently with the reaction

This reply action must be performed as `r(...)` in the reaction body exactly once, and cannot be performed afterwards.

It is a runtime error to write a reaction that either does not emit the reply action or uses it more than once.
It is a compile-time error to write a reaction that either does not perform the reply action or does it more than once.

Also, the reply action object `r` should not be used by any other reactions outside the reaction body where `r` was defined.
(Using `r` after the reaction finishes will have no effect.)

When a reaction is defined using the `run` macro, the compiler will detect some errors at compile time.
For instance, it is a compile-time error to omit the reply matcher variable from the pattern:
It is a compile-time error to omit the reply matcher variable from the pattern:

```scala
val f = b[Int, Unit]
Expand All @@ -190,12 +191,13 @@ val f = b[Int, Unit]

// this is incorrect usage because "r" is not being matched:
go { case f(x, _) => ... } // Error: blocking input molecules should not contain a pattern that matches on anything other than a simple variable
go { case f(_) = ... } // Same error message

```

## Reaction sites

Writing a reaction site (RS) will at once activate molecules and reactions:
Writing a reaction site (RS) will at once activate molecules and reactions.
Until an RS is written, molecules cannot be emitted, and no reactions will start.

Reaction sites are written with the `site` method:
Expand All @@ -212,6 +214,7 @@ All reactions listed in the RS will be activated at once.

Whenever we emit any molecule that is used as input to one of these reactions, it is _this_ RS (and no other) that will decide which reactions to run.
For this reason, we say that those molecules are "bound" to this RS, or that they are "consumed" at that RS, or that they are "input molecules" at this RS.
To build intuition, we can imagine that each molecule must travel to its reaction site in order to start a reaction, or to wait there for other molecules, if a reaction requires several input molecules.

Here is an example of an RS:

Expand Down Expand Up @@ -316,7 +319,7 @@ Whenever a reaction contains an idle blocking call, the corresponding thread wil
If the thread pool does not increase the number of available threads in this case, it is possible that the blocking call is waiting for a molecule that is never going to be emitted since no free threads are available to run reactions.
To prevent this kind of starvation, the user can surround the idle blocking calls with `BlockingIdle(...)`.

Emitterss of blocking molecules already use `BlockingIdle` in their implementation.
Emitters of blocking molecules already use `BlockingIdle` in their implementation.
The user needs to employ `BlockingIdle` explicitly only when a reaction contains blocking idle calls, such as `Thread.sleep`, synchronous HTTP calls, database queries, and so on.

Example:
Expand Down Expand Up @@ -346,17 +349,18 @@ site(pool, defaultReactionPool)(
## Fault tolerance and exceptions

A reaction body could throw an exception of two kinds:

- `ExceptionInJoinRun` due to incorrect usage of `JoinRun` - such as, failing to perform a reply action with a blocking molecule
- any other `Exception` in user's reaction code

The first kind of exception leads to stopping the reaction and printing an error message.
The first kind of exception is generated by `JoinRun` and leads to stopping the reaction and printing an error message.

The second kind of exception is handled specially for reactions marked as `withRetry`:
The second kind of exception is assumed to be generated by the user and is handled specially for reactions marked `withRetry`.
For these reactions, `JoinRun` assumes that the reaction has died due to some transient malfunction and should be retried.
Then the input molecules for the reaction are emitted again.
This will make it possible for the reaction to restart.

By default, reactions are not marked as `withRetry`, and any exception thrown by the reaction body will lead to the
By default, reactions are not marked `withRetry`, and any exception thrown by the reaction body will lead to the
input molecules being consumed and lost.

The following syntax is used to specify fault tolerance in reactions:
Expand All @@ -371,6 +375,9 @@ site(

As a rule, the user cannot catch an exception thrown in a different thread.
Therefore, it may be advisable not to use exceptions within reactions.
If there is an operation that could intermittently throw an exception, and if it is useful to retry the reaction in that case, the best way is mark the reaction `withRetry` and to make sure that all output molecules are emitted at the end of the reaction body, after any exceptions were thrown.
(Also, any other irreversible side effects should not happen before exceptions are thrown.)
In this case, the retry mechanism will be able to restart the reaction without repeating any of its side effects.

# Limitations in the current version of `JoinRun`

Expand All @@ -386,7 +393,7 @@ At the moment, this can happen with `scalatest` with code like this:

```scala
val x = m[Int]
site( & { case x(_) => } ) shouldEqual ()
site( go { case x(_) => } ) shouldEqual ()

```

Expand All @@ -396,13 +403,17 @@ A workaround is to assign a separate value to the reaction site result, and appl

```scala
val x = m[Int]
val result = site( & { case x(_) => } )
val result = site( go { case x(_) => } )
result shouldEqual ()

```

# Version history

- 0.1.3 Major changes in the API ("site", "go" instead of "join", "run") and in the terminology used in the tutorial and in the code: we now use the chemical machine paradigm more consequently, and avoid using the vague term "join". The build system now checks test code coverage (currently at 96%) and uses Scala "wartremover" plugin to check for more possible errors.

- 0.1.2 Bug fixes for singletons and for blocking molecules; benchmarks of blocking molecules.

- 0.1.0 First alpha release of `JoinRun`. Changes: implementing singleton molecules and volatile readers; several important bugfixes.

- 0.0.10 Static checks for livelock and deadlock in reactions, with both compile-time errors and run-time errors.
Expand Down
4 changes: 4 additions & 0 deletions docs/tables.css
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ code {
font-size: inherit;
}

code .highlighter-rouge {
background-color: #f8f8f8;
}

h1, h2, h3, h4, h5, h6 {
margin-top: 1em;
margin-bottom: 0.6em;
Expand Down
Loading

0 comments on commit b5a3f5d

Please sign in to comment.