Skip to content

Latest commit

 

History

History
257 lines (186 loc) · 10.7 KB

CHANGELOG.org

File metadata and controls

257 lines (186 loc) · 10.7 KB

0.4.4

  • Return results using test.check’s Result protocol, rather than custom exceptions. (Thanks to @r0man for doing the hard work here.)
  • Make mutation detection optional, through the :run :assume-immutable-results option. This delays converting command results into strings until the end of the test run, which may make it easier to provoke some race conditions.

    To demonstrate the difference between in output, let’s look at this test case:

    (def test-atom (atom 0))
(deftest returning-atom-as-result
  (is (specification-correct?
       {:commands {:inc {:command #(do (swap! test-atom inc)
                                       test-atom)
                         :next-state (fn [s _ _] (inc s))
                         :postcondition (fn [_ ns _ result]
                                          (is (= ns @result)))}}
        :setup #(reset! test-atom 0)
        :initial-state (constantly 0)})))

    With the default of :assume-immutable-results as false we see this output:

    Sequential prefix:
  #<4> = (:inc)  = #atom[1 0x71f8eaa9]
    >> object may have been mutated later into #atom[2 0x71f8eaa9] <<

    expected: (= ns @result)
      actual: (not (= 1 2))
  #<5> = (:inc)  = #atom[2 0x71f8eaa9]

    With :assume-immutable-results set to true we see this output:

    Sequential prefix:
  #<4> = (:inc)  = #atom[2 0x71f8eaa9]
    expected: (= ns @result)
      actual: (not (= 1 2))
  #<5> = (:inc)  = #atom[2 0x71f8eaa9]

    If you compare the result of command #<4> in both cases you can see the difference in output. The assertions in both cases reflect the final value (i.e. actual is 2 in both cases) because postconditions run after the execution phase.

  • Make the :command-frequency? table include a 0 count for commands that were not run.

0.4.3

  • Add equals/hashCode methods to LookupVars, to allow using the result of get on a symbolic value to be a key in a hash map
  • Resolve symbolic values within arguments, not just at the top level. This allows you to use arguments which are partially generated, and partially results from previous commands. For example: 
    (def create-user-command
  {:args (fn [{:keys [user-ids]}]
           [{:id (gen/elements user-ids), :name gen/string-ascii}])
   :command #(update-user %)})
  • Fix shrinking of parallel command sequences. Due to a bug, shrinking of parallel command sequences only explored removing a single command, rather than removing pairs of commands. This may have led to poorer shrinks for parallel test cases.
  • Add the ability to customise the shrink strategy for a run. The default shrink strategy is the same as before, but you can use the :gen :shrink-strategies option to tune shrinking for your use case. See stateful-check.shrink-strategies for the currently available strategies, or you can write your own (although this is undocumented).
  • Add the ability to use is in postconditions to get more specific feedback about failures. The expected/actual values from failing assertions will be captured and printed alongside the command trace, to aid in debugging failures. For example: 
    (def get-command
  {:requires (fn [state] (seq state))
   :args (fn [state] [(gen/elements test-keys)])
   :command #(.get system-under-test %1)
   :postcondition (fn [prev-state _ [k] val]
                    (is (= (get prev-state k) val) (str "Looking up " (pr-str k) " matches the model")))})

    This command from the associated test file will give a trace that looks like this:

    FAIL in (postcondition-prints-in-parallel-case) (core.clj:211)
Sequential prefix:

Thread a:
  #<2a> = (:put "" 0)  = nil

Thread b:
  #<1b> = (:put "a" 0)  = nil
  #<4b> = (:get "a")  = nil
    Looking up "a" matches the model
    expected: (= (get prev-state k) val)
      actual: (not (= 0 nil))

Seed: 1620654751933
  Note: Test cases with multiple threads are not deterministic, so using the
        same seed does not guarantee the same result.

expected: all executions to match specification
  actual: the above execution did not match the specification

    If the postcondition makes any assertions then the return value is not used. This means that the following postcondition will always pass.

    {:postcondition
 (fn [_ _ _ _]
   (is true)
   false)}

    When performing tests with multiple threads, postcondition failures are a bit tricky to interpret. The postcondition may result from any one (or more) of the interleavings of the commands in the parallel threads.

  • Add :command-frequency? to the :report options, to print out a table of how often the test runner executed each command.

0.4.2

  • Fix printing of exceptions within postconditions (#12).

0.4.1

  • Print out failing seed (#8).
  • Add :timeout-ms option to fail a test after a certain amount of time (suggestion made after Clj-Syd presentation).
  • Add forward compatibility for clojure.test.check version 0.10.0-alpha4.

0.4.0

Many, many, many changes. I’ll try to go through them.

  • Change to the MIT license.
  • Remove real/ and model/ prefixes from keys. They don’t mean as much, given the other changes that will be explained below.
  • Remove the specification :real/postcondition function. It doesn’t really fit in the context of parallel tests.
  • The command execution phase is now separate to the trace verification stage. This means that :postcondition functions on commands no longer run interleaves with runs of :command functions. Now all the :command functions are run in a sequence, which is then checked by the :postcondition functions. In particular this means that :postcondition shouldn’t interact with the SUT at all!
  • Re-work the options map. It now has three parts: :gen, :run, and :report.
  • Add support for running parallel tests, to try to find race conditions. Use {:gen {:threads 2}} to generate threads with two parallel threads, and {:run {:max-tries 10}} to try 10 times to provoke the race condition on each test.
  • Removed deprecated functions.

0.3.1

  • :model/args now coerces the returned values into a generator. Coercion works like the following:
    • if it’s a generator: return it
    • if it’s a sequential collection: coerce each element into a generator, then use gen/tuple to combine them
    • if it’s a map: coerce each value into a generator, then use gen/hash-map to combine each key/value-gen pair
    • anything else: return it using gen/return
  • :model/generate-command now has a default implementation. If you don’t provide an implementation then it will select a command at random (effectively: (gen/elements (:commands spec))).
  • If a value in the :command map is a var then dereference it (to facilitate breaking up specs a bit more).
  • Command results are now printed properly when the results of commands are mutated. Previously it would print the command results in their state at the end of the test, irrespective of where they actually were returned. Now the results will be printed prior to running the next command in the sequence.

    It used to print something like this:

    #<0> = (:new) => #{10}
#<1> = (:contains? #<0> 10) => false
#<2> = (:add #<0> 10) => true
#<3> = (:contains? #<0> 10) => true

    This incorrectly shows the state of the test (at the point when it was created) to have the element 10 in it. The 10 wasn’t added until command #<2>, however, so that output is incorrect. This could cause us to think the set’s implementation is wrong when it is actually a quirk of stateful-check causing this problem.

    It will now print something like this:

    #<0> = (:new) => #{}
#<1> = (:contains? #<0> 10) => false
#<2> = (:add #<0> 10) => true
#<3> = (:contains? #<0> 10) => true
  • :real/setup and :real/cleanup had some major issues (not running being prime among them) which are now fixed. A test has been added to hopefully avoid this happening again in future.
  • Add a :tries argument to the specification-correct? options map. This runs each test a number of times, with any failure causing the run to fail. (Useful for non-deterministic tests.)
  • Shrinking is now a bit more aggressive. In particular, now it will start by trying to shrink single commands (whether by removing the command or by shrinking its arguments), but then it will also try to shrink pairs of commands (removing/shrinking both at the same time). This can lead to dramatically better shrinks in some situations.

0.3.0

  • Breaking! Add next-state to the :real/postcondition function arguments in commands.

    Any command preconditions will need to be modified to take an extra argument.

    (fn [state args result]
  arbitrary-logic-for-postcondition)
;; needs to change to
(fn [prev-state next-state args result]
  arbitrary-logic-for-postcondition)
  • Breaking! Change reality-matches-model? to be called reality-matches-model (it’s not a predicate, so it shouldn’t have a ? in its name). This function is now deprecated, though, in favour of using deftest with our custom is form (see the next point).
  • Add support for a custom test.check is form: 
    (is (specification-correct? some-spec))
(is (specification-correct? some-spec {:num-tests 1000, :max-size 10, :seed 123456789}))
  • Make the command generator use the same size for all commands.
  • Rewrite the command verifier/runner to make it a whole lot cleaner (including breaking out extra namespaces).
  • Upgrade to test.check 0.7.0.
  • Tweak the format of print-test-results.

0.2.0

  • Add namespaces to some keys which didn’t have them before
    • :generate-command is now :model/generate-command
    • :setup is now :real/setup
    • :cleanup is now :real/cleanup
  • Add some more keys to the top-level spec object:
    • :model/initial-state, :real/initial-state, :initial-state for setting the initial state of the system
    • :real/postcondition on the top-level spec, to check for global invariants
  • Make symbolic values implement ILookup (to work with get)
  • Clean up exception handling during command runs

0.1.0

Initial release.