From 4803d56d394a7ad0f03d5bd515cd2c56f887dfce Mon Sep 17 00:00:00 2001 From: Markus Kohlhase Date: Mon, 24 Oct 2011 04:05:26 +0200 Subject: [PATCH 1/2] added description from osadl.org --- README.markdown | 87 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 README.markdown diff --git a/README.markdown b/README.markdown new file mode 100644 index 0000000..3ab6ddf --- /dev/null +++ b/README.markdown @@ -0,0 +1,87 @@ +# RT-Tests + +This repository contains some programs that test various rt-linux features. + +## Usage + +### Compile + + sudo apt-get install build-essential libnuma-dev + make + +### Run tests + +To run one test thread per CPU or per CPU core, each thread on a separate +processor, type + + sudo ./cyclictest -a -t -n -p99 + +On a non-realtime system, you may see something like + + T: 0 ( 3431) P:99 I:1000 C: 100000 Min: 5 Act: 10 Avg: 14 Max: 39242 + T: 1 ( 3432) P:98 I:1500 C: 66934 Min: 4 Act: 10 Avg: 17 Max: 39661 + +The rightmost column contains the most important result, i.e. the worst-case +latency of 39.242 milliseconds. On a realtime-enabled system, the result may +look like + + T: 0 ( 3407) P:99 I:1000 C: 100000 Min: 7 Act: 10 Avg: 10 Max: 18 + T: 1 ( 3408) P:98 I:1500 C: 67043 Min: 7 Act: 8 Avg: 10 Max: 22 + +and, thus, indicate an apparent short-term worst-case latency of 18 +microseconds. + +Running cyclictest only over a short period of time and without creating +appropriate real-time stress conditions is rather meaningless, since the +execution of an asynchronous event from idle state is normally always quite +fast, and every - even non-RT system - can do that. The challenge is to minimize + the latency when reacting to an asynchronuous event, irrespective of what code +path is executed at the time when the external event arrives. +Therefore, specific stress conditions must be present while cyclictest is +running to reliably determine the worst-case latency of a given system. + +### Latency fighting + +If - as in the above example - a low worst-case latency is measured, and this is +the case even under a system load that is equivalent to the load expected under +production conditions, everything is alright. +Of course, the measurement must last suffciently long, preferably 24 hours or +more to run several hundred million test threads. If possible, the `-i` command +line option (thread interval) should be used to increase the number of test +threads over time. +As a role of thumb, the thread interval should be set to a value twice as long +as the expected worst-case latency. If at the end of such a test period the +worst-cae latency still did not exceed the value that is assumed critical for a +given system, the particular kernel in combination with the hardware in use can +then probably be regarded as real-time capable. + +What, however, if the latency is higher than acceptable? Then, the famous +"*latency fighting*" begins. For this purpose, the cyclictest tool provides the +`-b` option that causes a function tracing to be written to +`/sys/kernel/debug/tracing/trace`, if a specified latency threshold was +exceeded, for example: + + ./cyclictest -a -t -n -p99 -f -b100 + +This causes the program to abort execution, if the latency value exceeds 100 +microseconds; the culprit can then be found in the trace output at +`/sys/kernel/debug/tracing/trace`. +The kernel function that was executed just before a latency of more than 100 +microseconds was detected is marked with an exclamation mark such as + + qemu-30047 2D.h3 742805us : __activate_task+0x42/0x68 (199 1) + qemu-30047 2D.h3 742806us : __trace_start_sched_wakeup+0x40/0x161 (0 -1) + qemu-30047 2DNh3 742806us!: try_to_wake_up+0x422/0x460 (199 -5) + qemu-30047 2DN.1 742939us : __sched_text_start+0xf3/0xdcd (c064e442 0) + +The first column indicates the calling process responsible for triggering the +latency. + +If the trace output is not obvious, it can be submitted to the OSADL Latency +Fight Support Service at +[latency-fighters@osadl.org](mailto:latency-fighters@osadl.org). +In addition to the output of `cat /sys/kernel/debug/tracing/trace`, the output +of `lspci` and the `.config` file that was used to build the kernel in question +must be submitted. We are sure you understand that OSADL members will be served +first, but we promise to do our best to help everybody to successfully fight +against kernel and driver latencies. From 259530bb56cd64afc1d44e2271e854beb8adea50 Mon Sep 17 00:00:00 2001 From: Markus Kohlhase Date: Tue, 21 Feb 2012 13:17:33 +0100 Subject: [PATCH 2/2] removed osadl description of the cyclictest and added overview of the available tests --- README.markdown | 90 +++++++++++-------------------------------------- 1 file changed, 20 insertions(+), 70 deletions(-) diff --git a/README.markdown b/README.markdown index 3ab6ddf..8b1af20 100644 --- a/README.markdown +++ b/README.markdown @@ -2,86 +2,36 @@ This repository contains some programs that test various rt-linux features. -## Usage +## Available Tests -### Compile +- backfire - send a signal from driver to user and measure time intervals - sudo apt-get install build-essential libnuma-dev - make - -### Run tests - -To run one test thread per CPU or per CPU core, each thread on a separate -processor, type - - sudo ./cyclictest -a -t -n -p99 +- cyclictest - ? -On a non-realtime system, you may see something like +- hackbench - measues the latency of schedulable entities which + communicate via sockets/pipes - T: 0 ( 3431) P:99 I:1000 C: 100000 Min: 5 Act: 10 Avg: 14 Max: 39242 - T: 1 ( 3432) P:98 I:1500 C: 66934 Min: 4 Act: 10 Avg: 17 Max: 39661 +- hwlatdetect - detect if System Management Interrupts (SMIs) are causing + event latencies in the Linux RT kernel -The rightmost column contains the most important result, i.e. the worst-case -latency of 39.242 milliseconds. On a realtime-enabled system, the result may -look like +- pi_tests - a stress test that is intended to exercise kernel and C + library code paths for POSIX mutexes using the Priority + Inheritance attribute (PTHREAD_PRIO_INHERIT) - T: 0 ( 3407) P:99 I:1000 C: 100000 Min: 7 Act: 10 Avg: 10 Max: 18 - T: 1 ( 3408) P:98 I:1500 C: 67043 Min: 7 Act: 8 Avg: 10 Max: 22 +- pmqtest - measures the latency of interprocess communication with + POSIX messages queues -and, thus, indicate an apparent short-term worst-case latency of 18 -microseconds. +- rt-migrate-test - ? -Running cyclictest only over a short period of time and without creating -appropriate real-time stress conditions is rather meaningless, since the -execution of an asynchronous event from idle state is normally always quite -fast, and every - even non-RT system - can do that. The challenge is to minimize - the latency when reacting to an asynchronuous event, irrespective of what code -path is executed at the time when the external event arrives. -Therefore, specific stress conditions must be present while cyclictest is -running to reliably determine the worst-case latency of a given system. +- signaltest - ? -### Latency fighting +- sigwaittest - measures the latency between sending and receiving a signal + between threads or processes -If - as in the above example - a low worst-case latency is measured, and this is -the case even under a system load that is equivalent to the load expected under -production conditions, everything is alright. -Of course, the measurement must last suffciently long, preferably 24 hours or -more to run several hundred million test threads. If possible, the `-i` command -line option (thread interval) should be used to increase the number of test -threads over time. -As a role of thumb, the thread interval should be set to a value twice as long -as the expected worst-case latency. If at the end of such a test period the -worst-cae latency still did not exceed the value that is assumed critical for a -given system, the particular kernel in combination with the hardware in use can -then probably be regarded as real-time capable. +- svsematest - measures the latency of SYSV semaphores -What, however, if the latency is higher than acceptable? Then, the famous -"*latency fighting*" begins. For this purpose, the cyclictest tool provides the -`-b` option that causes a function tracing to be written to -`/sys/kernel/debug/tracing/trace`, if a specified latency threshold was -exceeded, for example: - ./cyclictest -a -t -n -p99 -f -b100 +## Compile -This causes the program to abort execution, if the latency value exceeds 100 -microseconds; the culprit can then be found in the trace output at -`/sys/kernel/debug/tracing/trace`. -The kernel function that was executed just before a latency of more than 100 -microseconds was detected is marked with an exclamation mark such as - - qemu-30047 2D.h3 742805us : __activate_task+0x42/0x68 (199 1) - qemu-30047 2D.h3 742806us : __trace_start_sched_wakeup+0x40/0x161 (0 -1) - qemu-30047 2DNh3 742806us!: try_to_wake_up+0x422/0x460 (199 -5) - qemu-30047 2DN.1 742939us : __sched_text_start+0xf3/0xdcd (c064e442 0) - -The first column indicates the calling process responsible for triggering the -latency. - -If the trace output is not obvious, it can be submitted to the OSADL Latency -Fight Support Service at -[latency-fighters@osadl.org](mailto:latency-fighters@osadl.org). -In addition to the output of `cat /sys/kernel/debug/tracing/trace`, the output -of `lspci` and the `.config` file that was used to build the kernel in question -must be submitted. We are sure you understand that OSADL members will be served -first, but we promise to do our best to help everybody to successfully fight -against kernel and driver latencies. + sudo apt-get install build-essential libluma1 libnuma-dev + make