Skip to content

Commit

Permalink
Make test running concurrency dynamic based on nprocs (Qiskit#1973)
Browse files Browse the repository at this point in the history
* Make test running concurrency dynamic based on nprocs

In Qiskit#1228 we decreased the concurrency used for the test runner in CI to
avoid memory pressure and failing jobs. In that commit we set it to 2 as
a quick fix so we could unblock CI and continue landing patches. A
followup was to do the analysis and find the sweet spot for concurrency
and stability. At the same time we opened mtreinish/stestr#202 to add
support to the test runner for running fractional nprocs. That is still
under discussion, but in the meantime this commit adds support to the
makefile for using fractional nprocs concurrency. It adjusts are default
concurrency from 2 to be nprocs / 2 if nprocs > 3. This should enable us
to increase the number of test workers we run on travis linux jobs but
still maintain reliability.

* Hard code osx to 2 workers

* Add debug statement about core count and worker count

* Use printf to round floats to int
  • Loading branch information
mtreinish authored and kdk committed Apr 5, 2019
1 parent ea30399 commit a63275b
Showing 1 changed file with 24 additions and 1 deletion.
25 changes: 24 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,28 @@
# This source code is licensed under the Apache License, Version 2.0 found in
# the LICENSE.txt file in the root directory of this source tree.

OS := $(shell uname -s)

ifeq ($(OS), Linux)
NPROCS := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS), Darwin)
NPROCS := 2
else
NPROCS := 0
endif # $(OS)

ifeq ($(NPROCS), 2)
CONCURRENCY := 2
else ifeq ($(NPROCS), 1)
CONCURRENCY := 1
else ifeq ($(NPROCS), 3)
CONCURRENCY := 3
else ifeq ($(NPROCS), 0)
CONCURRENCY := 0
else
CONCURRENCY := $(shell echo "$(NPROCS) 2" | awk '{printf "%.0f", $$1 / $$2}')
endif

.PHONY: env lint test test_record test_mock test_ci

# Dependencies need to be installed on the Anaconda virtual environment.
Expand Down Expand Up @@ -34,7 +56,8 @@ test_recording:
env QISKIT_TESTS=rec python3 -m unittest discover -s test -v

test_ci:
stestr run --concurrency 2
echo "Detected $(NPROCS) CPUs running with $(CONCURRENCY) workers"
stestr run --concurrency $(CONCURRENCY)

profile:
python3 -m unittest discover -p "profile*.py" -v
Expand Down

0 comments on commit a63275b

Please sign in to comment.