Testing with Testscript

Testscript for E2E Testing

DSE Projects can use Testscript to run End-to-end tests (E2E) which are written in the (simple) txtar format. The containerised runtime supports both the Simer simulation runner as well as Taskfile based workflows.

Code Layout and Test Organisation

L- tests/testscript    Directory containing Testscript tests.
  L- e2e               Collection of E2E tests.
    L- testcase.txtar  Individual testcase (in txtar format).
L- Makefile            High-level build automation.

Example Test Files

env NAME=minimal_inst
env SIM=dse/modelc/build/_out/examples/minimal
env SANDBOX=dse/modelc/build/_out


# TEST: MStep executable
exec sh -e $WORK/test.sh

stdout 'Model Function: model_step'
stdout 'Starting Simulation \(for 10 steps\) ...'
stdout 'step 9 \(model_time=0.004500\)'
stdout 'Simulation complete.'
stdout 'value: 10'

-- test.sh --
cd /repo/$SIM
/repo/$SANDBOX/bin/mstep \
    --logger 2 \
    --name $NAME \
    data/model.yaml \
    data/simulation.yaml

env NAME=minimal_inst
env SIM=dse/modelc/build/_out/examples/minimal


# TEST: minimal example model
exec sh -e $WORK/test.sh

stdout 'Load YAML File: data/simulation.yaml'
stdout 'Loading symbol: model_create ... not found'
stdout 'Loading symbol: model_step ... ok'
stdout 'Loading symbol: model_destroy ... not found'
stdout 'Run the Simulation ...'
stdout 'Controller exit ...'
stdout 'SignalValue: 2628574755 = 4.000000 \[name=counter\]'

-- test.sh --
SIMER="${SIMER:-ghcr.io/boschglobal/dse-simer:latest}"
docker run --name simer -i --rm -v $ENTRYDIR/$SIM:/sim \
    $SIMER -valgrind $NAME -env $NAME:SIMBUS_LOGLEVEL=2

###############
## Docker Images.
TESTSCRIPT_IMAGE ?= ghcr.io/boschglobal/dse-testscript:main
SIMER_IMAGE ?= ghcr.io/boschglobal/dse-simer:main


###############
## Test Parameters.
export TESTSCRIPT_E2E_DIR ?= tests/testscript/e2e
TESTSCRIPT_E2E_FILES = \
	$(TESTSCRIPT_E2E_DIR)/exec.txtar \
	$(TESTSCRIPT_E2E_DIR)/simer.txtar \


.PHONY: test
test: test_e2e

.PHONY: test_e2e
test_e2e: do-test_testscript-e2e

do-test_testscript-e2e:
# Test debug; add '-v' to Testscript command (e.g. $(TESTSCRIPT_IMAGE) -v \).
ifeq ($(PACKAGE_ARCH), linux-amd64)
	@set -eu; for t in $(TESTSCRIPT_E2E_FILES) ;\
	do \
		echo "Running E2E Test: $$t" ;\
		docker run -it --rm \
			-e ENTRYDIR=$$(pwd) \
			-v /var/run/docker.sock:/var/run/docker.sock \
			-v $$(pwd):/repo \
			$(TESTSCRIPT_IMAGE) \
				-e ENTRYDIR=$$(pwd) \
				-e SIMER=$(SIMER_IMAGE) \
				$$t ;\
	done;
endif

Testing Features and Integrations

Testscript is provided as a containerised runtime with a minimal set of features and integrations. Use this runtime to run other containerised tools and advanced processing workflows.

Volumes and Paths

The containerised runtime has its working directory (WORKDIR) set to /repo and this path will typically be mapped to the repo being tested. In addition the following paths are available:

• ENTRYDIR - full host path to the repo directory. Use this path when mapping volumes into a docker container in your testscript.
• /repo - the mapped container path to the repo directory.
• SIM - path to simulation (relative from ENTRYDIR).

Docker

Docker is included in the runtime and setup to access the host docker system.

Hint: When mapping a volume into a container use the ENTRYDIR full host path rather than /repo as the docker system will consider mount paths from the host perspective (rather than the perspective of the running container).

Taskfile

Task is included in the runtime, along with a set of CLI tools (i.e. curl) that a task might use. Typically a task would run a containerised workflow where additional tools are installed.

Testscript

Testscript is included in the runtime and is also set as the container ENTRYPOINT.

Testing Techniques

E2E / Smoke Tests

Testscript can be used to write a simple E2E / Smoke test using the Simer simulation runner. The following minimal example shows how easily a test can be written:

env NAME=minimal_inst
env SIM=dse/modelc/build/_out/examples/minimal
exec sh -e $WORK/test.sh
stdout 'SignalValue: 2628574755 = 4.000000 \[name=counter\]'
-- test.sh --
SIMER="${SIMER:-ghcr.io/boschglobal/dse-simer:latest}"
docker run --name simer -i --rm -v $ENTRYDIR/$SIM:/sim \
    $SIMER -valgrind $NAME -env $NAME:SIMBUS_LOGLEVEL=2

References / Links

• Simer
• Testscript
• txtar
• Taskfile