.. # SPDX-FileCopyrightText: Copyright 2023-2024 Arm Limited and/or its # affiliates # # SPDX-License-Identifier: MIT .. _validation: ########## Validation ########## .. _validation_run-time_integration_tests: ************************** Run-Time Integration Tests ************************** The run-time integration tests are a mechanism for validating the Arm Auto Solutions' core functionalities. The tests are run on the image using the oeqa test framework. Refer to `OEQA FVP`_ for more information on this framework. .. note:: There is a rare known failure where a timeout might occur during test execution. Refer to :ref:`releasenotes_knownissues` for possible workarounds. In this section, details on the structure, implementation and debugging of the tests is given. OEQA tests used by the BSP ========================== The Processing Elements and Components tested by the framework are detailed below. The testing scripts can be found in :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases` and :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/`. All of the Processing Elements and Components have their terminal output logged for debugging. * RSE The script that implements the test is :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases/test_00_rse.py`. The test firstly waits for the successful programming of the GIC-Multiview and the NI-710AE. Then the test waits for the RSE to log that it is releasing the SCP. This is its last action as part of the RSE boot process. * Primary Compute * FVP devices The entry point to these tests is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_devices.py`. To find out more about the applicable tests, refer to :ref:`design_fvp_device_tests`. * FVP boot The script that implements the test is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_boot.py`. The test waits for Linux to boot on the Primary Compute then checks for common error patterns on all consoles. * OP-TEE The script that implements the test is :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases/test_00_secure_partition.py`. The test waits for the Primary Compute to log that OP-TEE loads the required Secure Partitions (SPs) and primary CPU switches to Normal world boot. .. _design_fvp_device_tests: FVP Device Tests ================ These tests consist of a series of device tests that can be found in :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_devices.py`. * networking Checks that the network device and its correct driver are available and accessible via the filesystem and that outbound connections work (invoking ``wget``). * RTC Checks that the Real-Time Clock (RTC) device and its correct driver are available and accessible via the filesystem and verifies that the ``hwclock`` command runs successfully. * cpu_hotplug Checks for CPU availability and that basic functionality works, like enabling and stopping CPUs and preventing all of them from being disabled at the same time. * virtiorng Check that the virtio-rng device is available through the filesystem and that it is able to generate random numbers when required. * watchdog Checks that the watchdog device and its correct driver are available and accessible via the filesystem. Integration Tests Implementation ================================ This section gives a high-level description of how the integration testing logic is implemented. To enable the integration tests, the `testimage.bbclass`_ is used. This class supports running automated tests against images. The class handles loading the tests and starting the image. The `Writing New Tests`_ section of the Yocto Manual explains how to write new tests when using the testimage.bbclass. These are placed under :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases` and will be selected by the different machines/configurations by modifying the ``TEST_SUITES`` variable. For example, the file :meta-arm-repo:`meta-arm-bsp/conf/machine/fvp-rd-kronos.conf` adds the ``fvp_devices`` test to the ``TEST_SUITES`` variable. Before running integration tests, some basic tests will be run firstly: * test_linux_login The test logs into the Linux shell as root. If it fails, the tests that depend on it will be cancelled. The test is implemented in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_linuxlogin.py`. * test_cluster{N} The test verifies the output when Zephyr boots on Safety Island Cluster N. The N is the Safety Island clusters number. Those tests are implemented in: :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_safety_island_c0.py` :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_safety_island_c1.py` :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_safety_island_c2.py` After running all the integration tests, the following test is run: * test_linux_shutdown The test verifies that the FVP can be terminated using a ``shutdown now`` command in the Linux console. The test is implemented in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_99_linuxshutdown.py`. .. _validation_actuation_demo: Integration Tests Validating the Safety Island Actuation Demo ============================================================= The ``test_player_to_analyzer`` integration test in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_actuation.py` does a full Player to Packet Analyzer functionality test. This test invokes the Actuation Player that plays a recorded driving scenario which triggers the Actuation Service to generate Control Commands to be forwarded to the host via BSD socket. These Control Commands are then captured by the Packet Analyzer which validates them against a recorded Control Commands list that is stored in the form of a CSV file. .. _validation_cam_tests: Integration Tests Validating the Critical Application Monitoring Demo ===================================================================== The script that implements the tests is :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_cam.py`. The tests verify: * The pack command of ``cam-tool`` on the Primary Compute. * The stream data calibration on the Primary Compute. * The startup of ``cam-service`` on the Safety Island Cluster 1. * Application monitoring from the Safety Island Cluster 1. * Application monitoring with multiple connections. * Logical and temporal failure detection. The tests are performed from the baremetal Linux userspace when building the Baremetal Architecture and from both the DomU1 and DomU2 Linux userspaces when building the Virtualization Architecture. .. _validation_hipc_demo: Integration Tests Validating the Safety Island Communication Demo ================================================================= The scripts that implement the tests are :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_hipc.py` and :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_hipc_virtualization.py`. The tests below are run for each Safety Island cluster for Baremetal and Virtualization Architectures. For the Virtualization Architecture, tests are run for each Xen guests created. * test_ping_cluster The test pings the Safety Island from the Primary Compute and vice versa and checks that an answer is received. * test_hipc_cluster The test verifies Heterogeneous Inter Processor Communication (HIPC) between the Safety Island (using ``zperf``) and the Primary Compute (using ``iperf``). The tested configurations are: * The Safety Island as an iperf server (UDP/TCP) and the Primary Compute as a client (UDP/TCP). * The Safety Island as an iperf client (UDP/TCP) and the Primary Compute as a server (UDP/TCP). * test_hipc_cluster_cl{M}_cl{N} The test verifies Heterogeneous Inter Processor Communication (HIPC) between the Safety Island Clusters (using ``zperf``) where M and N are the clusters number. The tested configurations are: * The Safety Island Cluster {M} as a Zperf server (UDP/TCP) and the Safety Island Cluster {N} as a Zperf client (UDP/TCP). .. _validation_gptp: Integration Tests Validating gPTP ================================= The scripts that implement the tests are :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_ptp.py` and :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_ptp_virtualization.py`. * test_ptp_linux_services The test ensures the ``ptp4l`` services are running. * test_ptp_si_clients The test verifies that the gPTP services running on the Safety Island clusters are in the expected client state. It then introduces a fault in the system by bringing down the relevant network interfaces on the server side and checks that the state machines on the Safety Island clusters are not in a client state anymore. The network interfaces are brought back up and the test validates that the state machines get back to the expected state. * test_ptp_domu_client The test has the same steps as the Safety Island test above, but targeted at the DomUs instead of the Safety Island clusters. It is skipped when not using the Virtualization Architecture. .. _validation_zephyr_bridge: Integration Tests Validating the Safety Island Cluster 0 Bridge =============================================================== The ``test_si{N}_bridge_ethernet0`` integration tests in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_30_si0_bridge_ethernet0.py` verify the connection between the Host and the bridged Safety Island clusters. The tested configuration is: * The Safety Island as an iperf server (TCP) and the Host as a client (TCP). UDP is not tested because the user networking of the FVP does not provide port forwarding for UDP traffic. .. _validation_parsec_enabled_tls_demo: Integration Tests Validating the Parsec-enabled TLS Demo ======================================================== The ``test_parsec_demo`` integration test in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_parsec.py` verifies the functionality of the crypto service provided by Parsec and the RSE. The test is only enabled when the use case is ``Safety Island Actuation Demo`` and the "Baremetal Architecture" is selected. The test invokes a TLS server and client application. The client consumes the Parsec service for asymmetric crypto operations. Parsec is configured with Trusted Services as the backend which further invokes the crypto service from the hardware isolated RSE. The test is performed under the following configuration: * The TLS server operates on the Primary Compute. * The TLS client application operates within a container that resides on the Primary Compute. .. _validation_xen: Integration Tests Validating Xen ================================ The ``test_ptestrunner`` integration test in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_virtualization.py` uses ``ptest-runner`` to execute ``01-xendomains.bats`` Bash Automated Test System (BATS) tests in :repo:`yocto/meta-arm-auto-solutions/recipes-test/xen/files/tests/01-xendomains.bats`, DomU lifecycle management -------------------------- The ``01-xendomains.bats`` BATS test verifies DomU lifecycle management, including status checking, destroy and restart. MPAM ---- The MPAM cache partitioning functionality is verified by the ``01-xendomains.bats`` BATS tests from the following aspects: * Verify that the Dom0 Cache Portion Bitmap (CPBM) value is consistent with the pre-set value from the Xen command line. * Verify that the CPBM values for DomU1 and DomU2 are consistent with the pre-set value from the Xen guest configuration files. * Verify if user can modify the domain CPBM values by the ``xl`` sub-commands in :ref:`design_components_xen_mpam`. .. _validation_gicv4_1_demo: GICv4.1 vLPI/vSGI Direct Injection Demo --------------------------------------- The ``01-xendomains.bats`` BATS test verifies GICv4.1 feature enablement, through pre-set keyword capture in Xen and Dom0 Linux boot log. The ``test_gicv4_1`` integration test in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_gicv4_1.py` verifies the functionality of GICv4.1 vLPI/vSGI direct injection. The test is only enabled when the use case is ``Safety Island Actuation Demo`` and the "Virtualization Architecture" is selected. The test consists of the following aspects and will only be run on DomU1: * PCI AHCI SATA disk ``ahci[0000:00:1f.0]`` has already been assigned to DomU1 with static PCI passthrough method. * Using ``lspci`` command to check if PCI AHCI SATA disk is properly probed. * Inspecting ``/proc/interrupts`` and reading non-zero MSI-X interrupts from ``ahci[0000:00:00.0]`` captured at domain boot-time to validate the functionality of GICv4.1 vLPI direct injection. * Inspecting ``/proc/interrupts`` and reading non-zero IPI0 interrupts captured at domain boot-time to validate the functionality of GICv4.1 vSGI direct injection. The testing of interrupt injection is not currently validated for run time operations, e.g. file system actions or data transfer. .. _validation_trusted_services_tests: Integration Tests Validating Primary Compute PSA APIs Architecture Test Suite ============================================================================= The meta-arm Yocto layer provides Trusted Service OEQA tests which can be used for automated `Trusted Services Test Executables`_. The script that implements the test is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/trusted_services.py`. Currently, only the following test cases for `psa-api-test` (from the `PSA Arch Tests`_ project) are supported: * ts-psa-crypto-api-test Used for PSA API conformance testing for `PSA Crypto API`_. * ts-psa-ps-api-test Used for PSA Protected Storage API conformance testing for `PSA Secure Storage API`_. * ts-psa-its-api-test Used for PSA Internal Trusted Storage API conformance testing for `PSA Secure Storage API`_. .. _validation_si_psa_arch_tests: Integration Tests Validating Safety Island PSA APIs Architecture Test Suite =========================================================================== The ``test_psa_si_cluster{N}`` integration tests in :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_si_psa_arch_tests.py` verify that the psa-arch-tests suite report is as expected. This test waits until the ``psa-arch-tests`` finish successfully and there are no failures in the tests suite report. .. _validation_fault_management: Integration Tests Validating the Fault Management Subsystem =========================================================== The Fault Management test suite at :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_fault_mgmt.py` contains a test class to validate the FMUs and another to validate the SSUs using the Zephyr shell commands described in the :ref:`design_applications_fault_mgmt_shell_reference`. ``FaultMgmtTest`` validates the configuration, injection, reporting and storage of faults in the FMU device tree: * ``test_tree`` minimally validates the existence of the expected devices. * ``test_system_fmu_internal_inject`` validates the injection and reporting of internal faults of the System FMU. * ``test_system_fmu_internal_set_enabled`` validates disabling System FMU faults. * ``test_gic_fmu_inject`` validates the injection and reporting of GIC-720AE FMU faults from Primary Compute and Safety Island (critical and non-critical). * ``test_fmu_fault_count`` validates the reporting of the overall fault count. * ``test_fmu_fault_list`` validates the list of reported fault counts. * ``test_fmu_fault_summary`` validates the fault summarization. * ``test_fmu_fault_clear`` validates that injected faults can be cleared. ``FaultMgmtSSUTest`` validates all possible transitions in the SSU state machine using three test cases (with a full system reset between each one to transition from ``ERRC`` back to ``TEST``): * ``test_gic_fmu_ssu_compl_ok``, which triggers a non-critical fault, recovers then triggers a critical fault, for each upstream GIC FMU. * ``test_ssu_nce_ok``, in which the self-test fails with a non-critical fault which is then signaled as critical. * ``test_ssu_ce_not_ok``, in which the self-test fails with a critical fault. .. _validation_sve2: Integration Tests Validating SVE2 ================================= The script that implements the tests is :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_sve.py` * test_sve_enabled The test ensures the ``sve2`` feature is enabled. * test_sve_config This test verifies that the SVE2 configurations on the Primary Compute are valid in both virtualization and baremetal cases. .. _validation_secure_firmware_update: Integration Tests Validating Secure Firmware Update =================================================== The script that implements the tests is :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_00_fwu.py` * test_securefirmwareupdate The test logs into Linux and runs commands to copy the update capsule from the MMC card to the boot partition. It starts the ``Secure Firmware Update`` process by rebooting and ensures that the ``Secure Firmware Update`` was completed successfully by monitoring the RSE terminal output. Integration Tests Validating EWAOL ================================== The script that implements the tests is :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_40_ewaol.py` * test_ptestrunner This test runs the EWAOL/Cassini integration tests which are implemented using BATS (Bash Automated Test System). The following test suites are invoked: * container-engine-integration-tests * parsec-simple-e2e-tests For more information see the `Cassini Run-Time Integration Tests documentation`_.