.. # SPDX-FileCopyrightText: Copyright 2024-2025 Arm Limited and/or its # affiliates # # SPDX-License-Identifier: MIT .. _rd-aspen_validation: ########## Validation ########## ************************** Run-time integration tests ************************** The run-time integration tests are a mechanism for validating the Arm Auto Solutions' core functionalities. The tests are run on the image using the OEQA test framework. Refer to `OEQA FVP`_ for more information on this framework. In this section, details on the structure, implementation and debugging of the tests is given. OEQA tests in the BSP ===================== The Processing Elements and Components tested by the framework are detailed below. The testing scripts can be found in :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases` and :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/`. * ``test_00_aspen_boot`` * ``test_safety_island_c0`` This validates that the CMN has been configured, the handshake from the RSE has been received and that the SCP-firmware module initialization has completed successfully. * ``test_uboot_boot`` This method monitors the console output for the expected U-Boot message within a defined timeout period, ensuring the uboot bootloader has successfully initialized. * ``test_00_rse`` * ``test_normal_boot`` This validates that the SI CL0 is released out of reset and the handshake from the SCP-firmware has been received for CSS-Aspen. * ``test_measured_boot`` This validates enhanced trustworthiness provided by measured boot functionality by reading the slot and sw_type from the boot logs. * Primary Compute * FVP devices The entry point to these tests is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_devices.py`. To find out more about the applicable tests, see :ref:`rd-aspen_design_fvp_device_tests`. * FVP boot The script that implements the test is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_boot.py`. The test waits for Linux to boot on the Primary Compute then checks for common error patterns on all consoles. * ``test_20_aspen_ap_dsu`` * ``test_dsu_cluster`` This validates that the AP's DSU-120AE has been configured correctly by checking the L3 cache size, shared CPU list and the DSU-120AE PMU counters. * ``test_01_systemd_boot`` * ``test_systemd_boot_message`` This test ensures that the RD-Aspen platform is using the UEFI boot manager, systemd-boot. It verifies that the boot message contains the string 'Boot in' to confirm systemd-boot is being used. * ``test_30_configurable_pc_cores`` * ``test_configured_pc_cpus_in_tf_a`` This validates that the TF-A correctly brings up the configured number of Primary Compute CPUs. * ``test_configured_pc_cpus_in_linux`` This validates that the configured number of Primary Compute CPUs is visible in Linux by checking the number of CPUs listed in the device tree and the number of CPUs started at runtime using the ``nproc`` command. * ``test_00_secure_partition`` * ``test_optee_normal`` The test waits for the Primary Compute to log that OP-TEE loads the required Secure Partitions (SPs) and primary CPU switches to Normal world boot. .. _rd-aspen_design_fvp_device_tests: FVP device tests ================ These tests consist of a series of device tests that can be found in :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/fvp_devices.py`. * ``networking`` Checks that the network device and its correct driver are available and accessible via the filesystem and that outbound connections work (invoking ``wget``). * ``RTC`` Checks that the Real-Time Clock (RTC) device and its correct driver are available and accessible via the filesystem and verifies that the ``hwclock`` command runs successfully. * ``cpu_hotplug`` Checks for CPU availability and that basic functionality works, like enabling and stopping CPUs and preventing all of them from being disabled at the same time. * ``virtiorng`` Check that the virtio-rng device is available through the filesystem and that it is able to generate random numbers when required. * ``watchdog`` Checks that the watchdog device and its correct driver are available and accessible via the filesystem. PSA APIs test suite integration on Primary Compute ================================================== The meta-arm Yocto layer provides Trusted Service OEQA tests which you can use for automated :link_subs:`rd-aspen:trusted-services-test-executables`. The script that implements the test is :meta-arm-repo:`meta-arm/lib/oeqa/runtime/cases/trusted_services.py`. Currently, the following test cases for `psa-api-test` (from the :link_subs:`rd-aspen:psa-arch-tests-repo` project) are supported: * ts-psa-crypto-api-test Used for PSA Crypto API conformance testing for :link_subs:`rd-aspen:psa-crypto-api-doc`. * ts-psa-ps-api-test Used for PSA Protected Storage API conformance testing for :link_subs:`rd-aspen:psa-secure-storage-api-doc`. * ts-psa-its-api-test Used for PSA Internal Trusted Storage API conformance testing for :link_subs:`rd-aspen:psa-secure-storage-api-doc`. * ts-psa-iat-api-test Used for PSA Initial Attestation API conformance testing for :link_subs:`rd-aspen:psa-attestation-api-doc`. Platform Fault Detection Interface (PFDI) Test ============================================== The Platform Fault Detection Interface (PFDI) test is designed to validate the correct functioning of the PFDI integration. It does this by verifying the systemd service status of `pfdi-app`, the execution of the PFDI application, and the validation of the PFDI command-line interface (CLI). The script that implements the test is :repo:`yocto/meta-arm-auto-solutions/lib/oeqa/runtime/cases/test_10_pfdi.py`. The following tests are executed to validate PFDI * ``test_init_systemd_service`` The `test_init_systemd_service` method verifies that the `pfdi-app` systemd service starts correctly on boot. It uses `journalctl` to inspect the logs, ensuring the presence of expected service initialization messages and confirming the absence of error patterns in the log output. * ``test_pfdi_app`` The `test_pfdi_app` method validates the end-to-end execution of PFDI tool commands. It uses `pfdi-tool` to generate and pack diagnostic configuration files, then runs those diagnostics using the `pfdi-sample-app`. The test checks that diagnostics execute successfully across all CPU cores configured in the system. * ``test_pfdi_cli`` The `test_pfdi_cli` method checks the CLI interface by running commands such as `--info`, `--pfdi_info`, and `--count`. It validates that version information is correctly reported and that each core passes the Out of Reset (OoR) diagnostic check using the `--result` command. * ``test_pfdi_cli_force_error`` The `test_pfdi_cli_force_error` method injects a simulated fault on CPU2 using the `pfdi-cli -e` command. It then checks the systemd journal to verify that the failure was captured correctly, with log entries indicating that the Online (OnL) test failed for a CPU and reporting the appropriate input/output error code. Safety Diagnostics tests ======================== These tests consist of safety island tests that can be found in :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases/ test_10_safetydiagnostics_ssu_fmu.py`. * ``test_10_safetydiagnostics_ssu_fmu`` * ``test_safety_island_fmu`` This validates that FMU collects all faults from upstream fault sources and collates them to a single pair of non-critical(NC) and critical(C) error signals. * ``test_safety_island_ssu`` This validates that SSU has mechanism to validate critical or non-critical state transition with SSU SYS_CTRL and SYS_STATUS registers. .. _rd-aspen_design_pc_cpus_ras_tests: Primary Compute CPUs RAS tests ============================== These tests consist of :ref:`rd-aspen_ras_error_processing` tests that can be found in :repo:`yocto/meta-arm-bsp-extras/lib/oeqa/runtime/cases/test_00_tftf.py`. The validation for RAS are :link_subs:`rd-aspen:tf-a-tests-doc` based, and a special build configuration is used where :link_subs:`rd-aspen:u-boot` is replaced with the :link_subs:`rd-aspen:tf-a-tests-doc`. The following test is executed as part of the validation. * ``TftfTest`` The `TftfTest` verifies that each RAS error is processed correctly by the firmware. The test injects a RAS error and then awaits the error to be cleared successfully.