..
 # SPDX-FileCopyrightText: <text>Copyright 2025 Arm Limited and/or its
 # affiliates <open-source-office@arm.com></text>
 #
 # SPDX-License-Identifier: MIT


##########
User Guide
##########

Our Test Automation uses `pytest <https://docs.pytest.org/en/stable/>`_ for
extensive plugin support meeting industry level standard and configurable YAML
based system for platform setup.

Features
********

* Telnet-based multi-terminal session handling for Arm FVP integration
  (extendable to FPGA and SoC environments).
* Pytest-based test execution with modular plugin extensions.
* Log-based result tracking and verification, including ANSI/log filtering
  utilities.
* Reusable command execution utilities with prompt detection and automatic
  login handling.
* Platform-based configuration system using properties and YAML templates.

Getting Started
***************

This section provides a detailed, step-by-step guide for setting up an
automated test environment for Arm Automotive Software Reference Stack.

Each run creates a parent folder inside the central ``logs`` directory.
The folder name follows this format: ``logs/<target>_<platform>_<timestamp>/``

Inside that folder, normalized log files are generated, such as:

- **Boot log** - e.g. ``<target>_<platform>_boot.log``
- **Telnet console logs** - e.g. ``telnet_5001.log`` (one per console/port)
- **Command outputs** - e.g. ``cmd.log`` for captured command results

This consistent layout provides a clear way to inspect results, share logs,
and debug issues.

.. _prerequisites:

Prerequisites
*************

To run tests locally on FVP, make sure to have:

1. Python environment

   Ensure Python **3.10 or higher** is installed.

2. FVP binary

   The required Fast Models (FVP) executable must be installed on the host machine.
   When running pytest, the path to the binary must be passed explicitly through:
   ``--fvp-binary /opt/arm/FVP/models/Linux64_GCC-9.3/FVP``

3. Crypto Library

   For targeting the **RD-Aspen FVP**, ``Crypto.so`` plugin is needed.
   The test will try to auto-detect this file under the FVP installation.

4. Build Images

   Tests require prebuilt images from the software reference stack build.
   The directory path provided to ``--build-dir`` must contain the required image files
   (such as ``rse-rom-image.img``, ``ap-flash-image.img``, and ``.wic``) needed
   to boot the platform.

   These images may come from a local build or downloaded from any source.
   The images must be present under the directory passed to ``--build-dir``.

   .. code-block:: bash

      $ pytest -s tests/test_sample.py \
        --config ./my_platform_config.yaml \
        --build-dir ~/my-reference-stack/build/tmp/deploy/images/rd-aspen \
        --fvp-binary /opt/arm/FVP \
        --platform fvp_rd_aspen

.. _installation:

Installation
************

1. Create Python environment
   Create and activate a virtual environment, then install dependencies in editable mode:

   .. code-block:: bash

      $ python3 -m venv venv
      $ source venv/bin/activate
      $ pip install -e .

Configuration
*************

Follow all the steps in :ref:`Prerequisites` and :ref:`Installation` before running tests.

.. note::

   Before testing, ensure the ``.wic`` image name in the default YAML
   configuration matches the correct build architecture.

   For baremetal builds, update this argument accordingly in the YAML file:

   .. code-block:: bash

      - "-C ros.virtio_block0.image_path=${BUILD_DIR}/\
         baremetal-image-fvp-rd-aspen.wic"

   This configuration has not been validated for virtualization-based builds.
   Users attempting to run should proceed at their own discretion and may need
   additional configuration updates.

   To run tests for other platform variants such as ``RD-Aspen Cfg2``, some
   additional parameters may need to be updated or modified in the YAML
   configuration. For example, enabling an additional UART cluster terminal and
   adjust related entries in ``required_terminals``, ``prompts``, and
   ``port_map`` to include ``terminal_uart_si_cluster1``.

   Other variant-specific changes (like additional telnet ports or prompt
   patterns) should be updated as needed before running tests. For instance:

   .. code-block:: bash

      --parameter css.smb.si.terminal_uart_si_cluster1.start_telnet=0

   Update or extend arguments like the above based on the specific platform
   variant being tested, such as RD-Aspen Cfg1 or Cfg2.

Running Sample Test
*******************

1. Default config:

   .. code-block:: bash

      $ pytest -s tests/test_sample.py \
        --config test_automation/configs/standalone_config.yaml \
        --build-dir /tmp/images/ \
        --fvp-binary /tmp/FVP \
        --platform fvp_rd_aspen

 A successful sample test run should produce output similar to the following:

   .. code-block:: none

      ------------------------------------------------------- live log call -------------------------------------------------------
      INFO    tests.test_samp: Running test on Platform: fvp_rd_aspen
      ...
      INFO    tests.test_samp: Command Executed ('touch demo.txt; ls')

      Command output :
      demo.txt
      PASSED
      ...
      ----------------------------------------------------- live log teardown -----------------------------------------------------
      INFO     tests.conftest: Powering off platform
      ...
      =============================================== 1 passed in 64.02s (0:01:04) ===============================================

 The sample test typically completes within 1-2 minutes on a standard host
 system (but may vary depending on host system performance).

2. Custom config:

   .. code-block:: bash

      $ pytest -s tests/test_sample.py \
        --config ./my_platform_config.yaml \
        --build-dir /tmp/images/ \
        --fvp-binary /tmp/FVP \
        --platform fvp_rd_aspen

.. note::

   - The ``--platform`` argument must exactly match the platform value in the config file (e.g., ``fvp_rd_aspen``).
   - The value should contain "rd_aspen" if tests are for RD-Aspen (e.g., ``fvp_rd_aspen``).
   - Default logging level is INFO. To enable DEBUG logs, add ``--debug-logs``.
   - For more detailed output during debugging, extra pytest arguments can be used:
     ``-rs --setup-show -vv``.
   - Environment variables can be used:

     - ``FVP_BINARY`` instead of ``--fvp-binary``
     - ``FVP_CRYPTO_PATH`` for location of ``Crypto.so``

Add New Test Cases
******************

Test files should be placed under the ``tests/`` directory and follow the naming convention ``test_*.py``.

Test Structure
--------------

A typical test file should contain:

1. A ``pytest`` test function
2. Usage of built-in fixtures such as ``platform_base_obj``, ``platform_name``, and ``telnet_mgr``
3. Prompt-based command execution and output validation

Example:

.. code-block:: python

   class TestMyExample:
      def test_my_example(self, platform_base_obj, platform_name):
         code, output = platform_base_obj.mgr.execute_command_with_prompt_capture(
            port=platform_base_obj.default_console,
            command="echo hello",
            timeout=10
         )
         assert code == 0
         assert "hello" in output

.. note::

   - Always use ``session_manager.wait_for_prompt_in_log()`` or ``session_manager.execute_command_with_prompt_capture()``.
   - Match terminal names as in the YAML config.
   - ``conftest.py`` provides global fixtures — no need to redeclare managers in each test.