EDA
System
IP
SolvNetPlus
Training & Education
Series
This article is part of the series
"Cloud-Based Embedded Testing: A Case Study."
Discover more:
Introduction
By leveraging cloud infrastructure, the aim is to streamline testing processes, reduce execution times, and enhance scalability. But how is automated cloud-based testing for functional software implemented, especially when it comes to test execution and result evaluation
This study demonstrates the feasibility and advantages of automated, scalable cloud testing for embedded systems, offering insights into optimizing both process efficiency and result accuracy in software quality assurance.
Key elements of the cloud testing approach include the use of an appropriate testing framework, the generation of relevant test data, and reliable assessment metrics to ensure functionality meets predefined standards. The case study also includes practical implementation steps, utilizing Amazon Web Services (AWS) and Jenkins for continuous integration, which are tailored to the unique requirements of automotive embedded software testing.
Testing Setup
The focus is on automated testing of simple functional software with an emphasis on test execution and evaluation. The resulting report provides insights into which tests were successful and which ones failed.
To conduct these tests, three essential components are required:
- a suitable testing framework
- test data to stimulate the test subject
- assessments for evaluation of correct functionality
System under Test
The test subject is a simple Simulink model that calculates the control of the vehicle’s external lights based on a light switch and light intensity: Testing Lights Control with TPT.
We assessed this simple example as highly suitable for the use cases because it serves as an exemplary representation of automotive embedded software development. It has a manageable size, encompasses well-known functionality, and has been tested frequently.
Test Environment
The testing is conducted in both Model-in-the-Loop (MiL) and Software-in-the-Loop (SiL) environments. The results of both types of tests are automatically compared as part of a Back-2-Back test.
TPT for Test Automation
The automated tests were created and executed with TPT. TPT is a widely used solution in the automotive industry and enables testing of MATLAB/Simulink and TargetLink models. In the comparative testing between MiL and SiL, the same test cases are used without modification. Test execution in MiL is achieved through the integration of TPT and Matlab/Simulink, with TPT automatically establishing the connection.
Two Cases for Implementation
We set up two cases with a different focus each, yet both are build on each other.
The objective: A tester should be able to start a test execution on multiple, parallel computing units at the push of a button. When the test execution is complete, there is a report that summarizes all test executions, measurements, and results as if the execution had occurred on a single computer.
To be able to implement this, the computer must be able to:
- Connection from the computer to the cloud via the internet.
- Setting up instances.
- Uploading files from the computer to instances (such as models, test scripts, etc.).
- Initiating test execution with selected test cases per instance.
- Downloading files from instances to the computer (such as test reports, etc.).
- Shutting down instances.
Implementation Steps
2. Instantiation of the Test Execution Environment
An AMI is a software configuration and needs a computing unit to be able to run. It must be “instantiated” in a computing unit and it needs one computing unit per instance.
How to choose the appropriate computing unit?
That´s individual and primarily depends on the function and potential utilization by the AMI. There is a wide range of computing units to choose from, including options such as: number of processors, random access memory (RAM), instance storage (ROM), network bandwidth, and many other factors.
The cost model is based on the instance runtime, and it doesn’t matter whether the available resources are utilized or not. For our first case with many instances, it is therefore advisable not to overdo it.
Specifically, our choice fell on EC2 instances of Type T2.medium. A base AMI must be defined for the EC2 instance to be created. The setup, shutdown, and communication with the EC2 instances are done through SSM (AWS Systems Manager Session Manager): an interactive shell and command-line interface for EC2 instance management.
3. Configuration for establishing communication between local PC and AWS
In order to start testing, additional settings are needed to allow secure and seamless communication between the local computer and the cloud.
Communication is configured, among other things, through Identity and Access Management (IAM), Virtual Private Clouds (VPCs), and security groups for inbound and outbound traffic. To safeguard the data/IP and the wallet, several security mechanisms are implemented for communication between the local computer and the cloud. It’s important to understand these mechanisms and then configure them appropriately for the specific use case, taking into account all elements of the application.
In our Case, there are several communication relationships:
- Local computer registration in the cloud = Initial login for further communication
- Communication from the computer to the S3 storage unit for upload and download
- Communication from the computer to the SSM (e.g. starting three EC2 instances)
- Communication from the computer to the EC2 instance (e.g. starting tests)
- Communication from EC2 instance to the S3 storage unit (loading models, storing test reports)
Summary of the Case 1-Implementation
Use case 1 could be fully implemented. Setting up the AMI and the other cloud computing resources was easy and fast thanks to good documentation from AWS.
Security activities
The biggest effort was in the security activities. Often we first had to understand which ports we needed to enable in order to allow communication between two entities.
For example, for file uploads from the local machine to the instances and for downloading the reports from the instance to the machine.
Communication relations
In numerous places, we also had to allow communication relationships between the elements used in AWS and the user. Once we had these chains of action in place and understood, we were able to implement an automation script in Python. Thanks to the very intuitive and comprehensive boto3 framework, this was done incredibly quickly and smoothly.
Risk local scripts
However, scripts executed on a local computer to control the process can also pose risks: If the connection between the local computer and the cloud is lost, unnecessary instances may continue to run and incur unnecessary costs.
Additionally, a script needs to be initiated by a user. This can lead to delays, can be inconvenient at times, and may also be error-prone.
To minimize such risks associated with local scripts, we expanded our first use case: We set up an additional instance in the cloud to initiate and monitor the test execution process. This is described in the second use case.
Because Use Case 1 was completed very quickly and successfully, the question of further automation arose immediately. Why? It is less error-prone, more precise, faster, and more robust.
The first case was to be extended by instantiating a Jenkins server in the cloud. Similar to setting up EC2 instances, Jenkins was installed in an AMI . The major difference between this instance and the instances used for testing is that this instance will run continuously.
The reason for this continuous operation: This instance is meant to monitor the connected software repository (version control) GIT for changes in the code and ongoing tests at all times.
The Advantage of a Continuous Integration Instance
Effective Quality Assurance through Jenkins
If a team member makes a change to a part of the software, like the Matlab/Simulink model, this change should be immediately detected and verified. Typically, in software version control, there are multiple levels.
For quality assurance, the following principle has been established: Only changes that meet the quality standard are merged into the production path. In contrast, in a development path, the development hands over the change to the version management. This is detected by Jenkins.
Now, first, quality assurance measures like our software tests should demonstrate that the change is mature enough for the production path.
If it is mature enough, it will be incorporated. If not, the change will be discarded, and the development team will receive feedback indicating that the change is causing issues. This is usually communicated through an email from Jenkins.
Benefits of Using Jenkins
- Jenkins takes over the user’s job and can initiate tests.
- Jenkins continuously monitors the correct execution and shuts down the instances once they are finished.
- Operational processes can be defined arbitrarily and implemented easily.
- Jenkins is very easy to configure.
- Web-based user interface that allows convenient configuration and control from any computer with the appropriate access rights.
- Seamless integration with version control systems like GitLab and GitHub through so-called webhooks.
Additionally, Jenkins offers a wide range of useful plugins, including the EC2 plugin. With its help, EC2 instances are automatically launched, monitored, and pipeline jobs (tests) are executed.
Summary of the Case 2-Implementation
Use Case 2 was successfully implemented and represents an impressive integration of cloud services into our CI environment. The seamless integration was achieved effortlessly, thanks to the excellent documentation of the plugins used.
Security Activities
A significant portion of the work focused on security activities, such as opening inbound ports and setting up webhooks to fetch files from private repositories, and managing access using access tokens.
Communication Relations
Additionally, communication relationships between the AWS services and the Jenkins server had to be established to enable automatic scaling through the automatic creation and connection of new instances. Thanks to this innovative solution, we were able to successfully overcome the challenges of Use Case 1.
Benefits of the Autoscaling Solution
As soon as changes are made in the Git repository, the Jenkins server automatically initiates the creation of new EC2 instances and starts the execution of the pipeline jobs.
The Jenkins server autonomously monitors the created instances and terminates them immediately once the test cases are completed and the results have been uploaded.
Current Challenges
The current solution does not automatically detect when test cases are stuck and run longer than expected, failing to terminate them in a timely manner.
Proposed Solution
Internal monitoring mechanisms, known as ‘watchdogs’, can be implemented. These watchdogs can stop test cases if they unexpectedly run for an extended period to control execution time and prevent bottlenecks.
Advantages of Testing with TPT
TEST FRAMEWORK
TPT constructs a test framework tailored to the software.
TEST DATA
TPT creates test data to stimulate the test subject.
TEST EXECUTION
In automotive software development, tests are typically executed on a PC. For very large models or extensive tests, the execution can even take several days. This primarily arises from the sequential processing of test cases and the limited scalability on a PC.
TEST ASSESSMENT
Assessments for evaluating whether the test subject functions correctly.
For insights into cost structures, continue to Part 3: Cost Structures in Cloud Testing.
Continue Reading
3D Chips: Socionext Achieves Two Successful Tape-Outs in Just Seven Months
New Fujitsu Chip to Power Greener, Smarter Data Centers
