MBSE: Aligning Systems Engineering with Verification and Validation

In model-based systems engineering, we tend to focus on requirements and design, but how many MBSE implementations consider aligning design with verification and validation (V&V)? The Vitech systems metamodel is especially capable of supporting V&V activities throughout the development lifecycle on either side of the “V.” Let’s take a look.

The typical “V,” illustrated in the following figure, depicts verification planning activities on the left side of the V, and verification execution activities on the right side of the “V.” In reality, verification and validation activities are also conducted on the left side of the “V” in conjunction with system development activities. Left side verification and validation typically consists of verification by analysis using simulation to validate the evolving system design, while right side V&V consists of the traditional unit, integration, system, and operational testing of the system as it is integrated and field tested.

Figure 1. The Engineering Model

If V&V is in reality aligned with system development, what metamodel can be defined to guide these activities? The world of verification and validation tends to vary more from organization to organization than typical systems engineering. Typically, systems engineers speak about requirements, behavior, and system design. V&V, on the other hand, is implemented in many different ways depending on your engineering domain, organization, or field. Despite this variability, the Vitech systems metamodel provides a means to formalize V&V methodologies and connect V&V with systems engineering. An excerpt of the Vitech systems metamodel, included in the following figure, depicts four unique classes specific to V&V, and describes the relationships among these classes, and with other classes in the systems metamodel.

Figure 2. The Vitech Systems Metamodel for V&V

The four classes pertaining to V&V align with the basic systems engineering concepts of requirements, behavior, and design. They are: 1) the Verification Requirement, 2) the Test Activity, and 3) the Test Configuration. Additionally, there is a class called Verification Event that relates each of the former classes together for a specific testing event, and a class called Test Procedure that can be substituted for Test Activities. Given this variation of V&V, let me describe how each of these V&V classes are defined and intended for use by the systems metamodel, and how they relate to other systems engineering concepts.

Verification Requirements
The notion of a verification requirement can be a new concept for many systems engineers, but you’ll recognize it with a bit more explanation. Referencing Figure 2, the verification requirement ties to the requirement class (or the function class) with a “verifies” relationship. This means each system requirement should have at least one verification requirement describing how that system requirement should be tested.

This may appear redundant at first, but the verification requirement is fundamentally different. A verification requirement contains no “shall” statements. It is not prescriptive, but instead is descriptive.  A verification requirement is designed to address five aspects of verification and validation:

• Objective. The verification requirement states the objective of the verification. The objective may represent some subset of the overall system requirement.
• Testing Method. Systems engineers are familiar with the testing methods of verification by analysis, demonstration, test or inspection. The verification requirement provides an opportunity to align a method with a specific verification objective.
• Environmental Conditions. This is a description of the environmental conditions necessary to sufficiently verify a system requirement.
• Special Conditions. This is a description of other testing conditions, equipment, or test setups that may need to occur to sufficiently measure testing to validate the system requirement.
• Success Criteria. Of special importance is a description of the success criteria. The success criteria provide test engineers another chance to make requirements testable in a feasible manner.

As an example of a verification requirement, consider:

The objective of this verification is to verify vehicle acceleration. Compliance of vehicle acceleration with system requirements will be verified by test. Verification will occur on a flat driving range over dry pavement. Verification will utilize timing systems and cameras to record the test. Five test runs will be conducted. Verification will be successful if the vehicle accelerates to 60 mph in 7.2 seconds or less in 4 of the 5 test runs and completes a quarter mile in 13.5 seconds or less in 4 of the 5 test runs.

There may be many verification requirements for each system requirement, with each verification requirement testing some aspect of the requirement with different methods or assumptions. Additionally, there may be many requirements for each verification requirement, since a particular verification requirement may describe testing that addresses more than one system requirement. So in the end, you may find a many-to-many relationship between verification requirements and system requirements.

Finally, while your organization may not use verification requirements directly, you will find that you most likely identify objectives, methods, conditions and success criteria for testing somewhere in your engineering documentation. Whether they are found in the form of a test plan, a test procedure, a test protocol, or a verification matrix, the ideas are equivalent.

Test Activity/Test Procedure
The test activity class describes step-by-step processes necessary for conducting a test, and can be related directly to a system requirement with the “based on” relationship. Test activities are modeled in the Vitech systems metamodel as a behavioral class, meaning they can be modeled and simulated just like functions as shown in the following figure. When a test activity is broken down in steps, the durations of each step and the required resources needed for each step can be simulated to provide greater confidence in the scheduling of test activities or of associated resources (people, equipment, facilities).

Figure 3. Behavioral Model for the Request Media Test

Test activities can also be modeled hierarchically to show the overall test program from a testing perspective.
The test procedure class may be used in lieu of test activities for organizations that have legacy testing material defined in documentation. A test procedure entity would be defined for each procedure document.

Figure 4. Hierarchy of Test Activities

Test Configuration
A test configuration refers to the test equipment and setup necessary to conduct testing across one or more verification requirements. The component class is used here to define specific tools, equipment or systems necessary for testing. Entities that make up a test configuration may be a system development effort unto themselves and managed elsewhere or in other models. For the purposes of test, they can simply be referenced as part of some configuration as depicted in the following hierarchy.

Figure 5. Makeup of the Geospatial Library Testing Test Configuration

Verification Event
The verification event is that class that ties all the above concepts together for a particular testing event. Verification events span multiple verification requirements, test activities, and test configurations. Verification events on the right side of the V usually show up on high-level programmatic schedules or work breakdown structures, but verification events can be defined on the left side of the V to represent systems engineering verification and validation of the emerging system design.

Examples of right side verification events may include ship testing, EMI testing, certification testing, qualification testing, field testing, site testing, or operational testing. Examples of verification events on the left side of the V may include items like: high-level design performance verification, or allocated design performance verification.

Since the verification event ties all the verification classes together, there are relationships to each class as depicted in the following spider diagram.

Figure 6. Spider Diagram for the Multiple Collector Verification Event

The V&V portion of the Vitech systems metamodel provides a starting point for your organization to tie V&V activities to your systems engineering design. With this short summary, I hope this helps you better maximize use of the V&V classes in your system development efforts.