Last time, in Part I, we discussed what emergent behavior is, where it ultimately originates from, and why unintended emergence is hard to mitigate when not caught early. We also introduced the Cha-Cha-Cha investigative approach utilized primarily in the scientific community and tailored it for use in systems engineering and systems thinking applications.
Today, we will look at systems from three viewpoints: internal, external-reflective, and external system of systems. We will discuss some possible scenarios where unintended emergent behavior may be observed. We will then look at how the Cha-Cha-Cha approach can be applied to not only observe that something unintended is occurring, but also to discuss possible originating causes and ways to mitigate the deviations to planned behavior. These are intended to demonstrate an initial application of the approach. The reader is encouraged to further evolve and elaborate on these examples.
Internal
We begin by focusing on the internal structure and behaviors of the system. The external interactions are downplayed in this viewpoint. The following figure shows a generic block diagram of a system, its internal components and interrelationships.
The observations captured below represent a possible subset of observations that could occur during the development of a representative system. Most of the observations resulting in unexpected behavior have their genesis in not properly understanding or documenting internal interfaces to the detail necessary to explain the observed emergent behavior.
| Observation | Inquisition method | Possible Issues / Root Causes | Possible Handling Approaches |
| Elements interact in different ways in different analytical models: a design engineer’s detailed model produces behaviors not observed in the systems engineer’s system model. | Charge – A constant difference in results appears under multiple design cases.
Challenge – Differences in results are noticed discretely over time. Patterns begin to emerge related to input conditions, corner cases, environmental conditions, or other constraints. Chance – Serendipitous behaviors with some benefit are observed that were unexpected. |
The System Model may be too abstract. It may omit interfaces captured in the detailed models that provide important interfacing behavior at the system level. | Increase the fidelity of the system level model to capture the needed additional interfaces. This task is complete when the resultant behavior of the system model and detailed models correlate sufficiently or the differences can be explained via known simplifying assumptions under multiple relevant conditions.
Implement a failure modes and effects analysis process throughout the project lifecycle. Identify through brainstorming faults and effects, design in detection methods. Develop mitigations. Design out failure modes when appropriate. Design resilient systems. Design for fault states, means to detect, probe, investigate off-nominal and unanticipated behavior. |
| Elements interact in unintended ways during lower level testing and don’t correlate to analysis. | Charge – The same behavior is seen repeatedly during testing.
Challenge – Different unexpected responses are observed at various test conditions. Chance – Serendipitous behaviors with some benefit are observed that were unexpected. |
Detailed physics and interactions are not captured in the analytical models | Test early and often. Implement a “build-test-fix” approach with subscale models under realistic boundary conditions as soon as possible to identify and correlate system-environment-user interrelated effects.
Adopt an agile systems engineering methodology for software and analysis development. Update failure modes and effects (FMEA) analysis. Update mitigation procedures. |
External – Self Reflective
From this perspective, we generally view the system as a black box and focus our attention to the interfaces with external entities. These can include other enabling systems, users, adversaries and the external environment.
Again, most of the emergent behaviors that are observed from this perspective originate from missing and not capturing interfaces between these external entities and the system of interest. In particular, users may exercise the system in unintended ways, sending it commands and requests that were unexpected; adversaries may try to disrupt the system in ways the designers never imagined; and the environment may be more complex than anticipated, causing interference or constraints not imagined.
| Observation | Inquisition method | Possible Issues / Root Causes | Possible Handling Approaches |
| System behaves in unintended ways with known users. | Charge – the same behaviors are seen repeatedly during standard user testing
Challenge – Different unexpected responses are observed at various user test conditions. Chance – Serendipitous behaviors with some benefit are observed that were unexpected. |
The system analytical models and test results may not have captured specific user use cases (validation issue) or interface means. | Re-interview / interrogate users to update Use Cases (after system integration). This will possibly require redesign.
Involve users early and often to validate use cases and requirements prior to final build and acceptance. Design resilient systems. Design for fault states, means to detect, probe, investigate off-nominal and unanticipated behavior. |
| System behaves in unexpected ways without user stimulus. | Charge – The same unexpected behavior is seen repeatedly during operational fielding under quiescent or ready state conditions.
Challenge – Different unexpected responses are observed at various times when the system is in quiescent state – or- different patterns are observed over time that vary under controlled user conditions. Chance – Serendipitous behaviors with some benefit are observed that were unexpected. |
Nefarious adversarial agents are interacting with the system in unanticipated ways
Environmental components and factors are interacting with the system that were not envisioned. |
Employ white hackers and integrate feedback early.
Expose prototype systems to realistic environments early (if an option). Design resilient systems. Design for fault states, means to detect, probe, investigate off-nominal and unanticipated behavior. Develop a Digital Twin that replicates the necessary physics of the system such that it represents the system’s behavior under specified external conditions. |
External – System of Systems Environment
From this perspective, the system must operate in a broader context with other engineered systems to accomplish a higher-level task or mission. Some of the challenges that typically arise with systems of systems are that some of the systems fielded were developed by other organizations and may incorporate legacy, outdated protocols and communications not under direct control or authority to upgrade or modernize by the overarching entity. This results in fixed and antiquated interfaces along with the application of systems to missions and operational environments for which the systems were not originally designed.
Most of the emergent behaviors that are observed from this perspective originate from interfaces not originally planned when the originating systems were designed and fielded as stand-alone systems. New users, new environments, and new adversaries introduced in a system of systems context create new interfaces. Fixed and outdated interfaces present a unique challenge in this environment. For this reason, all of the observations, possible root causes, and handling approaches from the previous section still apply. Vulnerabilities that exist in one system may open up new and unintended vulnerabilities in connected systems. The systems engineer must pay particular attention to legacy interfaces and those applied out of their original context when a system is placed into a system of systems.
Conclusion
None of these approaches are complete or foolproof, but they hopefully provide additional tools to put in your personal systems engineering toolbox. All of these approaches should be tailored and adapted to the particular needs and circumstances of a unique system of interest.
These investigative approaches allow the inquisitive systems engineer to perhaps gain insights and understanding that would otherwise take longer to uncover and notice. As we know, the later in the design lifecycle issues are caught, the costlier they are to rectify, if they even can be rectified. Charge, challenge, and chance viewpoints provide us opportunities to see these behaviors through different lenses. They allow us a means of probing, sensing, and responding to refine and evolve system views and models.
Creating representative system models of appropriate detail and fidelity, designing systems to be resilient and recoverable from faults, testing early and often, incorporating an agile development methodology, involving representative users and stakeholders early, and developing digital twins are all tools that can be used to either uncover or deal with unintended emergent behavior. Patterns may emerge that provide additional insight. What additional techniques have you used to uncover, manage, and handle emergent behavior?
