The Australian Senate Inquiry into Defence Procurement found that over the decade from 2000 to 2010 that some ten percent of projects by value ($7.6 billion) had failed to deliver the capability sought and that inadequate test and evaluation was a significant factor in the risks not being determined early enough for them to be mitigated. This presentation describes the new T&E policy, gives examples of how and when it has been applied.
Recommendations from the Defence First Principles Review are wide-ranging but will in most instances be judged by the success of reform of capability development. This paper proposes two key measures on which to judge the early success of Defence’s capability development reforms. First, the author proposes the formation of an industry-standard program management office (PMO) to oversee the life cycle of all acquisition projects from inception to final operational capability as part of comprehensive and balanced programs. Second, the author proposes the creation of a robust centralised branch to manage all test and evaluation (T&E), so that all projects have credible test results that underpin the PMO’s decision-making throughout the development and fielding of new capabilities.
This paper looks at why we need better quantitative test & evaluation techniques for project governance, the six-sigma test techniques used by U.S. DoD & major industries, and the building competencies in test design & analysis using the techniques.
The Complex System Governance (CSG) model foundationally fuses these disciplines in new governance meta-functions. Further the associated pathological approach developed for CSG implementation provides a path of least resistance and thus greatest parsimony for such change management. In this presentation we overview research illustrating the CSG underpinnings with the concepts of parsimony, requisite saliency, minimum critical specification, power law, and requisite hierarchy; thereby providing more elegant organizational approaches to complex program management and systems engineering than re-integration overlays.
Over the last two decades software functionality has become dominant, systems are more interconnected, globally sourced, intelligent and autonomous, and cyber-threats have become advanced, persistent and military. Yet, our Australian military acquisition models’ policy descriptions and standard contract suites remain rooted in a waterfall model of overly evolved requirements and a Single Step to Full Capability (SSFC).
Operational testing forever comes last in the dogma of the linear life cycle process, and with that goes any chance of early usability testing, serious simulation, feedback, real validation and flexible requirements until the budget and schedule are exhausted.
The second ‘V’ in V&V is almost invisible and the US version of VV&A has never crossed the Pacific because there is insufficient front-end simulation. The wherewithal to focus innovatively on Evolutionary Acquisition with Spiral Development (EA/SD) methods has been left to individual staff to explore. Worse still, systems engineers, project controls staff and testers have worn implied blame for the long acquisition times when they are not appropriately retrained and enabled because of an outdated acquisition model and outdated acquisition competencies, processes and test infrastructure.
In the last decade in the US, they have had serious infrastructure, process and competency reform of their acquisition. Connecting all Services and Groups they have large and distributed digital network integration centres with live, virtual and constructive simulation and cyber-ranging to do integration, usability testing, and cybersecurity testing, early and often.
MBSE and agile methods are now exploiting digital engineering for more innovation. Four significant organisations underpin the US successes in reforming their acquisition. Just as we import over 50 per cent of our Defence capabilities from the US DoD we need to import widely from these areas to lift Australia’s ability to catch up in digital and agile acquisition.
MBSE is revolutionising acquisition to enable broader digital engineering, agile acquisition and more evergreen resilience: as shown in this news article and short promotional video.
Northrop Touts Value of Digital Engineering as It Announces First Flight of Model 437
https://www.qinetiq.com/en/what-we-do/test-and-evaluation/test-and-evaluation-rich-picture
The reform is more than an industry zeitgeist, the trial of a new Master subject on MBSE by UNSW and Old Dominion University (ODU) in 2024 found over 60 credible research articles and two 2023 major textbooks rigorously chosen to cover the MBSE processes, impacts and tradeoffs. The learning outcomes were:
This masterclass will provide a short lesson and discussion forum for each of eight major theory weeks of the 13 weeks students studied the ZEIT8200 (ENMA 795) subject, including brief examples of student works.
1. Introduction to MBSE
2. Traceability, Requirements and Mission Engineering
3. Modelling with Capella: Requirements Analysis
4. Resilience and Cybersecurity Systems Modelling with MBSE
5. Supply Chain Modelling and Analysis with MBSE
6. MBSE Modelling and Common Modelling Pitfalls
7. V&V and T&E of, and by, MBSE: Model Management and Assessment
8. MBSE for System of Systems
The masterclass will raise awareness for those with little or no background in MBSE about the transitional issues without the deep reading, model adaptation and planning required in the Master subject for educational competence. The subject and thus this short masterclass uses an Object-Oriented Systems Engineering Methodology (OOSEM), as a top-down methodology, to create the required artifacts for analyzing stakeholder needs, defining system requirements, defining logical architectures, synthesizing candidate allocated architectures, optimizing and evaluating alternatives, and validating and verifying the system.
