Evaluation

Scenario workshop with project partners

As a prerequisite for a case study-based evaluation of CSI, we plan to conduct a workshop with our project partners to elicit evalua-tion scenarios. These scenarios should cover relevant functional criteria, targeting in particu-lar the functionality of the multi-perspective situation awareness and the co-evolution subsystem, as well as non-functional criteria focusing on scalability and performance issues.

Designing appropriate test cases

Obtaining real-world data for a case study from the productive environment of our demonstrators is of prime importance already during the early phases of CSI. Realistic test cases in terms of hotspots based on these test data will com-plement the scenarios from our initial workshop. Emphasis will also be put on the definition of sophisticated test cases being highly interrelated in terms of collaboration between au-thorities to thoroughly test the functional and non-functional capabilities of CSI under peak load times. Summarizing, the assumption to be verified is that, after a warming-up phase of adjusting CSI, consisting of several runs on basis of real-world data, the maintenance effort put in by the domain expert results in a positive return on investment in the form of more timely and accurate assessment and projection of hotspots. In that sense, our case study will aim on proving the underlying technical architecture's capability of enhancing distributed control systems by collaborative situation awareness.

Scenario-based real life tests with operators of demonstrators

Based on the scenarios agreed upon in the initial workshop, we plan to conduct real life tests together with operators from our demonstrators. For this, it is planned to install CSI prototypes at the control centers of our demonstrators, and operate CSI next to their current control system. Operators are then able to compare the information from their current system, with the situation awareness support of CSI. From this evaluation we aim to get feedback about the actual acceptance among operators. During peak load times, however, operators may naturally tend at the be-ginning of the evaluation to stick to their familiar system in order to avoid simultaneous han-dling of two systems. The evaluation hypothesis is that operators will – in the course of system usage – appreciate the aggregated hotspot information of CSI especially during pe-riods of increased workload and will more and more rely on CSI. The planned evaluation will be agreed upon by the industrial partners and demonstrators and based on experiences with previously employed test scenarios. Furthermore, it will be designed with an educational as-pect in mind, so that future manuals for a commercial product can be built upon the findings of the evaluation scenarios.