About CrowdSA - Evaluation
The overall evaluation hypothesis is that both, authorities and citizens will benefit from the enriched situational picture provided by crowdSA by having a basis for more accurate decision making – especially during periods of increased information demand that cannot appropriately coped with conventional information sources. To provide evidence for this hypothesis, the evaluation of crowdSA will be based on three pillars, comprising a collaborative case-study definition phase together with our project partners and additional stakeholders, a post-crisis case study and a real-life data case study, involving a dedicated crowd, namely Red Cross volunteers and with visitors at the yearly Nova Rock festival, where the ARC is involved in the crisis management. The evaluation will be planned with an educational aspect in mind, so that best practices for crowdsourced crisis management can be built upon the findings of the evaluation.
Collaborative Case Study Definition
As a prerequisite for a case study-based evaluation of crowdSA, we plan to conduct a workshop with our project partners to elaborate a catalogue of critical situation types and crisis scenarios, together with appropriate test cases. Input for this task may be derived from crisis and emergency literature (e.g., [Reut12] or [View10]) and gained through discussions with involved stakeholders. The designed situation types and crisis scenarios should exhibit different degrees of complexity, depending on the number of crisis objects involved. The same holds true for the test cases, which should cover data reflecting different perspectives on a crisis situation and exhibiting both, corroborative as well as contradictory information, distinguishing different stages in between, e.g., partly overlapping information. Thereby, the focus is not only on functional evaluation criteria, i.e., dedicated test cases for each functional component of crowdSA, but also on non-functional ones, primarily considering scalability. Concerning the domain targeted by the evaluation, we will focus on natural disasters such as floods not least since these are highly relevant for the Austrian region. Another emphasize will be put on the impacts on traffic, since being the main application area of our previous projects BeAware! and CSI.
Post-Crisis Case Study
Based on collected and publicly available crowd data from social media about previous crisis (cf. e.g.,[Came12]), forensic analysis will be performed with crowdSA. The case study comprises a simulated situational picture about the crisis based on artificially generated structured data (inspired by real mission protocols of our demonstrators, if available). From this simulated situational picture particular information items will be removed explicitly. The available crowd information will be then sensed from the system and further on used for situation enrichment (replacing removed information). We will evaluate the impact of utilizing crowd information during all crisis phases (what-if analysis: could the crowd have provided particular information). Also, relationships between crowd information and structured information in the past can be analysed to gain knowledge about indicators of crisis situations. Such knowledge then may be used to extend the situational knowledge-base of crowdSA by, e.g., defining new situation types or to adjust the existing configurations, e.g., wrt. quality checks.
Real-Life Data Case Study with Red Cross Volunteers
The project partner Austrian Red Cross comprises over 50.000 volunteers forming an interesting crowd for a read-life case study. During emergencies, this crowd may be mobilized by the Red Cross and will serve as a potential test crowd for the concepts being developed in the scope of crowdSA. In particular, it is intended to mobilise the crowd during exercise emergencies similar to the flooding scenario described in the beginning of the proposal. Besides that, crowdSA should be also tested in a potential pre-crisis phase by mobilizing visitors of large scale events like the Nova Rock festival (150.000+ visitors), instructed to share situational information of interest with the crisis manager ARC to prevent any critical situations such as eroded ground due to heavy rainfalls or even mass panic in overcrowded festival areas. They also serve for evaluating the usefulness of situation updates delivered from authorities via social media channels to the crowd. The specific test bed for executing real-world tests with crowdSA will be defined together with our domain experts.