Assessing the Effectiveness of Attack Detection at a Hackfest on Industrial Control Systems

A hackfest named SWaT Security Showdown (S3) has been organized consecutively for two years. S3 has enabled researchers and practitioners to assess the effectiveness of methods and products aimed at detecting cyber attacks launched in real-time on an operational water treatment plant, namely, Secure Water Treatment (SWaT). In S3 independent attack teams design and launch attacks on SWaT while defence teams protect the plant passively and raise alarms upon attack detection. Attack teams are scored according to how successful they are in performing attacks based on specific intents while the defense teams are scored based on the effectiveness of their methods to detect the attacks. This paper focuses on the first two instances of S3 and summarizes the benefits of hackfest and the performance of an attack detection mechanism, named Water Defense, that was exposed to attackers during S3.