Alarms Overload - Are You Suffering from Alarm Fatigue?

When you work in security for long enough, you get used to being tagged as 'paranoid' by non security people. Locking your computer when you walk away, even at home. Pulling an all nighter to rebuild a server you think is acting weird. Calling your mom to remind her to install the latest iPhone update. Everything you read or watch gets filtered through a lens of either 'how can i defend against this' or 'how can i use this to break stuff', depending on your role.

Recently I was listening to one of my favorite podcasts, 99 Percent invisible and found myself making comparisons against network security products. The episode had a piece discussing the huge amounts of alarms that we experience in our daily life. According to their expert, the average person probably hears about 100 different types of alarms a day, from your alarm clock, to your mobile, to the microwave. All those beeps, blings, and boops you run into are so common, most of us have been conditioned to ignore them.

"Alarms being overpowering is a big problem — if they are so obnoxious that people tune them out or turn them off, they aren’t effective. I think the whole world suffers of alarm fatigue.” - Judy Edworthy

And if that number is true for the average person, a security practitioner probably sees 10,000 times that. Firewalls, IPS, Endpoint protection, Web Gateways, and so on. Often gathered in a SIEM tool, so you can be overrun in a single place.

They also discussed two key examples that hit home for me. Three Mile Island, where the alarms went off, hundreds of alarms at the same time actually, but due to the sheer number of alarms the workers had difficulty in figuring out which one was the most critical and make those alarms actionable. On the opposite side the, Deep Water horizon oil spill where some of the alarms were disabled for at least a year because false alarms were waking up sleeping crew.

Anyone who has worked in security will be familiar with both scenarios above. The ever-scrolling window of alarms, and the disabling of those noisy alarms we don't understand or have not been given the mandate to fix properly. I've seen many many IPS systems that are no longer monitored due to the poor quality and sheer number of alarms they generate.

In the podcast, they call this 'false alarm syndrome' or 'alarm fatigue', and the struggle is real.

"Everything beeps, and for no good reason quite a lot of the time." - Judy Edworthy

If everything that happens generates an alert in your security infrastructure, then nothing is important. I've seen snort rule sets well over 20,000 rules, alerting on nearly every single packet that is transmitted. Security is a matter of quality, not quantity.

Since our Threat Protection Centers sit outside of a customers perimiter, we see hundreds of thousands of alarms every hour. Known bad IP addresses, malicious traffic, and of course DDoS attempts. If we were to just pass these wholesale to our customers, we would quickly approach a Three Mile Island scenario. With this in mind, we apply a number of techniques to minimize the amount of alarms we show our users. By employing a funnel principle, we ensure that the most severe events are brought to the top, while the thousands of attacks from known bad hosts are logically grouped. These events can be sent wholesale to your SEIM, or in an on demand basis per incident.

One additional benefit from this design is that it raises the overall quality of events in your on premise firewalls, IPSs, etc. By delivering cleaner traffic to your datacenter, network and security analysts will be able to focus on more specific attacks, and have more time in which to do their investigations.

Make sure the alarms you see actually mean something, and aren't just another beeping machine vying for your limited attention.

Links

https://99percentinvisible.org/episode/mini-stories-volume-4/2/ (Audio starts at 11:18 for those who want to listen).