28 Oct 2019

Security Engineering Advice: 4 Ways to Prepare for Security Incidents

Intro

The first thing that comes to mind when one thinks about security issues is typically some poorly written code that is prone to RCE, XSS, and similar attacks. But hardly anyone deliberately sets out with “I’m going to write some really bad, vulnerable code today!” intent in mind. Code is always written for a reason, driven by functional demands, culture, and processes.

From a software engineer's point of view, fixing a security issue equals removing an opportunity to exploit something. From a security engineer’s point of view, such fix is just putting a band-aid on a larger problem.

Let’s look into this through the eyes of a software architect. When a large distributed application hits performance bottleneck, how long until you’re unable to throw more resources into it and go bust? Sure, it depends on your workload and project goals but at some point, you’d still be forced to go back and redesign the architecture to eliminate that bottleneck, one way or another.

After fixing a series of similar issues, several times, you’d finally learn how to build systems with future load scaling in mind and start building processes for planning, measuring, and improving the reliability of your systems from the get-go. Software security can be approached in a similar fashion.

Secure Software Development Lifecycle (SSDLC)

If we look at software development process from a security perspective, the security industry has distilled the development lifecycle into a clear process, which also fits both waterfall and agile development strategies (the most popular SSDLC methodologies are OWASP SSDLC, OWASP CLASP and Microsoft SDL).

Let’s see what SSDLC looks like on the inside:

SSDLC stages.

SSDLC consists of several stages that go hand in hand with the development stages. Generally speaking, a secure SDLC is set up by adding security-related activities to an existing development process. Let’s say, we’re planning to add a new feature to our software product. First, product owner realises why the product needs a feature and how users will benefit from it, then software architect decides how to build it, and security engineers evaluate the risks (of users' data loss, of non-authorised access to the critical assets, of potential vulnerabilities’ exploitation, and of losing money due to incidents).

When developers decide upon the technological stack required to build a feature, security engineers build a threat model and a security plan. It contains a list of potential threats the application (or infrastructure) will face and the ways to prevent them. On every development stage, there’s some work for security engineers.

However, one important step is still missing from this picture — it is the incident response.

It doesn’t matter if you use SSDLC in your company or not, your company does or will react to incidents in some way.

We’ve described the 4 typical incident response scenarios, sorted by their cost efficiency levels below.

Scenario 1. Addressing security issues after incidents

The response typically takes place when a security deficiency is detected. For some companies, this only happens after a breach, when the company internally (or publically, learning about it from the news 😱) realises that they were hacked and some of the assets leaked.

"Incident response" is what happens when external forces cause a breach.

In this case, the cost of mitigation is staggering:

  • Legal and reputational damage after a successful breach;
  • Understanding the causes (unfortunately, the search for the real root causes often gets postponed into the indefinite future and the problems are only addressed with basic patches);
  • Chaotic attempts at hotpatching and hotfixing the security issues where possible, not where it’s necessary;
  • Aligning policies and processes with the patched-up infrastructure;
  • Realigning policies and processes to prevent similar breaches in the future (if some conclusions regarding the root cause have been made).

This is the most stressful and expensive way — you have to deal with and mitigate the damage, respond to the security incident, and at the same time, you still need to (urgently) fix the app or the whole infrastructure. For a taste of what can be waiting for you, you can check out the security incident post-mortems by StackOverflow, NPM (who decided to call themselves “Newly Paranoid Maintainers” after the incident), Sentry, or the massive infamous Equifax breach.

Scenario 2. Addressing security issues after assessment/pentests

Respond to incidents before actual incidents take place by fixing security issues found by auditors/pentesters.

In fact, from the perspective of SSDLC, fixing security issues after incidents is no different from fixing issues after a security review (let’s forget about the incident response / forensic / damage assessment parts for a second): the issues are there, already detected, now you have to respond.

More and more companies are acting preemptively — by hiring white-hat hackers to emulate realistic attacks on their systems and to find (and get the developers to fix) the security issues before the bad guys get to them. By fixing critical problems, companies try to decrease the financial damage in case of an actual attack. But can pentests find everything at once? Typically, they target and help to uncover only the most obvious problems so each next pentest might find something else that needs addressing and fixing if the system is sufficiently large.

Pentests are effective if you run them constantly — before each release or every 3-6 months — when you follow some Grand Plan instead of “let’s try to hack this thing again” approach. The Plan usually involves a security roadmap with risks & threats’ model and allows pentesters to focus on the most impactful threats (we’d already covered the topic of dealing with an external security team in our blog).

Scenario 3. Addressing security issues during the engineering process

Secure coding is more than just watching for buffer overflows. This is also configuring application as “secure by default”, setting up strict access control and authentication, encrypting sensitive data, handling errors and logs.

The next level of being smart is deploying SSDLC during the actual development — employing security-conscious engineers alongside feature engineers, spotting the security problems that may happen early on, and fixing them during development. Here the fixes are more consistent and efficient; it’s typically cheaper to find them at this stage as it requires less effort from the security team than the need to look for bugs later on.

There is a number of applicable security standards for any ecosystem that can be used by developers and security engineers alike to ensure that the code in development is secure enough. The security industry has accumulated a sufficient body of evidence and knowledge to come up with testing methodologies for and common causes of security issues (OWASP Secure Coding practices checklist).

Need assistance with improving your cybersecurity program? We can help.

However, many security issues would be avoidable, if the bug hunt from the POV of application security had started a few steps earlier. Also, understanding what data to encrypt can help you comply with the privacy regulations more easily and actually start building with security in mind.

Scenario 4. Preventing security issues during software architecture phase

Focus on preventing whole classes of risks, not just specific vulnerabilities.

The crown jewel of secure engineering is addressing the security concerns where they need to be addressed — during the design and planning stage of your system. By spending time on considering the security risks and implications of technical decisions during the initial stage of the project and integrating processes and tools necessary for providing basic security functions, you can eliminate many security issues and threats from the start. This reduces the chance of somebody potentially exploiting the security-related bugs as they’ve been eliminated from the project as early as at the planning stage.

This approach automatically utilises core SSDLC steps during the development process (because security requirements are now part of architecture, that needs to be implemented and verified), but the amount of time security engineers spend finding the vulnerable code is greatly reduced. It also implies using pentests to verify the security posture — but well-designed systems, even when vulnerable, still prevent attackers from achieving their goals.

In any architecture, security is one of the non-functional attributes, meaning, it does not instantly stand out as top priority. But if the business risks related to possible security incidents are considerable, then scenario 4 is a perfect entrance point into the world of proper security practices.

Why don’t we all start with scenario 4?

Each business is unique, with its individual constraints and requirements and it faces different problems at every stage. You’ve probably already heard all those “get to market”, “deliver fast”, and other popular excuses for not implementing proper security from the very beginning.

Picture from Kiwicon III, courtesy of @lady_nerd. Look at the date — not much has changed since 2009!

In fact, we’ve done a short Twitter survey asking fellow engineers to recall the excuses they’ve been given for dealing with security later (as in: “never”).

Our top picks are:

  • “we need to release the app first”;
  • "we don't have sensitive data";
  • “we are not a bank!”;
  • "it will never happen to us. You are being paranoid.";
  • "nobody will attack our website";
  • “doesn’t bring in any user value”;
  • “we can't afford the development costs”.

Doesn’t it ring a bell? ;)

Software development does not take place in a vacuum. Returning to the notion of “bad code” from the beginning of this article, in an ideal a rational world, any piece of code written is written to solve some existing real-world problems. So getting to any level beyond scenario 1 requires additional efforts of educating decision-makers about the actual necessity and value of the security efforts.

Scenario 1 will happen to you anyway, sooner or later, whether you want it or not, if you do nothing about the existing security issues.

Scenario 2 requires a bare minimum — compliance pressure looming over your business or a sudden realisation that something could happen, combined with having some money. If your company currently has no security stakeholders, but you start thinking about security, asking “white hat hackers” to crack your application(s) might be a good way to start.

Scenario 3 requires a certain maturity of development processes for it to be cost-efficient/productive — security engineers are efficient (bring noticeable value) when you manage your test suite well, document the code, etc. You can make your life much easier by using various tools — code scanners (SAST, DAST), dependency vulnerability management (Snyk, whitesourcesoftware) along with security engineering assistance.

But the situation changes radically if one data breach wipes you out as a business. Suddenly, affording the right approach towards security becomes an enabler and asset rather than an impediment. At this level, you might find the OWASP Software Assurance Maturity Model (OWASP SAMM) useful. It is a framework that helps organisations formulate and implement strategies for effective software security, depending on the specific risks that the business or organisation is facing.

Scenario 4 requires the readiness to support strong and mature processes combined with the presence of a strong security stakeholder who is authorised to make decisions. Implementing scenario 4 is cheaper in the long-term, but requires long-term thinking, which is not quite fit for most startups and is only partially suitable for regular SME.

Conclusions

Choosing the right approach requires a long honest look in the mirror, backed with careful planning and budgeting. But since even startups manage to pull off doing the security right (occasionally), most of the larger companies can do it, too.

Implementing good security and proper SSDLC is an ongoing process with inevitable pitfalls, but the choice is ultimately yours — to go full-on Equifax, with millions of dollars in fines or full-on Google, where the reaction to a suspected security breach resulted in the emergence of new technologies and frameworks.

Learn more

We provide security advisory and security engineering services and can help you on different stages of life of your company. If you process sensitive data and would like to have some assistance — contact us.