Cybersecurity Incident Simulation @ Uber

All the best things come in threes: the Three Musketeers, the Three Stooges, and, of course, your favorite three-cheese pizza ordered via the UberEats app. Engineering Security (EngSec) at Uber agrees and we have formed our own trio for how we simulate cybersecurity incidents at Uber to exercise our ability to act decisively should an incident occur. This three-pronged approach consists of tabletop exercises, red team operations, and atomic simulations.

Importance of Cybersecurity Incident Simulations

While having strong preventative measures in place is vitally important, it is essential that key people and functions are well prepared to both act, and importantly act together, should an incident occur.  

Multiple approaches can help reap the full benefits of cybersecurity incident simulations, and each approach can have different benefits and limitations. For example, a simulation requiring a large amount of planning can result in more sophistication and realism, but the preparation time can limit how frequently this type of simulation can be conducted. When combined, our trio of simulations provide an array of options for cybersecurity incident response readiness.

Architecture of Our Approach

Each of our three different simulation methods has its own unique focus:

Tabletop Exercises (TTX)

These exercises simulate a security incident over a multi-hour event. TTXs complement more technical simulations by focusing on processes, roles, and equipping leaders to make decisions. The following objectives are ones that we have identified as being broadly applicable to all of our TTXs and we reflect on these post-TTX to determine the success of the exercise:

• Exercise Uber’s capabilities to respond to large-scale cybersecurity incidents and improve collaboration across teams
• Facilitate executive leadership team (ELT) cybersecurity awareness and familiarity
• Exercise leadership team (LT) decision-making processes
• Identify strengths and areas of improvements to enhance Uber’s cybersecurity response capabilities  
• Improving the incident response team’s understanding of a technical area and gain a general understanding of incident handling in that area

At Uber, we have moved away from the more traditional, highly scripted TTX format and reimagined our TTXs with the goal of each participant playing themselves as realistically as possible. Our execution looks something like this:

• A finding or experience is forwarded to the virtual Security Operations Center (vSOC) by someone who has “discovered” it
• The vSOC receives the report, triages, investigates, and escalates the issue to appropriate personnel for additional investigation and coordination per SOP
• An “inject” is given to the partially assembled Cyber Incident Response Team (CIRT)
• Additional CIRT members are brought in and work streams are activated
• Additional inject is given to the assembled CIRT
• Executive leadership joins for a scheduled brief by the CIRT team, focusing on asking the CIRT team questions as if this was a real-world scenario
• Additional work streams are activated, injects are given, and leadership briefs conducted until the conclusion of the TTX

For our TTXs, injects are pieces of additional information the “game-master” provides to keep the simulation moving forward for the team. These could be anything from investigative findings, to questions from external stakeholders, to anything else those designing the simulation identify as beneficial to drill. These injects are given at intervals that allow CIRT members to exercise their process and solve problems between each inject.

TTXs are valuable in bringing together a range of cross-functional teams with a scenario playing out in close to real time, which subsequently provides more valuable findings for us to address post-exercise.

Red Team Operations

Uber’s Red Team operations are high overhead when compared to a TTX or Atomic Simulation. These operations take a large amount of research and planning but are very beneficial and can match the complexity of real-world attacks. Uber’s Red Team operations aim to mimic real-world threat actor activity from the point of intrusion through either action on objective or their eviction from the network. Uber Cyber Defense also puts on an annual capture the flag event, where we bring together teams from across the company to respond and solve problems collaboratively.

The simulations mimic attack chains that are seen (or realistically could be seen) in the wild. These allow us to test our detection, response, and investigative capabilities in a realistic way. The operations help us to respond to sophisticated attacks occurring in many possible environments. These simulations are unannounced and treated as real incidents as the response team usually does not determine they are Red Team until later stages of the investigation.

Uber also runs an annual Red vs. Blue event that forms an unofficial capstone to our simulation program. This pre-announced simulation comes with buy-in and participation from key stakeholders. Teams from around Uber get together for a pre-planned two weeks of responding to the Red Team and working together to track and evict them from our environment. This event is a fun culmination of all the simulation work and practice done in the past year, flexing the lessons learned, and concludes with multiple read-outs to further drive cybersecurity at Uber forward.

Atomic Simulations

Atomic simulations are the much smaller and less complex side-kick to Red Team operations. These simulations focus on testing detections, SOPs, small pieces of real world incidents, and details from our threat intelligence briefs. They are low overhead and repeatable, allowing us to test improvements near real time. Using these atomic simulations, we can identify areas where improvements can be made, and then quickly retest to gauge the effectiveness of the changes we implemented. 

We usually execute our Atomic Simulations as a chain of 5 or 6 tactics, techniques, and procedures (TTPs) that when brought together form a likely path that a threat actor would follow. As opposed to a Red Team operation, Atomic Simulations are more straightforward to plan and execute, with a basic scenario looking something like this:

• Simulation team places a remote access trojan (RAT) on an employee’s host in the ~/Downloads folder
• RAT is run connecting to a command and control (C2) server
• Simulation team then tries to run the following commands:
  • Downloads and runs Nmap to discover hosts on the network
  • Views and copies known hosts and SSH keys from the .ssh directory and dumps the host keychain
  • Laterally moves to different hosts/environments using the data found
  • cURLs down additional tools such as ngrok or ADfind
  • Conducts additional reconnaissance and persistence TTPs
  • Finally, sends the data collected to the C2 via DNS
• Response team conducts an after-action, Post-Incident Review discussing identified gaps and improvements to be made

We can plan, stage, and run this type of simulation over the course of about two days. Responding to these smaller scale incidents gives the response teams a great way to test their response to low frequency, high impact detections that may fire in response to the simulation.

We also use atomic simulations to help integrate new team members. Their flexibility and ease of use allow new members to leverage their new knowledge of our incident response process and technologies. By threading a simulation through a few environments at Uber with multiple IOCs, we are able to give new members of the team a great opportunity to pivot through our different tools and SOPs.

Figure 1

Our three-pronged approach to cybersecurity incident simulations offers a broad way to test our security posture. We track our coverage not only by environment (such as corp, prod, cloud, etc.) but also by utilizing the MITRE ATT&CK® navigator. This framework enables us to map our simulations to TTPs and quickly determine how many we have simulated in the past year. The ATT&CK map is also combined with our threat hunting program to give a comprehensive view of the TTPs we have covered thanks to a few of our proactive security programs.

Figure 2: Fictional example ATT&CK Matrix showing TTPs that have been simulated
© 2022 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation.

Conclusion

We utilize these simulations to keep up with the ever-evolving threat landscape and help prepare our cybersecurity teams to respond to the latest threats. We stand for safety and our approach to cybersecurity incident simulations is just one of the ways that we work to protect our riders, earners, eaters, and employees.

Header Image Attribution: The “Poseidon rising” image is covered by a CC BY 2.0 license and is credited tojanoma.cl.