Detection at Scale

Detection at Scale is a weekly newsletter that delves into the intricacies of scaling Security Information and Event Management (SIEM) systems and Detection Engineering. It covers technical guides on optimizing SIEM operations, transitioning to data lakes, correlation techniques, log filtering, incident response, and leveraging AI and automation in cybersecurity. The newsletter also discusses practical cybersecurity strategies, such as reducing alert fatigue and improving detection accuracy.

SIEM Optimization Detection Engineering Data Lakes for Security Log Filtering and Management Incident Response Cyber Threat Intelligence (CTI) AI and Machine Learning in Cybersecurity Rule Writing and Alert Handling Cloud Security Security Data Pipeline Management

The hottest Substack posts of Detection at Scale

And their main takeaways

Improving Security Data Lake Efficiency with Log Filtering

119 implied HN points • 08 Apr 24
🕹 Technology Security Data Management Cost Optimization Performance optimization
  1. Security teams can optimize SIEM costs and improve data management by filtering logs effectively before they are ingested into the system. Filtering can enhance security data lake efficiency, reducing unnecessary costs and improving overall data quality.
  2. Starting with clear intentions and asking key questions about data value, cost constraints, and threat visibility can help in creating a comprehensive and cost-efficient log filtering program.
  3. Filtering at various stages - source, in transit, and within the SIEM itself - allows security teams to reduce storage costs, optimize performance, improve data quality, and enhance the relevance of collected logs.

Unraveling SIEM Correlation Techniques

119 implied HN points • 01 Apr 24
🕹 Technology Security Monitoring Correlation Alerts Cybersecurity
  1. Correlation rules in SIEM define relationships between malicious behaviors and entities, helping in effective security monitoring and alert generation.
  2. Correlations can be simple, focusing on one technique like Brute Force, or complex, combining multiple techniques and tactics across various log sources for higher-fidelity alerts.
  3. Understanding the layers of SIEM correlation, from basic rule creation to more advanced chaining of techniques, is essential for effective cybersecurity defense.

SIEM 4.0: The Essentialist Evolution

59 implied HN points • 28 May 24
🕹 Technology Security AI Data Software Engineering Automation
  1. Security teams are moving towards prioritizing impactful MITRE tactics over complete ATT&CK coverage to reduce distracting alerts and focus on critical threats.
  2. Transitioning from individual behaviors to risk-based alerts allows for a more context-based approach, reducing alert volumes and enhancing significance.
  3. The evolution to SIEM 4.0 includes opening up data lakes, adopting 'as code' principles, and utilizing AI to automate routine tasks so human analysts can focus on high-value work.

The Five Layers of Incident Response (Part 1)

59 implied HN points • 21 May 24
🕹 Technology Security Automation Data Detection Incident Response
  1. Detection Engineering involves automating SecOps using software engineering and data principles to enhance defense capabilities without eliminating human roles.
  2. For effective Incident Response, utilize the 'Five Layers of IR': Playbook Management, Data Layer, and Presentation Layer.
  3. The Playbook sets the strategy, Data Layer defines necessary logs for playbooks, and Presentation Layer visualizes alerts and actions for human analysis.

5 SIEM Capabilities for Detection Engineering

59 implied HN points • 15 Apr 24
🕹 Technology Security Data Management Automation Cloud Computing Programming
  1. Detection Engineering involves moving from simply responding to alerts to enhancing the capabilities behind those alerts, leading to reduced fatigue for security teams.
  2. Key capabilities for supporting detection engineering include a robust data pipeline, scalable analytics with a security data lake, and embracing Detection as Code framework for sustainable security insights.
  3. Modern SIEM platforms should offer an API for automated workflows, BYOC deployment options for cost-effectiveness, and Infrastructure as Code capabilities for stable long-term management.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Leading Panther's Next Chapter as CTO

79 implied HN points • 05 Feb 24
🕹 Technology Data Analytics
  1. Transitioning from CEO to CTO to lead Panther's technical team, allowing more focus on delivering security outcomes via the product.
  2. Introduction of the concept of Detection Engineering, emphasizing reliability, scalability, and automation in security practices.
  3. Adapting Panther's approach to evolving security needs, enhancing code-driven detection for broader use and improving correlation, analytics, and visualization capabilities.

The Transition from Monolithic SIEMs to Data Lakes for Security Monitoring

139 implied HN points • 23 Oct 23
🕹 Technology Security Open Source
  1. Transitioning from monolithic SIEMs to data lakes for security monitoring involves decoupled data architecture, cloud storage, open data formats, and distributed query engines for improved performance, scalability, and pricing models.
  2. Usability tradeoffs exist when shifting to data lakes, with a need for detection engineers specializing in tool accuracy and performance, while security analysts require tools for exhaustive answers and simplistic searches.
  3. The data pipeline in a transition involves components like data routing, transformation, storage, query engines, metadata, and real-time analysis, each playing a unique role in pulling, transforming, and analyzing security data in a data lake environment.

The xz-utils Backdoor Quick Hits

39 implied HN points • 02 Apr 24
🕹 Technology Cybersecurity Software Cloud Monitoring Detection
  1. A security breach was discovered in xz-utils versions 5.6.0 and 5.6.1, allowing unauthorized remote access.
  2. Detection methods include monitoring cloud instances, correlating processes, KQL queries for Sentinel, binary analysis with YARA, Osquery, and Sysdig Falco.
  3. Reproducing the attack can be done using resources like Kali Blog and Xzbot, while there are infographics summarizing the background and timeline of the backdoor incident.

The GenAI D&R Revolution Begins

19 implied HN points • 13 May 24
🕹 Technology AI Security Automation Data Analysis Tools
  1. Security companies at RSA are increasingly focusing on AI to enhance Detection and Response (D&R) processes.
  2. Automated Tier 1 Triage using autonomous SOC analysts can streamline alert triage and analysis, improving efficiency for SecOps teams.
  3. GenAI can also improve D&R through AI-powered chatbots for automating organizational Q&A and log summarization for quicker insights and analysis.

Correlating AWS S3 Breaches

19 implied HN points • 29 Apr 24
🕹 Technology Cloud Infrastructure Security Monitoring Data Analytics Cybersecurity
  1. AWS S3 buckets are a common target for attackers due to misconfigurations and high-value data. Security teams should focus on monitoring S3 activity to ensure authorized access and detect breaches early.
  2. S3 serves as a major storage solution for various data types in the cloud. Its widespread use makes it a prime target for attackers seeking to compromise sensitive information.
  3. Monitoring S3 bucket activity is crucial for detecting suspicious behavior that could signal a breach. Using tools like CloudTrail, GuardDuty, and CloudWatch can provide valuable insights and enhance security measures.

Think Like a Detection Engineer, Pt. 1: Logging

199 implied HN points • 18 Jul 22
🕹 Technology Security Data Management Monitoring Infrastructure
  1. Detection Engineers build systems to validate security controls and detect suspicious behaviors with code to protect organizations.
  2. Security data comes from different layers like infrastructure, hosts, networks, applications, and databases, each providing unique context for monitoring.
  3. When collecting logs for security monitoring, consider tradeoffs like the value of data for detection, latency to get data into SIEM, and cost of obtaining and retaining data.

Zero False Positives from your SIEM

59 implied HN points • 05 Dec 22
🕹 Technology
  1. False positives in SIEM alerts waste time and can lead to missed real threats.
  2. Understand and categorize alerts to differentiate between those that need no action, those that require investigation, and those that indicate a serious incident.
  3. Prevent false positives by setting a true positive goal, refining rule logic, correlating behaviors, and using external enrichment for context.

The Snowflake, Live Nation, and Santander Quick Hits

2 HN points • 04 Jun 24
🕹 Technology Data Breach Cybersecurity Cloud Computing Malware Security Measures
  1. Snowflake faced claims of a massive data breach, with threats of stolen customer records from companies like Live Nation and Santander Bank.
  2. Confirmed affected companies include Live Nation and Santander Bank, with potential for more disclosures as more breaches may be revealed.
  3. To protect against breaches, Snowflake recommends enforcing multi-factor authentication, setting network policy rules, and resetting credentials.

Think Like a Detection Engineer, Pt. 2: Rule Writing

39 implied HN points • 25 Jul 22
🕹 Technology Security Software Development Testing Alerts Automation
  1. Analyzing security data effectively involves identifying and flagging bad behaviors near high-risk assets.
  2. Writing rules based on observed attacker techniques and behaviors allows for a clear path to action in response to detected threats.
  3. Testing rules through phases like unit testing, backtesting, staging, and production helps refine and ensure alert accuracy before implementation.

Five Lessons From Detection & Response Leaders

0 implied HN points • 26 Sep 22
🕹 Technology Security Engineering Automation Alerts Focus
  1. Start with high-quality log data to effectively protect what you can see and establish a reliable source during incidents.
  2. Detection teams are adopting software engineering practices to enhance scale and efficiency, promoting continuous improvement and collaboration.
  3. Automated response in security operations is crucial to reduce human error, focus on critical tasks, and evolve from reactive to proactive detection and response.

Practical Cyber Threat Intel (CTI)

0 implied HN points • 22 Apr 24
🕹 Technology Cybersecurity Threat intelligence Incident Response
  1. Cyber Threat Intelligence (CTI) helps identify malicious actors, active exploits, and ongoing attacks, guiding defenders on potential sources of attacks and hacker strategies.
  2. Tactical CTI involves indicators of compromise (IoCs) within attacker tactics and techniques (TTPs) while operational CTI tracks state-sponsored groups, enhancing detection accuracy and preparation for potential threats.
  3. Best practices for an effective CTI program include tracking prior incidents, using CTI in the context of behaviors, selecting relevant threat feeds, enriching IoCs during data ingestion, and periodically refreshing and updating threat intelligence to maintain effectiveness.

The Sisense Breach Quick Hits

0 implied HN points • 16 Apr 24
🕹 Technology Data Breach Cybersecurity
  1. Sisense experienced a breach that could have serious ramifications.
  2. The breach could impact a wide range of customers.
  3. Remediating the breach involves monitoring AWS logs and API calls, emphasizing the importance of continually monitoring data access.