The hottest Data Management Substack posts right now

And their main takeaways
Category
Top Technology Topics

Microsoft Sentinel SOC 101: How to Detect and Mitigate Multiple Microsoft Teams Deleted by a Single User with Microsoft Sentinel

Rod’s Blog • 39 implied HN points • 07 Feb 24
🕹 Technology Data Management
  1. Use Microsoft Sentinel to detect and respond to multiple Teams deletion events in your organization.
  2. Collect Teams activity logs in Microsoft Sentinel to monitor data and detect security risks.
  3. Write custom analytics rules in Microsoft Sentinel to generate alerts for suspicious activities, such as multiple Teams deletion by a single user.

How to be Notified When Microsoft Sentinel Data Stops Populating

Rod’s Blog • 79 implied HN points • 02 Oct 23
🕹 Technology Data Management
  1. Being notified when data ingestion stops is crucial for security analysts to maintain the integrity of security tools.
  2. A KQL query can be set up as an Analytics Rule to alert if a specific table has not received new data within a set timeframe, allowing for timely action.
  3. Email alerts can be configured instead of generating unnecessary security incidents, ensuring the operations team can address potential issues efficiently.

Why Discord ditched Cassandra [System Design Sundays]

Technology Made Simple • 79 implied HN points • 03 Apr 23
🕹 Technology Data Management
  1. Discord faced performance issues with Cassandra, requiring increasing maintenance effort and leading to unpredictable latency.
  2. Hot partitions were a problem in Cassandra, causing hotspotting and impacting the database's performance during concurrent reads.
  3. Garbage collection in Cassandra posed challenges, leading Discord to switch to ScyllaDB which does not have a garbage collector.

July 2023

Product Composition • 78 implied HN points • 21 Jul 23
🕹 Technology Data Management
  1. Decipad is launching its beta version, focusing on making sense of numbers in a dynamic way
  2. AI in the industry should prioritize collapsing data to enhance clarity and facilitate action-taking
  3. The future of jobs is facing a drastic shift, with issues around productivity, social contracts, asymmetrical compensation, and poor job descriptions

Joe's Nerdy Rants #4

Joe Reis • 78 implied HN points • 10 Jun 23
🕹 Technology Data Management
  1. Encourage kids and others to interact more in real life, consider alternatives to college, find careers that can't be easily automated, and learn to coexist with AI.
  2. Embrace lifelong learning and be open to change in order to adapt to evolving technologies and industries.
  3. Read up on interesting articles about tech, AI, data, and business topics for insights and inspiration.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

The Tech Buffet #19: How To Build and Deploy an LLM-Powered App To Chat with PapersWithCode

The Tech Buffet • 39 implied HN points • 03 Feb 24
🕹 Technology Data Management
  1. You can build a personal assistant to easily find and understand the latest machine learning research. This assistant will let you ask questions in simple language.
  2. The app uses a system that retrieves and generates information, utilizing a database and machine learning models. It processes data from a site called 'Papers With Code'.
  3. The guide provides step-by-step instructions on how to create, index, and deploy this assistant as a web application, including ready-to-use source code.

What is Data Orchestration and why is it misunderstood?

The Orchestra Data Leadership Newsletter • 39 implied HN points • 28 Jan 24
🕹 Technology Data Management
  1. Data orchestration is often confused with workflow orchestration, but it involves more than just triggering and monitoring tasks; it includes reliably and efficiently moving data into production.
  2. Reliably and efficiently releasing data into production is complex and involves elements like data movement, transformation, environment management, role-based access control, and data observability.
  3. Implementing end-to-end and holistic data orchestration offers transformative benefits such as intelligent metadata gathering, data lineage, environment management, data product enablement, and cross-functional collaboration for scalable data operations.

Get a Microsoft 365 Copilot security teardown without hours of documentation review

Deploy Securely • 39 implied HN points • 24 Jan 24
🕹 Technology Data Management
  1. Microsoft 365 Copilot provides detailed data residency and retention controls favored by enterprises in the Microsoft 365 ecosystem.
  2. Be cautious of insider threats with Copilot as it allows access to considerable organizational data, potentially leading to inadvertent policy violations.
  3. Consider the complexities of Copilot's retention policies, especially in relation to existing settings and the use of Bing for web searches.

Faster computers afford dumber solutions

Wednesday Wisdom • 104 implied HN points • 18 Dec 24
🕹 Technology Data Management
  1. Faster computers let us use simpler solutions instead of complicated ones. This means we can solve problems more easily, without all the stress of complex systems.
  2. In the past, computers were so slow that we had to be very clever to get things done. Now, with stronger machines, we can just get the job done without excessive tweaking.
  3. Sometimes, when faced with a problem, it's worth it to think about simpler approaches. These 'dumb' solutions can often work just as well for many situations.

How to Monitor the Microsoft Sentinel Trial Period

Rod’s Blog • 59 implied HN points • 07 Nov 23
🕹 Technology Data Management
  1. For Microsoft Sentinel customers, a 31-day trial period is available by enabling Microsoft Sentinel on a Log Analytics workspace.
  2. To monitor the trial period, look under the 'News & Guides' blade and access the 'Free Trial' tab to see how many days are left.
  3. In the past, the 31-day trial could be enabled unlimited times on new workspaces, but now it's limited to 20 times per Azure subscription.

The Case For An AI Productivity Suite

Cobus Greyling on LLMs, NLU, NLP, chatbots & voicebots • 19 implied HN points • 12 Apr 24
🕹 Technology Data Management
  1. An AI productivity suite helps people and businesses work more efficiently by combining tools for tasks like data analysis and automation.
  2. It allows users to automate regular tasks, freeing them to focus on more important work, and offers easy customization through no-code options.
  3. These suites also promote teamwork by improving communication and sharing among team members, leading to better project outcomes.

Databricks Unity Catalog: Enabling Data Democratization

Data Plumbers • 19 implied HN points • 08 Apr 24
🕹 Technology Data Management
  1. Data democratization is vital for modern data strategies, making data more accessible and understandable within an organization for informed decision-making and better customer experiences.
  2. Databricks Unity Catalog supports data democratization by providing a centralized governance layer, simplifying access management, enabling unified data management, and fostering data discovery, collaboration, and sharing.
  3. Implementing data democratization requires robust data governance and security measures to mitigate risks of privacy violations and data leaks.

Think Like a Detection Engineer, Pt. 1: Logging

Detection at Scale • 199 implied HN points • 18 Jul 22
🕹 Technology Data Management
  1. Detection Engineers build systems to validate security controls and detect suspicious behaviors with code to protect organizations.
  2. Security data comes from different layers like infrastructure, hosts, networks, applications, and databases, each providing unique context for monitoring.
  3. When collecting logs for security monitoring, consider tradeoffs like the value of data for detection, latency to get data into SIEM, and cost of obtaining and retaining data.

Building a Working Recommendation Engine from Scratch

Building a Recommendation Engine • 3 HN points • 04 Aug 24
🕹 Technology Data Management
  1. A recommendation engine can work without complex machine learning. Instead, it can be built using straightforward connections between content to suggest things users might like.
  2. Using an API from a platform like Are.na allows easy access to user content and helps find connections between different channels, making recommendations more relevant.
  3. It's important to filter out content that users already know or follow to give them fresh and exciting recommendations. Regular updates to the recommendations can also help keep things interesting.

Must Learn AI Security Part 8: Model Stealing Attacks Against AI

Rod’s Blog • 59 implied HN points • 05 Sep 23
🕹 Technology Data Management
  1. A Model Stealing attack against AI involves an adversary attempting to steal the machine learning model from a target AI system, potentially leading to security and privacy issues.
  2. Different types of Model Stealing attacks include Query-based attacks, Membership inference attacks, Model inversion attacks, and Trojan attacks.
  3. Model Stealing attacks can result in loss of intellectual property, security and privacy risks, reputation damage, and financial losses for organizations. Mitigation strategies include secure data management, regular system updates, model obfuscation techniques, monitoring for suspicious activity, and implementing multi-factor authentication.

A Quick Way to Verify the Connection Between Microsoft Defender External Attack Surface and Microsoft Sentinel

Rod’s Blog • 59 implied HN points • 13 Jun 23
🕹 Technology Data Management
  1. Check for custom tables starting with 'EASM' to verify connection between Microsoft Defender External Attack Surface and Microsoft Sentinel.
  2. In Microsoft Sentinel, tables will show up in the Custom Logs Solutions area.
  3. Connecting EASM to Microsoft Sentinel involves three steps: setting up EASM, configuring permissions, and enabling the connection.

How to build a dual Incremental + snapshot data ingestion pipeline

Practical Data Engineering Substack • 59 implied HN points • 01 Oct 23
🕹 Technology Data Management
  1. You can improve data accuracy by using two pipelines: one for getting recent updates quickly and another for regularly loading the entire dataset. This helps in keeping the data reliable over time.
  2. It's essential to manage pipeline scheduling based on your business's needs, like how often you need updates. You can choose faster updates or less frequent full reloads depending on how critical the data is.
  3. Using tools like Apache Airflow can help organize these pipelines efficiently. You can simplify tasks by dynamically generating them from a list, making it easier to handle many data tables.

The History and Evolution of Open Table Formats - Part I

Practical Data Engineering Substack • 2 HN points • 15 Aug 24
🕹 Technology Data Management
  1. Open Table Formats have changed how we store and manage data, making it easier to work with different systems and tools without being locked into one software.
  2. The transition from traditional databases to open table formats has increased flexibility and allowed for better collaboration across various platforms, especially in data lakes.
  3. Despite their advantages, old formats like Hive still face issues like slow performance and over-partitioning, which can make data management challenging as companies grow.

AI Observability, Orchestration, Consolidation

Gradient Flow • 179 implied HN points • 26 May 22
🕹 Technology Data Management
  1. Companies are likely to use at most two platforms for managing the entire machine learning pipeline: one for exploration and another for deployment and operations.
  2. Prefect 2.0 is a popular framework for data and workflow orchestration, emphasizing 'code as workflows' to address data engineering challenges.
  3. The survey on workflow orchestration tools revealed a growing interest in these systems, with startups raising over $450 million in funding for orchestration solutions.

Introducing Orchestra: rapidly Build and Monitor Data and AI Products

The Orchestra Data Leadership Newsletter • 19 implied HN points • 07 Mar 24
🕹 Technology Data Management
  1. Launching a free tier for Orchestra, a tool to build and monitor data and AI products, offering a lightweight approach to improving business value and AI integration.
  2. Addressing the challenges faced by data teams in balancing business value and software engineering best practices through tools like Nessie, dbt, and emerging 'as-code' BI platforms.
  3. Providing an end-to-end platform with features like declarative pipelines, data quality monitoring, granular alert control, and asset-based data lineage to empower data teams in accelerating their initiatives.

Supercharging Your Data and AI Platforms

Gradient Flow • 199 implied HN points • 10 Mar 22
🕹 Technology Data Management
  1. Data management trends are crucial for data teams and architects to stay updated on
  2. The Data Exchange podcast covers topics like Continuous Intelligence, NLP in Healthcare, and Graph Intelligence Stack
  3. New tools like TorchRec, EvoJAX, and managing public cloud resources are enhancing data and machine learning infrastructure

Risk Tolerance & Raising the Technical Debt Ceiling

Resilient Cyber • 79 implied HN points • 22 May 23
🕹 Technology Data Management
  1. Many organizations don't clearly define their risk tolerance in cybersecurity, impacting their ability to manage risks effectively. If a company doesn't know what risks it faces, it can't protect itself properly.
  2. There's a significant gap in measuring and understanding risks, especially with the rise of cloud services and software. Organizations often struggle to keep track of what software and hardware they use, leading to hidden vulnerabilities.
  3. Organizations are facing a backlog of vulnerabilities that they can't keep up with. If too many risks are left unresolved, it raises questions about their actual risk appetite and ability to protect themselves.

Five Lessons for Building Robust AI Agents from Coding Agents

Tanay’s Newsletter • 56 implied HN points • 22 Jan 25
🕹 Technology Data Management
  1. Having clear rules and structured frameworks helps AI work better. By defining specific inputs and outputs, AI can understand what to do more easily.
  2. Using well-organized and detailed data helps AI learn faster. The more context and reasoning behind data points, the better AI can make decisions.
  3. Measuring how well AI performs with clear goals and regular tests is important. This allows AI to keep improving and adapting to different situations.

Authorization in microservice architecture, P0: Introduction

Hung's Notes • 3 HN points • 18 Jul 24
🕹 Technology Data Management
  1. Building a solid authorization system in microservices is tough since there aren’t clear guidelines. It's vital to share experiences for better solutions.
  2. Managing permissions can get complicated as a business grows. A better approach is needed to handle access control efficiently.
  3. Security is critical in public safety products, and proper access management helps maintain trust and legal compliance.

Using Azure Open AI with Microsoft Sentinel Part 1 - Getting Keys and Endpoints

Rod’s Blog • 39 implied HN points • 30 Mar 23
🕹 Technology Data Management
  1. Consider transitioning from Logic App connector for Open AI ChatGPT to Azure Open AI's ChatGPT for more control over data.
  2. When working with Azure Open AI models, deployments should be done in the Azure console, not Azure OpenAI Studio, and need patience for the API to become accessible.
  3. In Microsoft Sentinel, use best practices like storing API keys and endpoints in Parameters for calls to Azure Open AI deployments.

Securing Data in an AI-First World with Microsoft Purview

Rod’s Blog • 19 implied HN points • 06 Feb 24
🕹 Technology Data Management
  1. Microsoft Purview is a top industry solution for managing data estates, offering governance, protection, and management.
  2. The latest enhancements to Microsoft Purview and Microsoft Defender focus on securing data in the context of generative AI, providing visibility, protection, and compliance controls.
  3. Organizations can leverage Microsoft Purview and Microsoft Defender to securely adopt AI, ensuring data protection while harnessing AI's full potential.

sqlmesh plan

davidj.substack • 59 implied HN points • 10 Dec 24
🕹 Technology Data Management
  1. Virtual data environments in SQLMesh let you test changes without affecting the main data. This means you can quickly see how something would work before actually doing it.
  2. Using snapshots, you can create different versions of data models easily. Each version is linked to a unique fingerprint, so they don't mess with each other.
  3. Creating and managing development environments is much easier now. With just a command, you can set up a new environment that looks just like production, making development smoother.

Securing Data that is Used by AI

Rod’s Blog • 19 implied HN points • 25 Jan 24
🕹 Technology Data Management
  1. Securing data used by AI is vital for security, performance, reliability, ethics, and trust.
  2. Data hygiene practices include collecting necessary data types, encrypting data, and maintaining data lineage.
  3. Ensuring data quality through validation, diversity, and detection methods is crucial for accurate and fair AI outcomes.

Tokenization, or How I Learned to Stop Worrying and Love the Vault

Money in Transit • 19 implied HN points • 08 Jan 24
🕹 Technology Data Management
  1. Tokenization is a powerful way to reduce costs and secure card payments by isolating parts of payment applications for PCI compliance.
  2. Tokens are non-exploitable and require a vault to store the actual data, providing security in case of a breach.
  3. Using Tokenization as a Service providers can strengthen a startup's position by avoiding vendor lock-in and enhancing pricing power.

sqlmesh model kinds - 2

davidj.substack • 47 implied HN points • 09 Dec 24
🕹 Technology Data Management
  1. There are three types of incremental models in sqlmesh: Incremental by Partition, Unique Key, and Time Range. Each type has its own unique method for handling how data updates are processed.
  2. Incremental models can efficiently replace old data with new data, and sqlmesh offers better state management compared to other tools like dbt. This allows for smoother updates without the need for full-refresh.
  3. Understanding how to set up these models can save time and resources. Properly configuring them allows for collaboration and clarity in data management, which is especially useful in larger teams.

All I want in AI is some context and a chat window

Pedram's Data Based • 20 implied HN points • 22 May 25
🕹 Technology Data Management
  1. Having a simple chat interface makes it easy for non-technical people to use AI tools. This helps in accessing valuable resources without needing complex setups.
  2. Providing relevant context is crucial for the effectiveness of AI. When the right information is fed to AI, it can give much better and accurate responses.
  3. Integrating tools and data sources can improve AI's capabilities but remains a challenge. Companies need better systems to pull together all the necessary information for their teams.

How Artificial Intelligence Will Change the Chemical Industry

The Polymerist • 116 implied HN points • 16 Jan 24
🕹 Technology Data Management
  1. Companies in the chemical industry can benefit from AI tools to improve efficiency and profitability.
  2. AI tools are becoming more accessible for functions like customer relationship management, inventory management, and data organization.
  3. While AI won't replace R&D functions, it can significantly enhance productivity and help companies stay competitive in specialized chemical sectors.

6 Technologies you Should Know as a Software Engineer in 2023

Implementing • 19 implied HN points • 18 Dec 23
🕹 Technology Data Management
  1. Importance of continuous learning in the field of web development, especially in mastering foundational concepts like math and computer science.
  2. Key technologies like Docker, Node.js, Git, Elasticsearch, Redis, and React are essential for developers to learn for successful software engineering in 2024.
  3. Utilizing online resources like free YouTube videos, paid courses on platforms like Udemy, and official documentation can assist in gaining proficiency in various technologies.

Job listing: Assistant Sports Analyst

Silver Bulletin • 30 implied HN points • 26 Feb 25
🎾 Sports Data Management
  1. An Assistant Sports Analyst position is open, mostly focusing on improving sports models for NFL, NBA, and college basketball. It's part-time and could turn into full-time.
  2. Candidates need skills in Stata, Python, and data analysis, along with a strong interest in sports. Pay ranges from $40-50 per hour, depending on work done.
  3. To apply, email with your materials and be prepared for interviews in early April. The deadline to apply is March 25, 2024.

Resiliency beyond the Classic Circuit Breaker

System Design Classroom • 2 HN points • 10 Jul 24
🕹 Technology Data Management
  1. To handle system failures, you can use different strategies like 'Fail Fast' which stops operations quickly to save resources. But this can affect user experience because they won't get a chance to recover from the error.
  2. Another approach is 'Fail Silent', where instead of showing an error, the system quietly returns a default value. It helps keep things running smoothly, but users might miss important information if data is missing.
  3. Lastly, there's 'Custom Fallback', which uses saved local data when a service fails. This keeps the service active, but the information might be outdated, which can confuse users.