The hottest Orchestration Substack posts right now

And their main takeaways
Category
Top Technology Topics

A detailed guide to running dbt Core in Production in AWS on ECS

The Orchestra Data Leadership Newsletter • 79 implied HN points • 28 Mar 24
🕹 Technology Cloud Computing Data Engineering AWS CI/CD Orchestration
  1. A detailed guide to running dbt Core in production in AWS on ECS is outlined, focusing on achieving cost-effective and reliable execution.
  2. Running dbt in production is not highly compute-intensive, as it primarily serves as an orchestrator, making it more cost-efficient compared to running Python code that utilizes compute resources.
  3. By setting up dbt Core on ECS in AWS and using Orchestra, you can achieve a scalable, cost-effective solution for self-hosting dbt Core with full visibility and control.

Running dbt-core on Github Actions: a comprehensive guide

The Orchestra Data Leadership Newsletter • 39 implied HN points • 18 Apr 24
🕹 Technology Data Transformation Automation Workflow management Orchestration
  1. Advantages of running dbt-core on GitHub Actions include easy workflow definition in Git, immediate access to latest code, and no need to provision instances for GitHub hosted runners.
  2. Disadvantages of running dbt-core on GitHub Actions include being limited by GitHub's workers, 'fire and forget' implementation, and overhead when connecting to external services.
  3. GitHub Actions workflows can be triggered from external sources like orchestrators using the repository dispatch event or the workflow_dispatch event, providing flexibility in integrating GitHub's CI/CD capabilities into larger automation strategies.

The Modern Data Stack is Dead, Long Live the Modern Data Stack - Part 2

davidj.substack • 107 implied HN points • 26 Jul 23
🕹 Technology Data Stack Orchestration Consumption Semantic Layers
  1. The modern data stack is evolving with new tools and options for data architecture.
  2. Key trends include the focus on data ingestion and telemetry, improved orchestration tools, and advancements in compute engines.
  3. Data consumption is being enhanced through self-serve AI capabilities, BI tools, and free-form analyst tools, all sitting on a semantic layer.

Microsoft Sentinel SOC 101: How to Detect and Mitigate Rare Domains Seen in Cloud Logs

Rod’s Blog • 39 implied HN points • 19 Oct 23
🕹 Technology Security Cloud Computing Threat Detection Automation Orchestration
  1. Rare domains in cloud logs can indicate malicious activities like command and control communication, phishing, or data exfiltration.
  2. Microsoft Sentinel offers a built-in hunting query to identify rare domains and mitigate potential security incidents.
  3. By using automation and orchestration in Microsoft Sentinel, organizations can efficiently respond to and manage incidents related to rare domains in cloud logs.

How We Built The Tech That Powers Our Serverless Cloud

Bismuth • 2 HN points • 20 Jun 24
🕹 Technology Cloud Computing Orchestration Containerization Platform Development
  1. The Bismuth Cloud platform enables quick deployment of applications using a homegrown system with a container orchestrator.
  2. Their approach offers more control and flexibility compared to existing solutions like Kubernetes or open source FaaS platforms.
  3. They use a unique service provider approach for each container to enhance reliability and simplify authentication.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Orchestration isn’t going anywhere

Data People Etc. • 36 HN points • 24 Apr 23
🕹 Technology Data Management Data Pipelines Orchestration Software Engineering Organizational dynamics
  1. Orchestration is essential and will continue to be important in the future of managing data pipelines.
  2. Orchestration involves coordinating and managing multiple systems and tasks to execute workflows.
  3. Tools like Dagster provide a control plane for managing data assets and metadata, ensuring a structured and cohesive data platform.