Data pipeline orchestrators - the emerging force in the MDS? timo's substack • 196 implied HN points • 18 Oct 23 🕹 Technology Data Orchestration Data Platforms Open Source Control of data flow is crucial in data platforms Data pipeline orchestrators help in managing data transformations Orchestrators are becoming essential tools in modern data stack evolution
28 Dags Later Data People Etc. • 302 implied HN points • 18 Apr 23 🕹 Technology Data Orchestration Cloud Infrastructure Tooling Modern systems are like a chaotic wasteland filled with numerous services that dehumanize everything. The role of an orchestrator is evolving to become flatter, more activity-driven, and adaptable to chaos. Tools and orchestrators should prioritize simplicity, speed, and individual usage to navigate the services hell effectively.
Life After Orchestrators Data People Etc. • 142 implied HN points • 06 Apr 23 🕹 Technology Data Orchestration Orchestrators can be time killers in data operations, focusing on managing tasks rather than letting data drive operations. Legacy needs drove the creation of orchestrators to manage complex logic dependencies in data operations. Post-orchestrator approaches like high-frequency batches and asynchronous processing are gaining popularity for more efficient data operations.
April, Etc. Data People Etc. • 88 implied HN points • 01 May 23 🕹 Technology Data Orchestration The symposium didn't reach a consensus but celebrated chaos, boredom, and laziness. Orchestrators are essential but face challenges such as being perceived as boring and needing to evolve to handle event-driven workloads. Substack Notes can be used to share in-progress writing for feedback and to give discarded ideas new life.
What Suboptimal Public Transit and Your Data Pipelines Have in Common Bytes, Data, Action! • 19 implied HN points • 05 Sep 23 🕹 Technology Data Pipelines Public Transit Monitoring Data Orchestration Data Management Public transit and data pipelines both aim to move things from point A to point B smoothly and quickly. Issues like delays, lack of visibility, and missed connections can disrupt the experiences of both public transit and data pipelines. Efficient, transparent, and reliable practices are key to ensuring a smooth journey for both public transit users and data pipelines.