Data People Etc.

Data People Etc. explores the complex interplay between data management, organizational culture, and technology's impact on society. It discusses metadata, data engineering efficiency, the philosophical aspects of the internet, and the role of data professionals in shaping organizational truth and efficiency through insights, product sketches, and industry advancements.

Data Management Organizational Culture Technology and Society Metadata Data Engineering Internet Philosophy Product Development Career Development

The hottest Substack posts of Data People Etc.

And their main takeaways

James and the Giant Pitch

106 implied HN points • 08 Mar 24
📚 Literature Fiction Friendship Loneliness Adventure Emotions
  1. Founding a company is about more than just getting rich; it can be driven by a deeper desire for meaning and friendship.
  2. The story of James and the Giant Peach suggests that entrepreneurs may be seeking to fill emotional voids through their ventures.
  3. Loneliness and personal pain may be key motivators for individuals to embark on entrepreneurial journeys.

Book Notes: The Internet is not what you think it is

302 implied HN points • 27 Sep 23
📚 Literature Internet
  1. The Internet is not just a new technology, but a continuation of humanity's quest to connect across time and space.
  2. The book warns of new exploitative aspects of the internet, including the exploitation of human lives as resources and the impact on attention and perception.
  3. To resist the dehumanizing aspects of the internet, it's important to limit consumption, reclaim attention, and remember our individuality amidst the data-driven world.

28 Dags Later

302 implied HN points • 18 Apr 23
🕹 Technology Data Orchestration Cloud Infrastructure Tooling
  1. Modern systems are like a chaotic wasteland filled with numerous services that dehumanize everything.
  2. The role of an orchestrator is evolving to become flatter, more activity-driven, and adaptable to chaos.
  3. Tools and orchestrators should prioritize simplicity, speed, and individual usage to navigate the services hell effectively.

Data, Despair, and the FLOWBEE

266 implied HN points • 13 Mar 23
💼 Business Data Metrics Organization Intelligence Thinking
  1. Data professionals may feel isolated due to externalized intelligence and lack of integration into daily activities.
  2. Thinkers in organizations may become untethered without proper recognition and integration with doers.
  3. To be effective, thinkers must be tightly integrated into their environment and endorsed by leadership.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Data in a Downturn

231 implied HN points • 20 Mar 23
🕹 Technology Data Analysis Tech Tools Data Teams Data Stack
  1. Data teams are facing challenges with tool abandonment in the current economy.
  2. Databases remain crucial in the data stack, with less need for new, specialized tools.
  3. Building trust and bridging gaps between data and engineering teams is vital for successful data applications.

On Sight and Insight

248 implied HN points • 06 Feb 23
  1. Sight alone is not enough, we need insight to truly understand.
  2. Reading is not just about knowledge, it's a control of focus and attention.
  3. Insight must come from within an organization, it can't be outsourced.

Data Materialization is a Convergence Problem

159 implied HN points • 10 Apr 23
🕹 Technology Data Workflow Operations
  1. Data materialization is not just a workflow orchestration problem but also a convergence problem.
  2. In a convergence-based approach to data materialization, a materialization controller could continuously compare the state of the warehouse with the desired state of models to automate the materialization process.
  3. Challenges in implementing a materialization controller include explainability, managing over-eagerness, and dealing with drift in the system.

Life After Orchestrators

142 implied HN points • 06 Apr 23
🕹 Technology Data Orchestration
  1. Orchestrators can be time killers in data operations, focusing on managing tasks rather than letting data drive operations.
  2. Legacy needs drove the creation of orchestrators to manage complex logic dependencies in data operations.
  3. Post-orchestrator approaches like high-frequency batches and asynchronous processing are gaining popularity for more efficient data operations.

Symposium: Is the orchestrator dead or alive?

159 implied HN points • 13 Feb 23
🕹 Technology Writing Philosophy
  1. A symposium is being hosted to explore the question of whether the orchestrator is still relevant in 2023.
  2. The symposium encourages submissions of essays with a single, unique idea presented in argumentative and spirited manners.
  3. The submission process allows for both direct publication on Data People Etc. or cross-posting from personal Substack accounts.

In Search of the Übergraph

213 implied HN points • 29 Nov 22
  1. Data problems are fundamentally about knowledge transfer and can be formalized as graphs.
  2. The Übergraph symbolizes the interconnected web of knowledge that extends beyond organizational boundaries.
  3. Managing the Übergraph requires prioritizing data quality, responsibility, and acknowledging the natural tendency towards entropy.

Limits of the Event-Driven Orchestrator

106 implied HN points • 03 Apr 23
🕹 Technology Data processing Stream Processing
  1. Event-driven orchestrators are not suitable for stream processing because they cannot handle tasks with definite starts and ends.
  2. Event-driven applications operate asynchronously by triggering tasks based on events like files appearing in a directory.
  3. Unlike stream processors, orchestrators like Airflow and Dagster do not have the ability to hold state, distribute tasks for parallel execution, or shuffle data between tasks.

Product Sketch: Segment for Metadata

373 implied HN points • 25 Jan 22
  1. Defrag platform simplifies handling scattered data platform metadata.
  2. Defrag product consists of three main components - metadata extractors, dynamic data source map, and versatile export functionality.
  3. Defrag aims to make metadata accessible and valuable, focusing on enhancing data management and user applications.

April, Etc.

88 implied HN points • 01 May 23
🕹 Technology Data Orchestration
  1. The symposium didn't reach a consensus but celebrated chaos, boredom, and laziness.
  2. Orchestrators are essential but face challenges such as being perceived as boring and needing to evolve to handle event-driven workloads.
  3. Substack Notes can be used to share in-progress writing for feedback and to give discarded ideas new life.

Wander Well

231 implied HN points • 14 Mar 22
  1. Learning can involve climbing towards a goal or wandering without a set destination.
  2. Wandering through experiences can provide valuable and immersive learning opportunities.
  3. Exploratory data analysis and dogfooding can be powerful tools for learning and understanding in the data professional field.

The Optimetricist

159 implied HN points • 27 Jun 22
  1. Metrics are simplifications of reality, but they can be powerful proxies.
  2. Optimetricists find meaning in numbers and believe they represent the world accurately.
  3. Metrics can simplify complex realities or create new worlds for teams to organize within.

March, Etc.

53 implied HN points • 15 Mar 23
🕹 Technology Data Modeling Data Management Data Quality
  1. Intermediate data modeling can be valuable following Kimball design principles.
  2. Attending events like Data Council can provide insights and networking opportunities.
  3. Engaging in ongoing discussions and being part of a community can enhance the writing and learning experience.

Perennial Truth Architectures

159 implied HN points • 01 Mar 22
  1. Organizations' systems reflect their communication structures.
  2. Establishing a 'Truth Architecture' helps align collective action.
  3. Data professionals can play a key role in shaping the truth architecture of their organization.

Kicking the tires on dbt Metrics

159 implied HN points • 21 Feb 22
  1. The dbt metrics layer has arrived and is expected to bring new life in the next 40 weeks.
  2. The metrics layer streamlines data processing by consolidating business logic on the database side.
  3. The metrics layer is an open-source backend for BI tools, simplifying SQL generation and offering new possibilities for analysis.

Product Sketch: The New Corp Times

142 implied HN points • 04 Apr 22
  1. Period is a tech venture that simplifies communication within remote-first companies.
  2. Editors play a crucial role in creating a shared community narrative through Period publications.
  3. Lore, as shared subjective experiences, is a powerful tool in building organizational culture and understanding.

Product Sketch: Airflow, The Game

124 implied HN points • 09 May 22
  1. Transform your data engineering tasks into a strategic video game with OrchCraft
  2. Simulate platform battles and analyze vulnerabilities through the Battle Engine feature
  3. Access additional campaign missions to master data challenges and climb the global leaderboard in OrchCraft

Orchestration isn’t going anywhere

36 HN points • 24 Apr 23
🕹 Technology Data Management Data Pipelines Orchestration Software Engineering Organizational dynamics
  1. Orchestration is essential and will continue to be important in the future of managing data pipelines.
  2. Orchestration involves coordinating and managing multiple systems and tasks to execute workflows.
  3. Tools like Dagster provide a control plane for managing data assets and metadata, ensuring a structured and cohesive data platform.

Data Is The New Motor Oil

106 implied HN points • 01 Feb 22
  1. Data teams can have a significant impact beyond just creating dashboards.
  2. A common thread in all data careers is the need for speed in pushing the company forward.
  3. Investing in a data team can help companies navigate changes, connect old and new systems, and focus on moving towards the future.

Comment on 'Tuning Whatnot's Data Platform for Speed and Scale'

71 implied HN points • 06 May 22
  1. Start with understanding the problem before jumping to a solution
  2. Prioritize high-impact work over small projects to make a significant difference
  3. Iterating on top of a problem can lead to valuable improvements in data platforms

How to Feed a Data Professional

71 implied HN points • 08 Feb 22
  1. Snacktime is an experience full of possibilities, different from mealtime's mundane reality.
  2. Everyone loves a good data snack, but a shallow snacktime data culture is limiting for growth.
  3. Data professionals need a balanced diet of tasks that connect them to the bigger picture, not just data snacks.

Welcome to Data People Etc.

35 implied HN points • 31 Dec 21
  1. The newsletter focuses on data-related topics but also covers related subjects like people and literature.
  2. Stephen Bailey, a data professional, writes the newsletter.
  3. To subscribe to the newsletter, click below!