Data People Etc.

Data People Etc. explores the complex interplay between data management, organizational culture, and technology's impact on society. It discusses metadata, data engineering efficiency, the philosophical aspects of the internet, and the role of data professionals in shaping organizational truth and efficiency through insights, product sketches, and industry advancements.

Data Management Organizational Culture Technology and Society Metadata Data Engineering Internet Philosophy Product Development Career Development

The hottest Substack posts of Data People Etc.

And their main takeaways

How to destroy a world

53 implied HN points • 24 Feb 25
🕹 Technology AI Data science Software Systems Influencers
  1. Frameworks can be used for both building and breaking worlds. It's important to understand how to exploit weaknesses in these structures.
  2. To weaken a dominant system, you can undermine its narrative, disrupt key players, and challenge established norms. This approach can create doubts and resistance.
  3. Destroying a world can teach us about resilience. Strengthening systems and protocols is crucial to support and maintain their relevance in changing times.

Strong worlds, weak worlds, SaaS worlds, link worlds

71 implied HN points • 19 Feb 25
🕹 Technology Digital Culture Virtual Worlds Worldbuilding
  1. Strong worlds offer structure and meaning, helping people feel like they belong and know what's expected of them. This creates a sense of purpose and connection.
  2. The Internet helps create many weak worlds where it's easy to join or leave. These worlds often lack strong narratives but can have a big impact on our daily lives.
  3. Platforms are tools for building worlds online, but strong systems matter more than just good stories. Companies must ensure their systems work well to keep their teams functioning smoothly.

Worldbuilding with data

231 implied HN points • 11 Feb 25
🕹 Technology Data Engineering Data science Big Data Data Visualization Analytics
  1. Data is more powerful when it has a purpose. It should tell a clear story, otherwise it's just clutter.
  2. Building a strong data system is like creating a world. A good structure connects different pieces and helps everyone understand the bigger picture.
  3. Data engineering is important because it helps manage and present large amounts of information, making sure everything works smoothly and accurately.

Data's final format

391 implied HN points • 09 Dec 24
🕹 Technology Data Management Cloud Computing Software Development Tech Innovation
  1. Apache Iceberg™ is a popular way to manage data, offering features like scalability and openness. However, using it can feel complicated and less exciting than expected.
  2. CSV format is an easy and humble way to manage data, requiring no special knowledge or complex setups. It’s simple and widely understood, making it a go-to choice for many.
  3. The transformation of data management, like Iceberg™, is like building a transcontinental railroad. It's a huge effort aimed at improving the way we process and use information in the modern world.

Discussion: The AI Era Liberal Arts

71 implied HN points • 13 Jan 25
🚌 Education Liberal Arts Curriculum design Skill development Critical Thinking
  1. We might need a new set of subjects for students today. The old tools aren't enough for the modern world that we live in.
  2. The seven liberal arts today should focus on skills like logic, statistics, and understanding psychology. These help us make sense of our world and make better decisions.
  3. It's more important to prepare kids to be adaptable and motivated than to worry about specific job skills. Being able to learn and think critically is key for the future.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

The Three Jobs

302 implied HN points • 28 Oct 24
  1. Being a parent is like having a job with specific duties, such as keeping kids safe, helping them grow, and bringing joy. It's not just about the relationship, but about actions and decisions.
  2. Teaching kids about safety and respect helps them understand the world around them. These lessons are important for their survival and social interactions.
  3. Joy and laughter are essential parts of parenting. While there's a lot to teach and guide, having fun together creates strong, lasting memories.

5 Great Books from 2024

35 implied HN points • 08 Jan 25
📚 Literature Book Reviews Fiction Non-fiction Reading
  1. Reading a mix of books can provide new ideas and insights, making it enjoyable to explore different genres and styles.
  2. Some books can be very challenging or not as enjoyable, but it's important to keep trying new things and learn from those experiences.
  3. Books like 'Endurance' show the strength of the human spirit and remind us of the importance of persistence, even in tough situations.

Ant farms and PEER Protocols

53 implied HN points • 17 Dec 24
🕹 Technology AI Protocols Interfaces Privacy Economy
  1. The PEER protocol is all about making sure that AI assistants are safe and respect our privacy. They should only act on our permission, keep our personal info secure, and even be stored directly on our devices.
  2. AI agents, referred to as 'ants', represent a collective intelligence instead of individual personalities. They're designed to work tirelessly and learn about our preferences to provide better service.
  3. Removing screens from our interactions with technology may reduce information overload, but it also raises trust issues. Users need to believe that their AI assistants will share only what's essential, without important details going missing.

Get the weekly rites right

71 implied HN points • 25 Nov 24
  1. Weekly activities are important for building strong community relationships. Regularly going to the library, for instance, can turn a simple trip into a valuable friendship.
  2. Consistency matters more than talent. For children, being a 'regular' at a place like the library helps them create connections and receive support.
  3. Investing time weekly with family and friends can enhance relationships. These repeated activities create joy and harmony in life.

Decision Tree

17 implied HN points • 23 Dec 24
📚 Literature Books Reading Education Homeschooling Philosophy
  1. When choosing gifts for kids, it's helpful to have a simple plan. For example, giving kids something they want, need, wear, and read can keep things organized and manageable.
  2. Books can be tricky to categorize. Often, one book leads to another which makes it hard to choose just one thing to read.
  3. Buying books for kids can also reflect personal interests. It's okay to want to share your favorite books even if they are a little nostalgic.

Below the API

17 implied HN points • 19 Nov 24
⚔ Fiction Short Story Literature Speculative Fiction Creative Writing Contemporary Issues
  1. Marcie is trying to make art through pottery, but faces pressures from her brother and struggles with self-doubt about her work's value. She finds joy and control in creating but feels judged.
  2. Unexpectedly, a surge of orders comes in for her pottery, but it turns out to be driven by her brother's manipulation and automation. This raises questions for her about the nature of her art and who actually wants it.
  3. In the end, Marcie realizes she must take control back from external influences and prioritize genuine creativity over making money. She decides to focus on her true passions rather than just chasing trends.

Book Notes: The Internet is not what you think it is

302 implied HN points • 27 Sep 23
📚 Literature Internet
  1. The Internet is not just a new technology, but a continuation of humanity's quest to connect across time and space.
  2. The book warns of new exploitative aspects of the internet, including the exploitation of human lives as resources and the impact on attention and perception.
  3. To resist the dehumanizing aspects of the internet, it's important to limit consumption, reclaim attention, and remember our individuality amidst the data-driven world.

James and the Giant Pitch

106 implied HN points • 08 Mar 24
📚 Literature Fiction Friendship Loneliness Adventure Emotions
  1. Founding a company is about more than just getting rich; it can be driven by a deeper desire for meaning and friendship.
  2. The story of James and the Giant Peach suggests that entrepreneurs may be seeking to fill emotional voids through their ventures.
  3. Loneliness and personal pain may be key motivators for individuals to embark on entrepreneurial journeys.

28 Dags Later

302 implied HN points • 18 Apr 23
🕹 Technology Data Orchestration Cloud Infrastructure Tooling
  1. Modern systems are like a chaotic wasteland filled with numerous services that dehumanize everything.
  2. The role of an orchestrator is evolving to become flatter, more activity-driven, and adaptable to chaos.
  3. Tools and orchestrators should prioritize simplicity, speed, and individual usage to navigate the services hell effectively.

Data, Despair, and the FLOWBEE

266 implied HN points • 13 Mar 23
💼 Business Data Metrics Organization Intelligence Thinking
  1. Data professionals may feel isolated due to externalized intelligence and lack of integration into daily activities.
  2. Thinkers in organizations may become untethered without proper recognition and integration with doers.
  3. To be effective, thinkers must be tightly integrated into their environment and endorsed by leadership.

Data in a Downturn

231 implied HN points • 20 Mar 23
🕹 Technology Data Analysis Tech Tools Data Teams Data Stack
  1. Data teams are facing challenges with tool abandonment in the current economy.
  2. Databases remain crucial in the data stack, with less need for new, specialized tools.
  3. Building trust and bridging gaps between data and engineering teams is vital for successful data applications.

On Sight and Insight

248 implied HN points • 06 Feb 23
  1. Sight alone is not enough, we need insight to truly understand.
  2. Reading is not just about knowledge, it's a control of focus and attention.
  3. Insight must come from within an organization, it can't be outsourced.

Data Materialization is a Convergence Problem

159 implied HN points • 10 Apr 23
🕹 Technology Data Workflow Operations
  1. Data materialization is not just a workflow orchestration problem but also a convergence problem.
  2. In a convergence-based approach to data materialization, a materialization controller could continuously compare the state of the warehouse with the desired state of models to automate the materialization process.
  3. Challenges in implementing a materialization controller include explainability, managing over-eagerness, and dealing with drift in the system.

In Search of the Übergraph

213 implied HN points • 29 Nov 22
  1. Data problems are fundamentally about knowledge transfer and can be formalized as graphs.
  2. The Übergraph symbolizes the interconnected web of knowledge that extends beyond organizational boundaries.
  3. Managing the Übergraph requires prioritizing data quality, responsibility, and acknowledging the natural tendency towards entropy.

Product Sketch: Segment for Metadata

373 implied HN points • 25 Jan 22
  1. Defrag platform simplifies handling scattered data platform metadata.
  2. Defrag product consists of three main components - metadata extractors, dynamic data source map, and versatile export functionality.
  3. Defrag aims to make metadata accessible and valuable, focusing on enhancing data management and user applications.

Life After Orchestrators

142 implied HN points • 06 Apr 23
🕹 Technology Data Operations Data Orchestration
  1. Orchestrators can be time killers in data operations, focusing on managing tasks rather than letting data drive operations.
  2. Legacy needs drove the creation of orchestrators to manage complex logic dependencies in data operations.
  3. Post-orchestrator approaches like high-frequency batches and asynchronous processing are gaining popularity for more efficient data operations.

Symposium: Is the orchestrator dead or alive?

159 implied HN points • 13 Feb 23
🕹 Technology Writing Philosophy
  1. A symposium is being hosted to explore the question of whether the orchestrator is still relevant in 2023.
  2. The symposium encourages submissions of essays with a single, unique idea presented in argumentative and spirited manners.
  3. The submission process allows for both direct publication on Data People Etc. or cross-posting from personal Substack accounts.

Limits of the Event-Driven Orchestrator

106 implied HN points • 03 Apr 23
🕹 Technology Data processing Workflow Orchestration Stream Processing
  1. Event-driven orchestrators are not suitable for stream processing because they cannot handle tasks with definite starts and ends.
  2. Event-driven applications operate asynchronously by triggering tasks based on events like files appearing in a directory.
  3. Unlike stream processors, orchestrators like Airflow and Dagster do not have the ability to hold state, distribute tasks for parallel execution, or shuffle data between tasks.

Wander Well

231 implied HN points • 14 Mar 22
  1. Learning can involve climbing towards a goal or wandering without a set destination.
  2. Wandering through experiences can provide valuable and immersive learning opportunities.
  3. Exploratory data analysis and dogfooding can be powerful tools for learning and understanding in the data professional field.

April, Etc.

88 implied HN points • 01 May 23
🕹 Technology Data Orchestration
  1. The symposium didn't reach a consensus but celebrated chaos, boredom, and laziness.
  2. Orchestrators are essential but face challenges such as being perceived as boring and needing to evolve to handle event-driven workloads.
  3. Substack Notes can be used to share in-progress writing for feedback and to give discarded ideas new life.

The Optimetricist

159 implied HN points • 27 Jun 22
  1. Metrics are simplifications of reality, but they can be powerful proxies.
  2. Optimetricists find meaning in numbers and believe they represent the world accurately.
  3. Metrics can simplify complex realities or create new worlds for teams to organize within.

Will active metadata eat the orchestrator?

88 implied HN points • 27 Mar 23
🕹 Technology Data Management Metadata Data Governance Collaboration Innovation
  1. Active metadata is a dynamic way to manage and use metadata across different parts of the data stack.
  2. Active metadata can potentially replace triggering mechanism aspect of data orchestrators, but not the optimization intelligence.
  3. The true value of active metadata lies in empowering business users by acting as a personal data assistant.

Perennial Truth Architectures

159 implied HN points • 01 Mar 22
  1. Organizations' systems reflect their communication structures.
  2. Establishing a 'Truth Architecture' helps align collective action.
  3. Data professionals can play a key role in shaping the truth architecture of their organization.

Kicking the tires on dbt Metrics

159 implied HN points • 21 Feb 22
  1. The dbt metrics layer has arrived and is expected to bring new life in the next 40 weeks.
  2. The metrics layer streamlines data processing by consolidating business logic on the database side.
  3. The metrics layer is an open-source backend for BI tools, simplifying SQL generation and offering new possibilities for analysis.

Product Sketch: The New Corp Times

142 implied HN points • 04 Apr 22
  1. Period is a tech venture that simplifies communication within remote-first companies.
  2. Editors play a crucial role in creating a shared community narrative through Period publications.
  3. Lore, as shared subjective experiences, is a powerful tool in building organizational culture and understanding.

Product Sketch: Airflow, The Game

124 implied HN points • 09 May 22
  1. Transform your data engineering tasks into a strategic video game with OrchCraft
  2. Simulate platform battles and analyze vulnerabilities through the Battle Engine feature
  3. Access additional campaign missions to master data challenges and climb the global leaderboard in OrchCraft

March, Etc.

53 implied HN points • 15 Mar 23
🕹 Technology Data Modeling Data Management Data Quality
  1. Intermediate data modeling can be valuable following Kimball design principles.
  2. Attending events like Data Council can provide insights and networking opportunities.
  3. Engaging in ongoing discussions and being part of a community can enhance the writing and learning experience.

Data Is The New Motor Oil

106 implied HN points • 01 Feb 22
  1. Data teams can have a significant impact beyond just creating dashboards.
  2. A common thread in all data careers is the need for speed in pushing the company forward.
  3. Investing in a data team can help companies navigate changes, connect old and new systems, and focus on moving towards the future.

Orchestration isn’t going anywhere

36 HN points • 24 Apr 23
🕹 Technology Data Management Data Pipelines Orchestration Software Engineering Organizational dynamics
  1. Orchestration is essential and will continue to be important in the future of managing data pipelines.
  2. Orchestration involves coordinating and managing multiple systems and tasks to execute workflows.
  3. Tools like Dagster provide a control plane for managing data assets and metadata, ensuring a structured and cohesive data platform.

Comment on 'Tuning Whatnot's Data Platform for Speed and Scale'

71 implied HN points • 06 May 22
  1. Start with understanding the problem before jumping to a solution
  2. Prioritize high-impact work over small projects to make a significant difference
  3. Iterating on top of a problem can lead to valuable improvements in data platforms