The hottest Data Teams Substack posts right now

Web scraping with AI can enhance intelligence gathering by efficiently collecting and processing data from various public sources on the internet.
Leveraging Large Language Models (LLMs) can improve the accuracy and robustness of web scraping systems when dealing with changes in HTML code structure.
Using tools like Nimble for web scraping allows for more efficient and accurate data collection by training models on different types of websites for specific use cases.

Setting up your data team is crucial for success in the field.
Data teams can be organized in centralized, decentralized, or federated ways.
Consider different organizational structures to optimize your data team's efficiency.

Analysts should reflect on their role and avoid behaving like Jared Kushner
Being a data analyst involves providing informed insights, not just being a 'nicer, kinder' Jared Kushner
Focusing on keeping the company well-informed through regular updates can be more effective than traditional data reporting

Data quality is a tradeoff in balancing stability and agility.
Data resiliency tools like SDF focus on tracing data lineage to improve debugging and fixing issues.
Managing messy data often requires making choices between stability and adaptability in data infrastructure.

Get a weekly roundup of the best Substack posts, by hacker news affinity:

Data teams are increasingly turning to low-code solutions to streamline data release pipelines, utilizing tools like Airflow but questioning the need for extensive code writing and infrastructure maintenance.
The complex cloud environment has led to the development of specialized data tools, making the orchestration of data pipelines challenging and highlighting the importance of governance, data quality, and scalability.
No-code solutions like dbt core and Hightouch are already integrated into many data tools, simplifying the orchestration process and indicating that the future of data architecture might involve a combination of workflow orchestrators and efficient data quality checks.

Don't focus too much on best practices and what other companies are doing in data.
Deal with challenges and adversity in data, but focus on doing the right thing.
Prioritize data quality when your company truly becomes data-driven.

Data teams are facing challenges with tool abandonment in the current economy.
Databases remain crucial in the data stack, with less need for new, specialized tools.
Building trust and bridging gaps between data and engineering teams is vital for successful data applications.

Consider data contracts if your internal data changes often to ensure collaboration between software engineering and data engineering teams.
If you have important metrics that depend on software engineering actions, like defining 'Active Users,' data contracts can help maintain data quality.
In cases where software engineering and data engineering roles overlap, implementing data contracts can streamline data ingestion processes and improve data quality.

Your data team size should be about 1-5% of your total company staff. Fintech companies usually have a higher percentage of data roles.
The mix of different data roles is important. Having too many analysts can slow things down, while too many engineers might not deliver useful insights.
Data salaries in Europe vary by experience. For example, a junior data role typically pays about $70k, while senior roles can reach $110k or more.

Dbt Labs acquired Transform to target a new market segment beyond analytics engineers.
Tech companies typically expand by starting small, then broadening their market focus and adding features.
Data is not the same as analytics; a top-down approach to making data vital in a company is crucial.