The hottest System Design Substack posts right now

And their main takeaways
Category
Top Technology Topics

Understanding Goodhart's Law [Math Mondays]

Technology Made Simple • 179 implied HN points • 11 Mar 24
🕹 Technology Computer Science Math System Design AI Tech industry
  1. Goodhart's Law warns that when a measure becomes a target, it can lose its effectiveness as a measure.
  2. The law often unfolds due to complications in real-world systems, human adaptability, and evolutionary pressures.
  3. To address Goodhart's Law, consider using multiple metrics, tying metrics to ultimate goals, and being prepared to adapt metrics as needed.

Understanding Space Based Architecture for efficient Data Processing[System Design Sundays]

Technology Made Simple • 379 implied HN points • 12 Feb 24
🕹 Technology Data processing System Design AI Architecture
  1. Space-Based Architecture (SBA) distributes processing and storage across multiple servers, enhancing scalability and performance by leveraging in-memory data grids.
  2. The components of SBA include Processing Units (PU) for executing business logic, Virtualized Middleware for managing shared infrastructure, and data pumps for data marshaling.
  3. SBA offers benefits such as scalability, fault tolerance, and low-latency data access, but comes with challenges like complexity in design, debugging, and data security.

Building a Simple Live Streaming Platform[System Design Sundays]

Technology Made Simple • 119 implied HN points • 18 Mar 24
🕹 Technology System Design Platforms Coding Software Engineering
  1. When designing a live streaming platform like Twitch, key steps include ingestion, transcoding, packaging, CDN utilization, and database management.
  2. Challenges like low latency, scalability, and reliability must be addressed for the success of a live streaming platform.
  3. To enhance a streaming service further, consider advanced technologies like adaptive bitrate algorithms, advanced caching, and community features.

How I Taught Myself to Critique AI Research [Storytime Saturdays]

Technology Made Simple • 119 implied HN points • 10 Mar 24
🕹 Technology AI Research Writing Machine Learning System Design
  1. Writing allows you to store knowledge for future reference, spot cognitive blindspots, and engage with topics more deeply for better understanding.
  2. Challenges in self-learning writing include lack of contextual understanding, a defined learning path, and a peer network for feedback.
  3. Addressing challenges in self-learning involves finding strategies to gain clarity, identifying knowledge gaps, and seeking feedback from peers.

[System design] Sharding : In and Out Explained

The ZenMode • 42 implied HN points • 16 Mar 24
🕹 Technology System Design Scaling Optimization
  1. Sharding is a technique to horizontally partition a data store into smaller fragments across multiple servers, aiding in scalability and reliability.
  2. Before sharding a database, consider options like vertical partitioning, database optimization, replication, and caching to improve performance without the added complexity of sharding.
  3. Different sharding strategies like Hash Sharding, Range Sharding, and Directory-Based Sharding have unique considerations and advantages based on factors like data distribution, queries, and maintenance.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

Single Tenant vs Multi Tenant Architecture for SAAS Platforms [System Design Sundays]

Technology Made Simple • 139 implied HN points • 04 Dec 23
🕹 Technology System Design Software Architecture Cloud Services Data Systems
  1. Single Tenant Architecture provides each customer their own independent database and software instance, offering security and customization like living in a detached house.
  2. Multi-Tenant Architecture is akin to an apartment building where multiple tenants share common infrastructure, allowing for economies of scale but potentially limiting customization.
  3. Single Tenant architecture is known for high user engagement, control, and stability, while Multi-Tenant architecture favors compliance, security, and quick onboarding for better scalability.

SwirlAI Table of Contents

SwirlAI Newsletter • 432 implied HN points • 28 Jun 23
🕹 Technology Data Engineering MLOps System Design Systems Thinking
  1. The newsletter provides a Table of Contents with more than 90 topics, making it easier to find the content of interest.
  2. Topics covered include Data Engineering fundamentals, Spark architecture, Kafka use cases, MLOps deployment processes, System Design examples, and more.
  3. If interested, it's recommended to support the author's work by subscribing and sharing the content.

Optimising costs to enhancing UX - it is all engineering

system bashing • 117 implied HN points • 18 Jul 23
🕹 Technology Cost Optimization User Experience Cloud Services Engineering System Design
  1. In a tech company, engineering involves balancing cloud costs and user interface to optimize costs and enhance user experience.
  2. Reducing costs significantly is crucial for a company's profitability regardless of other measures like discounts or marketing strategies.
  3. Engineering decisions impact user experience constraints and cloud costs, requiring a balance between the two for system efficiency.

Amazon Prime Video reduced costs by 90% by ditching Microservices [System Design Sundays]

Technology Made Simple • 159 implied HN points • 07 May 23
🕹 Technology System Design Microservices Architecture Scaling
  1. Amazon Prime Video saw a 90% cost reduction by moving away from Microservices to a monolith architecture. This change improved scalability and reduced infrastructure costs significantly.
  2. The challenges Amazon faced with their initial microservices implementation included hitting scaling limits and high overall costs of the system. Moving to a monolith architecture helped address these issues and allowed for better scaling.
  3. While the debate between Microservices and Monoliths continues, the decision should depend on factors like team size, emphasis on scale, and complexity. Microservices offer scalability but require careful planning, while monoliths are easier to design and manage.

Google Cloud's Core Principles of System Design [System Design sundays]

Technology Made Simple • 59 implied HN points • 04 Sep 23
🕹 Technology System Design Cloud Computing Documentation Architecture
  1. A robust system design should be secure, reliable, scalable, and independent, allowing for iterative changes without disruption.
  2. Document everything to help visualize deployments, collaborate effectively, and guide future design decisions.
  3. Simplify system design, use fully managed services, decouple architecture, and strive for a stateless architecture to improve reliability and scalability.

System Design of Uber's CacheFront - Serving more than 40 million low-latency reads per second

Engineering At Scale • 4 HN points • 03 Mar 24
🕹 Technology System Design Caching Databases Scalability Integrations
  1. Uber developed CacheFront, an integrated caching solution to overcome problems like maintenance overhead, reduced developer productivity, and region failovers caused by using Redis for caching
  2. Docstore's architecture includes a Control plane, Query Engine, and Storage Engine, with relevant responsibilities for each layer like query execution, data persistence, transaction management, and more
  3. CacheFront's design addressed non-functional requirements like consistency guarantees, cache warming & region failovers, fault tolerance, hot partition issues, and performance & cost improvements

3 quick tips on how to build more scalable systems [System Design Sundays]

Technology Made Simple • 119 implied HN points • 17 Apr 23
🕹 Technology System Design Scalability Engineering Software Architecture
  1. Location matters: Place software close to clients for faster response times using CDNs, edge computing, or geo-replication.
  2. Cache wisely: Optimize speed by using in-memory caching, database caching, or web caching to avoid repeated actions.
  3. Async is key: Improve efficiency with asynchronous processing through message queues, event-driven architectures, or microservices.

Why you should design systems that ship fast [System Design Sundays]

Technology Made Simple • 79 implied HN points • 15 May 23
🕹 Technology System Design Software Development Programming
  1. Shipping software quickly has many benefits, including improved efficiency and cost savings.
  2. A slow release process can lead to increased expenses as more resources are required for tasks like testing, outage management, and manual deploys.
  3. Investing in designing systems that support fast shipping is valuable as it helps build a culture of efficiency and productivity.

Why Discord ditched Cassandra [System Design Sundays]

Technology Made Simple • 79 implied HN points • 03 Apr 23
🕹 Technology System Design Databases Data Management Programming Tech Education
  1. Discord faced performance issues with Cassandra, requiring increasing maintenance effort and leading to unpredictable latency.
  2. Hot partitions were a problem in Cassandra, causing hotspotting and impacting the database's performance during concurrent reads.
  3. Garbage collection in Cassandra posed challenges, leading Discord to switch to ScyllaDB which does not have a garbage collector.

How to ARCHITECT a search engine like Google Search[System Design Sundays]

Technology Made Simple • 99 implied HN points • 29 Jan 23
🕹 Technology System Design Machine Learning Search Engine Software Engineering
  1. Design complex systems by layering multiple smaller solutions for better results instead of focusing on individually engineered tasks.
  2. Building a search engine like Google involves accommodating various types of search results like images, text, gifs, and videos while ensuring search quality.
  3. Handling the massive scale of data in Google's search engine system involves using semi-supervised labeling techniques to manage unlabeled data efficiently.

How bad is Architectural Complexity [System Design Sundays]

Technology Made Simple • 79 implied HN points • 06 Mar 23
🕹 Technology System Design Developer Productivity
  1. Complex architectures can significantly impact developer productivity, software quality, and turnover, with potential for 50% drops in productivity and significant increases in defect density and staff turnover.
  2. Architectural complexity can lead to increased defect density in codebases, higher time consumption, and a higher probability of developers leaving the firm.
  3. Complexity can breed more complexity, creating a cycle that hampers future system developments.

How Khan Academy Successfully Handled 2.5x Traffic in a Week[System Design Sundays]

Technology Made Simple • 79 implied HN points • 14 Nov 22
🕹 Technology System Design
  1. Combining common ideas can lead to great results. The fundamentals of utilizing serverless architecture and CDNs like Google Cloud and Fastly CDN were key to Khan Academy handling increased traffic.
  2. CDNs are important for scalability. They consist of servers distributed worldwide, enabling faster user interactions by caching content and optimizing server resources.
  3. Serverless architecture provides scalability and performance. By hosting applications on external servers, like those at Khan Academy, the system handled increased traffic efficiently without manual intervention.

Design a Tic-Tac-Toe Game [with Code]

The ZenMode • 42 implied HN points • 15 Feb 23
🕹 Technology Design Programming System Design Game Development Web Development
  1. Designing a Tic-Tac-Toe game involves creating a user interface with components like a game board, player information, game status, and start/restart buttons.
  2. Implementing game logic requires components like game state, move validation, win condition detection, tie condition detection, and player management.
  3. For scalability, maintainability, and extensibility, consider using components like load balancer, cloud-based infrastructure, modular architecture, clean code, and automated testing.

How to design a Database Management System [System Design Sundays]

Technology Made Simple • 39 implied HN points • 06 Feb 23
🕹 Technology System Design Query Language
  1. A Database Management System (DBMS) is a tool to manage data, providing an abstraction to store and retrieve data without directly interacting with databases.
  2. DBMS operates using a Query Language, offering guarantees for performance, but the specifics can vary between different systems.
  3. Guarantees provided by a DBMS include serving different data storage purposes, using a client/server model, and incorporating components like transaction managers, lock managers, and storage engines.

How Discord Handles Billions of Messages [System Design Sundays]

Technology Made Simple • 39 implied HN points • 17 Oct 22
🕹 Technology System Design Messaging Database Search Engineering
  1. Storage is crucial in system design, requiring detailed analysis of data generated for efficient handling.
  2. Discord uses Cassandra as its database to manage the storage of messages efficiently.
  3. Indexing messages and using tools like ElasticSearch are key for efficient search functionality in large data sets.

System Design Fundamentals: What is a Load Balancer ?

Engineering At Scale • 2 HN points • 15 Jan 24
🕹 Technology System Design Algorithms Hardware Software
  1. Load Balancers distribute client requests to different servers, improving system reliability and scalability.
  2. Load Balancers handle growing internet usage by evenly distributing workloads, preventing servers from being overwhelmed.
  3. Different types of Load Balancers include Hardware, Software, and Cloud Load Balancers, each with unique benefits for system optimization.

Writeup System Design — Backend for Google Photos[Systems Design Sundays]

Technology Made Simple • 39 implied HN points • 24 Jul 22
🕹 Technology System Design Backend Interviews Engineering
  1. To ace System Design Interviews, read/watch more system design mock interviews on YouTube, engineering blogs, and learn about important technologies/concepts.
  2. When designing a system like the backend for Google Photos, start with domain analysis, note requirements (functional and non-functional), expected load/performance, and user profiles for valuable solutions.
  3. Engage with the content you find helpful by liking, sharing, and offering feedback to creators. It helps them reach more people and improve their work.

The CAP Theorem [System Design Sundays]

Technology Made Simple • 39 implied HN points • 09 May 22
🕹 Technology System Design
  1. The CAP Theorem states that in a distributed system, you can only have two out of three characteristics: consistency, availability, and partition tolerance. It helps in choosing the right database solutions based on system needs.
  2. Understanding the CAP Theorem is crucial for Systems Design Interviews where you work on building distributed systems. It provides clarity on why different databases like MongoDB and Cassandra are chosen for specific scenarios.
  3. To grasp the concepts of the CAP Theorem and learn about suitable databases, watching educational videos and reading articles can be very beneficial.

How to Make Money in Machine Learning, without doing the Math/Theory[Finance Fridays]

Technology Made Simple • 39 implied HN points • 07 May 22
🕹 Technology Machine Learning Software Engineering System Design
  1. There are various ways to make money in Machine Learning beyond the traditional roles like AI research and Data Analysis, such as specializing in software engineering aspects like developing hardware, building data sources, creating pipelines, and designing platforms.
  2. Important skills to succeed in these alternative paths include writing good tests, mastering data compression and handling, and becoming proficient in large-scale system design to ensure scalability.
  3. Staying updated with ML resources and technologies like Airflow, Kubernetes, and Snowflake can be valuable for maximizing income opportunities in Machine Learning without needing to focus on the mathematics and theory aspects.

How Netflix handles it operations[System Design Sundays]

Technology Made Simple • 39 implied HN points • 04 Apr 22
🕹 Technology System Design Engineering Interview Preparation Tech Tools
  1. Good systems are created using a top-down approach, focusing on important decisions with the highest ROI, especially in limited time/resource scenarios like interviews.
  2. Understanding recent tools and technologies is crucial for systems design, different from Leetcode-style problems.
  3. Thinking from multiple perspectives such as developer, investor, and user is key to mastering systems design, helping to tackle complex questions effectively.

New Newsletter Structure

Technology Made Simple • 39 implied HN points • 13 Mar 22
🚌 Education Interview Preparation Math Techniques Finance System Design
  1. The newsletter is now going daily, offering a comprehensive schedule for coding interview preparation.
  2. The schedule includes themed days like Math Monday and Technique Tuesday, covering important aspects of coding interviews.
  3. Premium subscribers gain access to in-depth solutions, stories, and system design discussions, enhancing their interview preparation.

How does Live Streaming Work [System Design Sundays]

Technology Made Simple • 19 implied HN points • 03 Oct 22
🕹 Technology System Design Social media
  1. Systems design is complex and involves understanding intricacies built over years by many people.
  2. Avoid overwhelming yourself by trying to study everything when preparing for systems design interviews; focus on what you know, are interested in, or relates to your work.
  3. In systems design interviews, emphasize your expertise, ability to manage stakeholder needs, handle ambiguity, and foresee challenges; mentioning industry practices from engineering blogs can make you stand out.

Federated Learning [Systems Design Sundays]

Technology Made Simple • 19 implied HN points • 31 Jul 22
🕹 Technology Machine Learning Data Privacy System Design
  1. Federated Learning is a system where individual devices have their own mini ML models that update based on user input, then the updates are shared with a central server for a collective model.
  2. Federated Learning offers cheaper training and better security, making it a good solution for dynamic systems that constantly change based on user preferences.
  3. Federated Learning is beneficial for applications like social media or recommendation systems that require distributed learning processes and can handle multiple user input interfaces.

Undertstanding Kafka [System Design Sundays]

Technology Made Simple • 19 implied HN points • 03 Jul 22
🕹 Technology Tech news System Design Cloud Computing
  1. Apache Kafka is an open-source distributed event streaming platform used for handling large amounts of data generated by multiple sources simultaneously.
  2. Kafka's key functions include publishing and subscribing to streams of records, storing records in order, and processing streams in real-time, making it essential for real-time streaming data pipelines and applications.
  3. Kafka offers 5 APIs (admin, producer, consumer, streams, connector) for managing topics, publishing streams, subscribing, implementing stream processing, and building data connectors, demonstrating its versatility and usability in system design.

How would you design Twitter

Technology Made Simple • 19 implied HN points • 16 Mar 22
🕹 Technology System Design Coding Interviews YouTube
  1. When designing a system like Twitter, it's important to approach the problem methodically and break it down into smaller sub-problems to find solutions.
  2. System design interviews don't have strict right or wrong answers, so it's okay to focus on different aspects based on your expertise.
  3. Engaging with resources and trying to solve the problem yourself first before looking at solutions can be a valuable learning approach for system design.

System Design Concepts: Dive deep into Database Sharding Strategies

Engineering At Scale • 2 HN points • 05 Aug 23
🕹 Technology System Design Scalability Performance Data security
  1. Range-Based Sharding divides data based on ranges like organizing books in bookshelves to make searches easier.
  2. Hash-Based Sharding evenly distributes data across different shards using a hash function, but may require data rebalancing when the number of shards changes.
  3. Consistent Hashing minimizes data movement when adding or removing shards, improving scalability while Geo-Based Sharding stores data close to users for better performance.

Microservices impair the productivity of high performing programmers

Sheep Code • 1 HN point • 10 Aug 23
🕹 Technology Software Engineering Microservices System Design
  1. Microservices can limit the productivity of high performing coders due to the complexity of dependencies across teams.
  2. In a microservices system, expertise shifts from individual coding skills to system design and team collaboration.
  3. For a 10x engineer in a microservices environment, success is more about harmonizing microservices and teams than individual coding prowess.

System Design Fundamentals: Understanding Scalability

Engineering At Scale • 0 implied HN points • 10 Jun 23
🕹 Technology System Design Scalability Cloud Computing Infrastructure Software Development
  1. Scalability is crucial for software systems to handle increasing demand and data.
  2. Building scalable systems can involve horizontal scaling (adding more machines) or vertical scaling (adding more resources to the same machine).
  3. Cloud technologies, like auto-scaling and managed databases, offer solutions for building scalable systems.