The hottest Software Architecture Substack posts right now

And their main takeaways
Category
Top Technology Topics

Designing Instagram's Video Uploads: Optimizing for Low Latency and scalability

Engineering At Scale • 255 implied HN points • 20 Jan 25
🕹 Technology System Design Software Architecture Scalability
  1. Instagram's video upload system needs to handle millions of uploads daily while keeping the process fast and efficient. It converts videos into different formats for users with varying internet speeds.
  2. The system can be designed in approaches, starting from simple methods to more complex asynchronous solutions. Improving reliability and speed is key to making the service work better.
  3. Using segmented video uploads allows faster processing. By uploading smaller parts of the video, the service can work on them at the same time, reducing wait times for users.

Nature's many attempts to evolve a Nostr

Subconscious • 2095 implied HN points • 13 Oct 24
🕹 Technology Decentralization Software Architecture Data Ownership
  1. Centralized apps control user data and accounts, making users dependent on their servers. This creates a situation where users can be locked out at any time.
  2. Federated systems allow users to choose servers, but can still lead to centralization as large nodes dominate, limiting true user independence.
  3. Peer-to-peer networks promote equality among users, but they often evolve to create 'superpeers' that can mirror centralization, so a new approach using relays could improve user control and ownership.

How did Facebook design their Real-Time Processing ecosystem

VuTrinh. • 279 implied HN points • 17 Aug 24
🕹 Technology Data Engineering Real-Time Processing System Design Software Architecture Big Data
  1. Facebook's real-time data processing system needs to handle huge amounts of data quickly, with only a few seconds of wait time. This helps in keeping things running smoothly for users.
  2. Their system uses a message bus called Scribe to connect different parts, making it easier to manage data flow and recover from errors. This setup improves how they deal with issues when they arise.
  3. Different tools like Puma and Stylus allow developers to build applications in different ways, depending on their needs. This means teams can quickly create and improve their applications over time.

Diving Deep into LinkedIn's Data Infrastructure: My 6-Hour Learning & Key Takeaways

VuTrinh. • 299 implied HN points • 03 Aug 24
🕹 Technology Data Engineering Software Architecture Databases Distributed Systems Cloud Computing
  1. LinkedIn's data infrastructure is organized into three main tiers: data, service, and display. This setup helps the system to scale easily without moving data around.
  2. Voldemort is LinkedIn's key-value store that efficiently handles high-traffic queries and allows easy scaling by adding new nodes without downtime.
  3. Databus is a change data capture system that keeps LinkedIn's databases synchronized across applications, allowing for quick updates and consistent data flow.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

The Hadoop Distributed File System

VuTrinh. • 259 implied HN points • 18 May 24
🕹 Technology Data Storage Cloud Computing Big Data Software Architecture
  1. Hadoop Distributed File System (HDFS) is great for managing large amounts of data across many servers. It ensures data is stored reliably and can be accessed quickly.
  2. HDFS uses a NameNode that keeps track of where data is stored and multiple DataNodes that hold actual data copies. This design helps with data management and availability.
  3. Replication is key in HDFS, as it keeps multiple copies of data across different nodes to prevent loss. This makes HDFS robust even if some servers fail.

Single Tenant vs Multi Tenant Architecture for SAAS Platforms [System Design Sundays]

Technology Made Simple • 139 implied HN points • 04 Dec 23
🕹 Technology System Design Software Architecture Cloud Services Data Systems
  1. Single Tenant Architecture provides each customer their own independent database and software instance, offering security and customization like living in a detached house.
  2. Multi-Tenant Architecture is akin to an apartment building where multiple tenants share common infrastructure, allowing for economies of scale but potentially limiting customization.
  3. Single Tenant architecture is known for high user engagement, control, and stability, while Multi-Tenant architecture favors compliance, security, and quick onboarding for better scalability.

I spent another 6 hours understanding the design principles of Snowflake. Here's what I found

VuTrinh. • 79 implied HN points • 10 Feb 24
🕹 Technology Data Engineering Cloud Computing Software Architecture Database Systems Data Analytics
  1. Snowflake separates storage and compute, allowing for flexible scaling and improved performance. This means that data storage can grow separately from computing power, making it easier to manage resources.
  2. Data can be stored in a cloud-based format that supports both structured and semi-structured data. This flexibility allows users to easily handle various data types without needing to define a strict schema.
  3. Snowflake implements unique optimization techniques, like data skipping and a push-based query execution model, which enhance performance and efficiency when processing large amounts of data.

3 quick tips on how to build more scalable systems [System Design Sundays]

Technology Made Simple • 119 implied HN points • 17 Apr 23
🕹 Technology System Design Scalability Engineering Software Architecture
  1. Location matters: Place software close to clients for faster response times using CDNs, edge computing, or geo-replication.
  2. Cache wisely: Optimize speed by using in-memory caching, database caching, or web caching to avoid repeated actions.
  3. Async is key: Improve efficiency with asynchronous processing through message queues, event-driven architectures, or microservices.

Monoliths vs Microservices [System Design Sundays]

Technology Made Simple • 99 implied HN points • 24 Oct 22
🕹 Technology Software Architecture
  1. Monolith architecture involves building and maintaining all components together, requiring a full rebuild when changes are made.
  2. Microservices architecture deploys and maintains different parts of the system independently, allowing for scalability and flexibility but requiring careful planning.
  3. Monoliths offer simplicity and easier testing, while Microservices provide scalability and quicker deployment but demand more setup effort and planning.

Clean Architecture the Right Way

🔮 Crafting Tech Teams • 19 implied HN points • 02 Aug 23
🕹 Technology Software Architecture
  1. Clean Architecture is not just about the folder structure, it involves more practical considerations beyond that.
  2. Copying someone else's 'ideal, clean architecture' may not be the right approach; understand the underlying reasons for implementing Clean Architecture.
  3. To truly grasp the essence of Clean Architecture, focus on the practical aspects and implications rather than just the superficial folder structure.

The effects of splitting a project into multiple teams[Systems Design Sundays]

Technology Made Simple • 39 implied HN points • 18 Apr 22
🕹 Technology Software Architecture Systems Design
  1. As projects grow, you may need multiple teams to handle different components, changing how you work from being in one team to collaborating across teams.
  2. Conway's Law emphasizes that a system's design structure mirrors the organization's communication structure, highlighting the importance of how teams interact when developing a project.
  3. Learning about the risks in current software architecture design approaches can help in adapting and improving your skills for dealing with larger project scopes.

A Dispatch On All Things Tools

Building Rome(s) • 7 implied HN points • 11 Sep 23
🕹 Technology Tools Program Management Software Architecture Software Development Product Management
  1. Building timelines can be frustrating due to lack of ideal tools like Gantt charts or whiteboarding tools.
  2. C4 Diagrams are a great visual language to discuss software architecture for TPMs and PMs.
  3. There is a desire for an all-in-one tool to manage product development, simplifying the use of multiple tools like Slack, Linear, Coda, Figma.

Data is not a Microservice

Data Products • 1 HN point • 07 Jul 23
🕹 Technology Data Engineering Software Architecture Data Management Microservices
  1. Data requires a source of truth that microservices cannot inherently provide without a shift in software engineering practices
  2. Not all data is equally valuable, so treating all data as microservices can be costly and restrictive
  3. The data development lifecycle differs from software development, requiring flexibility, reuse, and tight coupling that conflict with typical microservices architecture

Dataflow 101: Exploring Essential Modes for Efficient Applications

DataSketch’s Substack • 0 implied HN points • 13 Feb 24
🕹 Technology Data Engineering APIs Databases Real-Time Processing Software Architecture
  1. Databases are key for storing and managing data, supporting both everyday transactions and complex analysis. Using them effectively helps data engineers connect different platforms and applications.
  2. Different data transfer methods, like REST and RPC, help systems communicate efficiently, just like a well-organized library or a quick phone call. Choosing the right method depends on the speed and precision needed for the task.
  3. Message-passing systems allow for flexible and real-time data processing, making them great for applications like IoT or e-commerce. They help ensure communications between services happen smoothly and reliably.

😎 21 Years of Notepad++; Modules Vs. Microservices

ppdispatch • 0 implied HN points • 05 Nov 24
🕹 Technology Software Development Coding Practices Open Source Performance optimization Software Architecture
  1. Notepad++ has been a reliable text editor for 21 years, helping developers and writers with its user-friendly features and community-driven support.
  2. Linus Torvalds has made a small update to the Linux kernel that improves its performance by 2.6%, showing that even tiny changes can have a big impact.
  3. Microservices might not be as new as they seem; their benefits have roots in older technologies, and while they support independent development, they also introduce challenges in communication.