The hottest Web scraping Substack posts right now

And their main takeaways
Category
Top Technology Topics

The Most Complicated System I Ever Worked On

Jacob’s Tech Tavern • 1312 implied HN points • 17 Feb 26
🕹 Technology Web scraping
  1. A single feature can balloon into a ludicrously elaborate pipeline that combines webscraping, long-running downloads, parsing and storage of large data, real-time analysis, and high-volume upload/polling.
  2. Most engineering work is routine, but rare peak challenges require orchestrating many moving parts and constant attention so they don’t overwhelm the team.
  3. Making a reliable system on top of unreliable third-party services takes sustained hardening and ongoing “whack-a-mole” maintenance to turn an MVP into production-grade software.

Artificial Intelligence is ushering in a new era of web scraping possibilities

The Orchestra Data Leadership Newsletter • 79 implied HN points • 14 May 24
🕹 Technology Web scraping
  1. Artificial Intelligence is revolutionizing web scraping by offering accelerated development processes and increased adoption of scraping use-cases in Data.
  2. The complexity of parsing HTML and the challenges associated with web scraping, such as changing schemas, time durations, and legality, can be mitigated with AI-enabled tools.
  3. AI-enabled web scraping tools like Nimble and Diffbot provide reliable solutions for efficiently extracting data from the internet and handling challenges like managing proxies and optimizing scraping speed.

Block AI tools from scraping your site in 3 minutes

Deploy Securely • 216 implied HN points • 10 Jan 24
🕹 Technology Web scraping
  1. Block major generative AI tools from scraping your website by adding specific directives to your robots.txt file.
  2. Consider modifying your site's terms and conditions to prevent undesired activities like scraping by AI tools.
  3. Blocking AI tools may impact your search and social media rankings, so find a balance between cybersecurity and potential repercussions.

AI Web scraping use-cases for Data Teams: Intelligence gathering

The Orchestra Data Leadership Newsletter • 39 implied HN points • 21 May 24
🕹 Technology Web scraping
  1. Web scraping with AI can enhance intelligence gathering by efficiently collecting and processing data from various public sources on the internet.
  2. Leveraging Large Language Models (LLMs) can improve the accuracy and robustness of web scraping systems when dealing with changes in HTML code structure.
  3. Using tools like Nimble for web scraping allows for more efficient and accurate data collection by training models on different types of websites for specific use cases.

The Lab #37: Bypassing Cloudflare with anti-detect browsers - Part 2

The Web Scraping Club • 98 implied HN points • 18 Jan 24
🕹 Technology Web scraping
  1. Kameleo is an anti-detect tool that creates different profiles for scrapers with unique characteristics like OS and browser settings.
  2. Profiles created with Kameleo can have realistic fingerprints mimicking various devices and operating systems.
  3. Using Kameleo can help in bypassing Cloudflare's anti-bot protection measures by generating profiles that seem legitimate.
Get a weekly roundup of the best Substack posts, by hacker news affinity:

The Lab #40: start a web data monetization project with Data Boutique

The Web Scraping Club • 78 implied HN points • 08 Feb 24
🕹 Technology Web scraping
  1. Data Boutique is a marketplace for legally obtained web-scraped data with a focus on quality and easy accessibility.
  2. Sellers on Data Boutique align interests with the platform by offering affordable, high-quality data which encourages more purchases and recurring buyers.
  3. Ensuring data quality on Data Boutique involves embedded checks and a Peer Review program, promoting stackable standard data schemas for wider use cases.

Monetize your web scraping skills

The Web Scraping Club • 58 implied HN points • 14 Jan 24
💼 Business Web scraping
  1. Freelance your web scraping skills through traditional marketplaces or job boards like Upwork and Fiverr.
  2. Consider selling your web scraping code on the Apify platform to monetize your skills.
  3. Explore selling the data you extract from web scraping on marketplaces like Data Boutique for additional income.

How scraping a single website costed thousands of dollars in proxy

The Web Scraping Club • 39 implied HN points • 28 Jan 24
🕹 Technology Web scraping
  1. Using different credentials per website when using proxies helps detect outliers and set thresholds per website.
  2. Be selective about the resources to load in your browser when using automation tools like Playwright to avoid unexpected usage of resources.
  3. Set a threshold to proxy usage for your scraper, even for seemingly trivial scrapers, to avoid unexpected costs.

Web scraping from 0 to hero: Microsoft Playwright

The Web Scraping Club • 39 implied HN points • 21 Jan 24
🕹 Technology Web scraping
  1. Microsoft Playwright is a browser automation tool used for web scraping that supports multiple browsers and can handle dynamic content and complex user interactions.
  2. Browser automation tools like Playwright are crucial for scraping modern websites with dynamic content, interactive elements, and sophisticated front-end frameworks.
  3. Playwright excels at rendering JavaScript, simulating user interactions, and evading anti-scraping measures, but may be slower and require more resources compared to Scrapy.

Web scraping from 0 to hero: creating our first Scrapy spider - Part 2

The Web Scraping Club • 39 implied HN points • 07 Jan 24
🕹 Technology Web scraping
  1. The web scraping course provided by The Web Scraping Club is always free and offers practical articles on complex topics.
  2. Scrapy allows for easy crawling of product category pages by making requests per URL retrieved.
  3. Using JSON is preferred for extracting and cleaning data in web scraping, offering more stability compared to manual selectors.

How to Summarize Youtube Video using ChatGPT Api and Node.js

Implementing • 39 implied HN points • 02 Jan 24
🕹 Technology Web scraping
  1. The system architecture for summarizing YouTube videos involves extracting text from videos and generating text summaries using OpenAI's completions API.
  2. The process includes scraping YouTube automatic captioning for text extraction and dividing large text into smaller parts to handle limitations of the completions API.
  3. A command line interface (CLI) was created to allow users to easily summarize YouTube videos by passing the video link and desired language code.

Discovering Shopify Domains: A Journey Through Common Crawl Data

Ali's Tech Tales • 7 HN points • 17 Jun 24
🕹 Technology Web scraping
  1. Utilizing object storage like MinIO can streamline processes and reduce the amount of code needed for handling large data sets efficiently.
  2. Efficiently processing large volumes of data using multiprocessing in Python can significantly speed up tasks like parsing vast numbers of URLs in parallel.
  3. By merging dictionaries containing hostnames and then splitting them into manageable chunks, it's possible to handle huge amounts of data effectively, such as discovering over 140 million unique website hostnames.

How to Create a Bot that Notifies you on Telegram

Implementing • 19 implied HN points • 22 Jan 24
🕹 Technology Web scraping
  1. Creating a bot to monitor computer temperature and send notifications can be useful to prevent overheating issues.
  2. Learning how to create a Telegram bot involves steps like creating the bot on Telegram using BotFather and deploying the code on platforms like Heroku.
  3. Setting up a Cron job using tools like Heroku Scheduler allows the bot to execute functions periodically to send notifications at specified intervals.

AI Scraping Benchmarks

serious web3 analysis • 20 HN points • 24 Sep 24
🕹 Technology Web scraping
  1. AI can make web scraping super easy by letting users scrape information in plain English instead of complicated coding. This can help many more people access scraping tools.
  2. It's important to track the costs of using AI for scraping. Choosing the right AI model can save money while still getting accurate results.
  3. Benchmarking AI scrapers based on accuracy, runtime, and cost is essential. It helps users find the best tools for their specific scraping needs.

Meta v. Bright Data Perspectives: A Data Score Interview with Glacier Networks

The Data Score • 1 HN point • 20 Feb 24
💼 Business Web scraping
  1. The court ruling in the Meta v. Bright Data case may lead to more defenses against web scraping and offers clarity on accessing public data while underscoring the importance of adhering to individual website terms.
  2. Before starting a web mining project, individuals should carefully review each website's terms, assess intended usage of scraped data, and consider the legal implications of accessing specific content.
  3. Upcoming court cases, like those involving Meta and other companies, may set standards for web mining governance while Glacier Network emphasizes a standardized risk policy to simplify data exchange and compliance in a rapidly evolving data industry.

How to hunt down TikTok's top videos

12challenges • 3 HN points • 13 Feb 24
🕹 Technology Web scraping
  1. Hunting down TikTok's top videos is challenging because the data is not easily accessible through conventional methods like Google search.
  2. Using TikTok's Research API is limited and not helpful in obtaining the top TikTok videos by view count.
  3. Scraping TikTok's platform or using social monitoring tools are options to consider, but these methods come with challenges like legal implications and high costs.

The Lab 38: Bypassing Kasada for web scraping 2024 edition

The Web Scraping Club • 0 implied HN points • 25 Jan 24
🕹 Technology Web scraping
  1. Kasada is an anti-bot solution that collects data points from browsers and uses AI to determine if a request is legitimate or not.
  2. Scrapers that do not use a Javascript rendering engine, like Scrapy, are blocked by Kasada's challenges.
  3. One way to bypass Kasada for web scraping is to use tools like undetected-chromedriver, a Selenium Chromedriver patch that avoids triggering anti-bot services.