The hottest Benchmarks Substack posts right now

Gemini 1.0 comes in three sizes: Ultra, Pro, and Nano for different tasks.
Gemini Ultra achieves high accuracy and surpasses GPT-4 in many benchmarks.
Gemini Pro is a substantial upgrade, but the full potential of Gemini is yet to be seen with Bard Advanced.

Benchmarks for large language models have limitations in reflecting real-world utility.
Overfitting in benchmarks can hinder adaptability to new scenarios and tasks.
Multi-modal capabilities are often lacking in benchmarks, missing out on testing language and image understanding together.

Superhuman AI can use concepts beyond human knowledge, and we need to understand these concepts to supervise AI effectively.
Transformers can generalize tasks differently based on the complexity and structure of the task, showing varying capabilities in different scenarios.
Implementing preprocessing defenses like random input perturbations can be effective against jailbreaking attacks on large language models.

Inheritance can impact performance negatively in C++ due to issues like indirection and virtual function dispatch.
Data-oriented design (DOD) can lead to improved performance by optimizing data organization over code organization.
Using a struct of arrays approach instead of std::variant can offer better performance and minimize memory overhead in certain scenarios.

Devin is an AI-powered software engineer with features like a built-in terminal, IDE, website preview, and a text assistant.
Devin demonstrated capabilities like finding and fixing bugs in GitHub repos and running tests on code, showing potential for automating debugging tasks.
Cognition Labs, the company behind Devin, has notable supporters like Thiel's Founders Fund and founders with strong backgrounds in software engineering and machine learning.

Get a weekly roundup of the best Substack posts, by hacker news affinity:

Consider using lighter embedding models before heavier ones.
If you are using a large model like Instructor XL, then consider trying OpenAI's embeddings for blind comparison.
Be cautious using OpenAI's embeddings due to internet dependency and potential future changes.

Practical approach for Poker on blockchain involves playing out hands normally and cancelling any with duplicate cards.
On-chain protocol for Poker involves multiple steps of committing to and revealing images for cards and calculations.
Benchmarks for practical Poker protocol include computation time, round trips, and data transfer limits.

ChatGPT is not getting dumber, just misunderstood when instructions aren't clear.
LLM Benchmark: A new model has surpassed Chat GPT 3.5 on Enterprise Workloads.
ML Product Labs offers two new guides for building products with LLM technology.