Google’s new technique gives LLMs infinite context

Google researchers have developed a new method called Infini-attention, designed to extend the context window of large language models (LLMs) to potentially infinite lengths without additional memory costs. This breakthrough addresses the traditional quadratic complexity of transformers, where memory and computation needs increase exponentially with input size. By integrating a compressive memory module into the classic attention mechanism, Infini-attention allows for efficient handling of extended inputs, maintaining high performance on very long sequences while drastically reducing memory requirements.

Infini-attention has been tested on various benchmarks, demonstrating its superiority in managing long-context tasks by maintaining lower perplexity scores and reducing memory usage by up to 114 times compared to other models. This technique enables the handling of sequences as long as one million tokens and shows promising results in tests like passkey retrieval and text summarization, significantly outperforming existing long-context models.

The introduction of LLMs capable of processing infinite context opens new possibilities for custom applications, reducing the need for complex engineering solutions like fine-tuning and retrieval-augmented generation. This could simplify the development of AI applications, allowing for more straightforward incorporation of extensive data directly into prompts, thereby facilitating the creation of tailored responses and enhancing model adaptability without extensive additional training.

DeepMind’s New AI Saw 15,000,000,000 Chess Boards!

Scientists at Google DeepMind have developed a groundbreaking AI that challenges the traditional methods used in creating chess-playing systems like those used by grandmasters. Unlike previous iterations that relied on self-play and intricate search algorithms, this new AI adopts a radically simplified approach by learning directly from Stockfish, a leading chess engine. This AI, without engaging in any games from start to finish, studied 15 billion board states and the corresponding moves made by Stockfish, focusing solely on single moves rather than entire games. Impressively, despite its simplicity and relatively smaller size compared to giants like GPT-4, it achieves grandmaster-level play and can process moves at a significant speed even on low-cost hardware.

This project diverges from the aim of building a superior chess engine. Instead, the focus is on demonstrating that transformer-type neural networks can effectively learn and replicate the expertise of a chess master through observation alone. This AI not only mimics but also understands and generalizes the strategies to new situations, showing a profound shift in how AI can learn complex tasks. This method of learning, which eschews deep strategic calculations for direct pattern recognition from expert moves, points to a new way of teaching AI complex decision-making.

The implications of this research extend far beyond the realm of chess. By learning algorithms through observation, this technique paves the way for advancements in various fields, such as autonomous driving and advanced computing algorithms. Such capabilities hint at a future where AI can decipher and replicate complex algorithms from observing tasks, offering a more intuitive and accessible way to harness AI for practical and innovative applications. This could revolutionize how we develop AI systems, making them more efficient and adaptable across diverse domains.

Adobe’s ‘Ethical’ Firefly AI Was Trained on Midjourney Images

Adobe Inc. introduced its AI image-generating software, Firefly, which was promoted as being trained primarily on Adobe Stock images, boasting hundreds of millions of licensed images. This move was intended to make Firefly a commercially safe alternative in the AI image generator market, differentiating itself from competitors like Midjourney, who use internet-scraped images. However, it was later revealed that Adobe also used AI-generated images from these competitors as part of Firefly's training data, a fact not initially disclosed in their public representations.

In response to the growing use of AI-generated content, Adobe allowed creators to submit images generated by AI into the Adobe Stock marketplace, which contributed to the training dataset of Firefly. About 5% of the images used to train Firefly came from other AI platforms, despite Adobe's assurance that all content undergoes rigorous moderation to ensure no intellectual property or recognizable logos are included. The inclusion of AI-generated content in Firefly's training set has led to internal debates within Adobe about the ethics and future use of such data.

Adobe's strategy to differentiate Firefly by using controlled, legally obtained data has been undermined by their reliance on AI-generated content, which is often cheaper and readily available. This practice has brought up ethical concerns about transparency and the true uniqueness of Firefly compared to other AI tools that use openly scraped data. Although Adobe has been criticized for this approach, the company continues to promote Firefly as an ethical and commercially safer option for generating images.

Google goes all in on generative AI at Google Cloud Next

The recent Google Cloud event in Las Vegas focused heavily on generative AI, emphasizing its potential to revolutionize various aspects of business and technology. Google showcased enhancements to their Gemini large language model (LLM), aiming to boost productivity and demonstrate the model's capabilities through various demos. However, the presentations sometimes seemed overly simplistic and primarily confined within the Google ecosystem, potentially limiting broader applicability.

Critics argue that while the demonstrations were impressive, they didn't always necessitate AI, such as an e-commerce transaction that could have been completed manually. Moreover, the event seemed to gloss over the substantial challenges and complexities involved in implementing such advanced technologies within large organizations. Despite the potential of generative AI, real-world application remains a significant hurdle, particularly for companies that are not as digitally advanced.

The discussions around data—the fundamental ingredient for training effective AI models—highlighted the principle of "garbage in, garbage out." For many companies, the prerequisite of having organized, clean data is still a major barrier. Thus, while Google's tools aim to simplify data integration and engineering, organizations lagging in digital transformation may find these tools less beneficial.

Additionally, concerns about governance, security, and ethical use of AI continue to be significant, underscoring the need for careful consideration beyond the initial excitement of new AI capabilities.

Other stuff

• Who’s gonna win the white house in 2032?

• The rise of the chief AI officer

• Generative AI is coming for healthcare, and not everyone’s thrilled

• 14 LLMs fought 314 Street Fighter matches. Here's who won

• Holodeck: Language Guided Generation of 3D Embodied AI Environments

• ChatGPT Can Predict the Future when it Tells Stories Set in the Future About the Past

Enhance Your Workflow with Superpower ChatGPT Pro: Introducing the Right-Click Menu Feature!

Hope you enjoyed today's newsletter

⚡️ Join over 200,000 people using the Superpower ChatGPT extension on Chrome and Firefox.