MIT debuts a large language model-inspired method for teaching robots new skills

Researchers at MIT have developed a new technique to train general-purpose robots by pooling diverse data from different sources into one system. Unlike traditional methods that train robots for specific tasks in a controlled environment, this versatile technique combines data from various domains, such as simulations and real-world robots, into a shared "language" that can be processed by a generative AI model. This method aims to teach robots a wide range of tasks more efficiently, reducing the cost and time involved in gathering task-specific data.

The approach takes inspiration from large language models, which are pretrained using vast amounts of diverse language data and then fine-tuned with smaller amounts of task-specific data. MIT's new architecture, called Heterogeneous Pretrained Transformers (HPT), integrates multiple modalities, including vision sensors and robotic arm encoders, using a transformer model similar to those used in language processing. By aligning vision and proprioceptive data into tokens that the transformer can process, the researchers created a scalable system that can learn from a large, varied dataset, improving adaptability and dexterous performance in robots.

HPT was pretrained on 52 datasets containing more than 200,000 robot trajectories across categories such as human demonstration videos and simulations. In testing, HPT improved robot performance by over 20% compared to training from scratch, even when applied to tasks quite different from those it had seen before. The team envisions that in the future, users could simply download a universal robot brain for their robot, similar to large language models like GPT-4, eliminating the need for extensive retraining.

The researchers plan to continue exploring how diverse datasets can enhance HPT's capabilities and hope to eventually achieve a breakthrough in robotic policies, enabling robots to be more adaptive and versatile. Their work could pave the way for robots to handle a wide array of tasks with minimal additional training, just as large language models have revolutionized the processing and generation of human language.

Finally, a developer tool that makes digging around for the answer a thing of the past.

Other developer tools can’t tell you how your codebase works and why. Unblocked can. We augment your source code with context from Slack, Confluence, Jira (and more), so your team gets helpful answers without having to search for them.

Unblocked gives you:

• Instant answers about every aspect of your codebase 

• Relevant, historical context for any code open in an IDE

• Automated answers to questions asked in Slack, removing interruptions to you

⌛Engineers who use Unblocked save an hour or more a day.

“Every developer now has the ability to tap into past discussions and decisions to fill their knowledge gaps, regardless of their tenure. We are moving faster and making more accurate decisions as a team as a result.” - Alex Mallet, EVP of Engineering at Forto

Google's 'Big Sleep' AI Project Uncovers Real Software Vulnerabilities

Google's experimental AI project, called "Big Sleep," has recently made headlines by uncovering a previously unknown and exploitable vulnerability in SQLite, an open-source database engine. According to Google's researchers, this marks the first public example of an AI agent autonomously finding a memory-safety issue in widely used real-world software. The AI's discovery led SQLite to issue a fix before the vulnerable code made it into an official release, which represents a significant advancement in the use of AI for cybersecurity.

Big Sleep was designed to mimic the workflow of a human security researcher, using specialized tools to identify complex vulnerabilities and analyze code for variants of existing security issues. The project builds on previous research, showing that large language models can be used to find software vulnerabilities. While similar AI projects have been used to identify flaws in software, Google claims this recent discovery demonstrates the potential of AI to detect deeper and more complex bugs before software reaches users.

The AI agent's success in finding the SQLite bug was the result of extensive variant analysis, where Big Sleep reviewed recent code changes and triggered a system crash to perform a root-cause analysis. Google's researchers acknowledge that traditional bug-finding tools, such as target-specific fuzzers, might have also found the same issue. Nevertheless, they see Big Sleep as a promising step towards enhancing defenders' capabilities by providing more effective root-cause analysis, making it easier and cheaper to triage and fix vulnerabilities.

The achievement suggests a future in which AI-driven tools could give developers an edge in defending software against malicious actors. With the ability to autonomously identify vulnerabilities and provide detailed analysis, AI systems like Big Sleep could help shift the balance in favor of defenders, offering an asymmetrical advantage over attackers.

Perplexity CEO offers AI company’s services to replace striking NYT staff

Perplexity CEO Aravind Srinivas recently offered to help the New York Times maintain services amidst a strike by the NYT Tech Guild, sparking backlash online. The Guild, representing tech staff responsible for software and data services at the Times, began its strike on November 4 after contract negotiations broke down. Workers are seeking a 2.5% annual wage increase and a commitment to a two-day in-office requirement, among other terms. However, the NYT publisher, AG Sulzberger, criticized the strike timing, highlighting the public’s need for uninterrupted election coverage.

On social media platform X, Srinivas offered Perplexity’s services to support the NYT’s operations during the strike, sparking criticism for what some perceived as a “scab” move — a term for those willing to work in place of striking employees. Responding to the backlash, Srinivas clarified that his intention was not to replace striking staff with AI but to offer technical support on a high-traffic day. Still, critics argue that Perplexity’s involvement could undermine the strike’s purpose by reducing the impact of collective action.

This situation follows recent legal tension between Perplexity and the NYT, as the Times issued a cease-and-desist letter last month over Perplexity's alleged unauthorized use of NYT content in its AI models. In a conversation with TechCrunch, Srinivas refrained from commenting on the issue of “plagiarism,” highlighting ongoing friction between the companies.

Other stuff

• Apple users can soon upgrade to ChatGPT Plus within the Settings app

• Meta’s former hardware lead for Orion is joining OpenAI

• ChatGPT Search is not OpenAI’s ‘Google killer’ yet

• Netflix Bullish on Gen AI for Games After Laying Off Human Game Developers

• Meta says it’s making its Llama models available for US national security applications

• Chinese researchers build military AI using Meta’s open-source Llama model

• Mystery Surrounds Discovery of TSMC Tech Inside Huawei AI Chips

• Anthropic releases Claude 3.5 Haiku, but increases the price

• New OpenAI feature: Predicted Outputs

Hope you enjoyed today's newsletter

⚡️ Join over 200,000 people using the Superpower ChatGPT extension on Chrome and Firefox.