Latest

We present LlamaFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. LlamaFusion leverages existing Llama-3’s weights for processing texts...

Multiple-choice question (MCQ) datasets like Massive Multitask Language Understanding (MMLU) are widely used to evaluate the commonsense, understanding, and problem-solving abilities of large language models (LLMs). However, the open-source nature of these benchmarks and the broad sources of training...

When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing...

In Embodied Question Answering (EQA), agents must explore and develop a semantic understanding of an unseen environment in order to answer a situated question with confidence. This remains a challenging problem in robotics, due to the difficulties in obtaining useful semantic representations,...

We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular video. CAT4D leverages a multi-view video diffusion model trained on a diverse combination of datasets to enable novel view synthesis at any specified camera poses and timestamps. Combined with a novel sampling approach,...

Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video. However, prior work has focused evaluations on semantic-related tasks $\unicode{x2013}$ action classification, ImageNet classification, etc. In this paper we focus on evaluating self-supervised learning...

High-performance Multimodal Large Language Models (MLLMs) are heavily dependent on data quality. To advance fine-grained image recognition within MLLMs, we introduce a novel data synthesis method inspired by contrastive learning and image difference captioning. Our key idea involves challenging the...

Large Language Models (LLMs) possess vast amounts of knowledge within their parameters, prompting research into methods for locating and editing this knowledge. Previous work has largely focused on locating entity-related (often single-token) facts in smaller models. However, several key questions...

In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering capabilities in the field of software engineering. CodeRepoQA encompasses five programming languages and covers a wide range of scenarios, enabling comprehensive...

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and resource management. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs...

Gaussian Splatting has enabled real-time 3D human avatars with unprecedented levels of visual quality. While previous methods require a desktop GPU for real-time inference of a single avatar, we aim to squeeze multiple Gaussian avatars onto a portable virtual reality headset with real-time drivable...

Despite significant ongoing efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors, including through the use of adversarial suffixes. Building on prior research, we hypothesize that these...

Large language models (LLMs) have triggered a new stream of research focusing on compressing the context length to reduce the computational cost while ensuring the retention of helpful information for LLMs to answer the given question. Token-based removal methods are one of the most prominent...

In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts, necessitating query rewriting to better describe user’s information needs. However, traditional context-based rewriting has minimal enhancement on downstream generation tasks...

Despite the impressive capabilities of Multimodal Large Language Models (MLLMs) in integrating text and image modalities, challenges remain in accurately interpreting detailed visual elements. Vision detection models excel at recognizing fine-grained image details, prompting researchers to use them...

Over 1.4 billion chest X-rays (CXRs) are performed annually due to their cost-effectiveness as an initial diagnostic test. This scale of radiological studies provides a significant opportunity to streamline CXR interpretation and documentation. While foundation models are a promising solution, the...

We propose an unsupervised model for instruction-based image editing that eliminates the need for ground-truth edited images during training. Existing supervised methods depend on datasets containing triplets of input image, edited image, and edit instruction. These are generated by either existing...

Diffusion models, and their generalization, flow matching, have had a remarkable impact on the field of media generation. Here, the conventional approach is to learn the complex mapping from a simple source distribution of Gaussian noise to the target media distribution. For cross-modal tasks such as...

Reconstructing the geometry and appearance of objects from photographs taken in different environments is difficult as the illumination and therefore the object appearance vary across captured images. This is particularly challenging for more specular objects whose appearance strongly depends on the...

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of conversational agents, making them applicable to various fields (e.g., education). Despite their progress, the evaluation of the agents often overlooks the complexities of real-world conversations,...

A study by Anthropic and Redwood Research, has identified an issue in large language models (LLMs) referred to as ‘alignment faking’.

OpenAI’s efforts to develop its next major model, GPT-5, are running behind schedule, with results that don’t yet justify the enormous costs, according to a new report in The Wall Street Journal. This echoes an earlier report in The Information suggesting that OpenAI is looking to new strategies as...

Welcome back to Week in Review. This week, we’re looking at OpenAI’s last — and biggest — announcement from its “12 Days of OpenAI” event; Apple’s potential entrance into the foldable market; and why Databricks is choosing to wait to go public. Let’s get into it. P.S. We’re off for the holidays! Week...

Google has offered up its own proposal in a recent antitrust case that saw the US Department of Justice argue that Google must sell its Chrome browser. US District Court Judge Amit Mehta ruled in August that Google had acted illegally to maintain a monopoly in online search, with the DOJ then...

Google has filed a proposal outlining how it would remedy the antitrust violations it’s been accused of by the Department of Justice, after the DOJ called for Google to sell off Chrome and face restrictions that would prevent it from favoring its own search engine in Android. Judge Amit Mehta of the...

Google dominated the last era of search. Now the company and the US Justice Department are battling over how to set a fair playing field for generative AI.

It’s past time we applauded the creative skills of LLMs—and it might make us appreciate humans more too.

A federal judge in California has agreed with WhatsApp that the NSO Group, the Israeli cybersurveillance firm behind the Pegasus spyware, had hacked into its systems by sending malware through its servers to thousands of its users’ phones. WhatsApp and its parent company, Meta, sued the NSO Group...

Movie icon and super spy James Bond seemed to be on another rise to the top of the box office just a few years ago, but things have been almost as quiet as a shot from a Walther PPK with a silencer ever since then. Daniel Craig took on the role in 2006 with Casino Royale and passed the baton with...

A little more than a week after announcing Gemini Deep Research, Google is making the tool available to more people. As of today, the feature, part of the company’s paid Gemini Advanced suite, is available in every country and language where Google offers Gemini. In practice, that means Gemini...

AIOpsLab is an open-source framework designed to evaluate and improve AI agents for cloud operations, offering standardized, scalable benchmarks for real-world testing, enhancing cloud system reliability. The post AIOpsLab: Building AI agents for autonomous clouds appeared first on Microsoft...

Let’s explore what Salesforce does, the ways it helps strengthen customer relationships, and how the world’s #1 AI CRM empowers businesses to create limitless digital labor with autonomous agents.

Learn three key approaches to managing multi-agent systems responsibly with collective intelligence and collaborative governance.

Consider existing options and whether the task needs a conversational user experience before designing a new Agentforce action.

Think you know agents? Think again. We’ll debunk some common myths surrounding AI agents and show why these misconceptions could be holding your business back.

Awardees, who represent 10 universities, have access to Amazon public datasets, along with AWS AI/ML services and tools.

Agentforce tackles intricate queries and more on its own to transform your B2B customer service.

Recurrent drafting (referred as ReDrafter) is a novel speculative decoding technique developed and open-sourced by Apple for large language model (LLM)...

PromptWizard from Microsoft Research is now open source. It is designed to automate and simplify AI prompt optimization, combining iterative LLM feedback with efficient exploration and refinement techniques to create highly effective prompts in minutes. The post PromptWizard: The future of prompt...

Meta’s Llama collection of open large language models (LLMs) continues to grow with the recent addition of Llama 3.3 70B, a text-only instruction-tuned model....