• About
  • Advertise
  • Privacy & Policy
  • Contact
HK Businesswire
  • Home
  • News
    • All
    • Business
    • Politics
    • PR Newswire
    • Science
    • World

    White House national cyber director to leave post

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Why Brisbane Airport selected LG for its landmark digital infrastructure upgrade

    LG Electronics appoints Jeff Roh as President and Managing Director for Australia and New Zealand

    Denmark ratifies security agreement with US, Greenland

    NASA probe achieves light-day milestone

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • PR Newswire
  • Business
  • World
  • Entertainment
  • Sports
  • Tech
    • All
    • Apps
    • Gadget
    • Mobile
    • Startup

    Cathay Cargo Launches AI Compliance Tool to Streamline Global Trade Operations

    PCCW Global and Druid Technology Partner on Belt and Road IoT Connectivity

    Alipay Launches Domestic Full-Stack Intelligent Agent Platform in Hangzhou

    HKT Posts 4% Profit Rise Amid Steady Local Recovery and AI-Driven Demand

    Alipay Launches AI-Powered Version ‘Abao’ to Streamline Services

    Xiaohongshu Prepares Confidential Hong Kong IPO Filing

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
  • Feature
No Result
View All Result
  • Home
  • News
    • All
    • Business
    • Politics
    • PR Newswire
    • Science
    • World

    White House national cyber director to leave post

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

    Why Brisbane Airport selected LG for its landmark digital infrastructure upgrade

    LG Electronics appoints Jeff Roh as President and Managing Director for Australia and New Zealand

    Denmark ratifies security agreement with US, Greenland

    NASA probe achieves light-day milestone

    Trending Tags

    • Trump Inauguration
    • United Stated
    • White House
    • Market Stories
    • Election Results
  • PR Newswire
  • Business
  • World
  • Entertainment
  • Sports
  • Tech
    • All
    • Apps
    • Gadget
    • Mobile
    • Startup

    Cathay Cargo Launches AI Compliance Tool to Streamline Global Trade Operations

    PCCW Global and Druid Technology Partner on Belt and Road IoT Connectivity

    Alipay Launches Domestic Full-Stack Intelligent Agent Platform in Hangzhou

    HKT Posts 4% Profit Rise Amid Steady Local Recovery and AI-Driven Demand

    Alipay Launches AI-Powered Version ‘Abao’ to Streamline Services

    Xiaohongshu Prepares Confidential Hong Kong IPO Filing

    Trending Tags

    • Nintendo Switch
    • CES 2017
    • Playstation 4 Pro
    • Mark Zuckerberg
  • Feature
No Result
View All Result
HK Businesswire
No Result
View All Result
Home News Science

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

David Lee by David Lee
20 November 2026
in Science
0
0
SHARES
0
VIEWS
Share on FacebookShare on Twitter

Humans experience and interact with the world through sensory information. This could mean learning correspondences between the sound of bird chirps and the word “robin,” or being able to visually identify and follow your friend “Sarah” in a crowd of people. In many cases, people are able to draw on their latent “multimodal” knowledge — information across multiple data types, from audio to video to text or spoken word — to perform these tasks without many, or sometimes even any, examples.Multimodal foundation models of artificial intelligence are increasingly gaining capabilities that mimic this. However, their apparent multimodal fluency often belies their actual understanding. An MIT-IBM Computing Research Lab team, co-led by Jim Glass, senior research scientist in the MIT Computer Science and Artificial Intelligence Laboratory, has been investigating critical failures of state-of-the-art multimodal models and developing benchmarks and data-efficient methods to overcome these performance gaps with limited supervision. He speaks about this work in the following interview.Q: What were your goals for this project, and why would you opt to address AI models’ weaknesses over scaling to larger models?   A: This project has been going on for many years now, with roughly the same MIT-IBM team of speech and vision experts. We’ve wanted to explore the relationship between perception (things that you see and hear) and language, largely speaking: We have all of this absorbed knowledge and language skills, text-based, in the form of a large language model, so we wanted to know how you can relate them without using a whole bunch of annotated training data. Sometimes the models have seen certain phenomena, a latent representation of something that they have internally, but maybe due to subsequent fine-tuning or something, it disappears, so, developing techniques to try to make this capability re-emerge is a useful thing. As you study these publicly available models, you can see weaknesses even in big ones, so I think it’s important to try to address these limitations. If you had an unlimited budget, you could go out and create the annotations that you need, but it’s also good to be able to try to get these models to learn more on their own. We’ve done that by creating novel benchmarks that we can evaluate against, then developing techniques that perform better on these benchmarks, and getting these models to learn on their own. Throughout our research collaboration — and this has very much been a strong collaboration between MIT and IBM — we have continued to release public benchmarks for people to use, and that’s a lasting artifact.I think the overriding themes are multimodalities with language models, trying to address the shortcomings of these models, and trying to develop techniques that do not require a lot of annotated data to complete tasks better.Q: Can you walk us through how the project has evolved?A: When we first started this project, we were trying to build models that could learn correspondences automatically, not telling the machine anything more than, “This audio caption goes with this picture,” sort of like reading a picture book to a child who’s learning language. We would just show it these matching examples, and we’d show it mismatching examples. They could learn that the sequence of sounds that says “lighthouse” corresponds to a picture of a lighthouse, without any text being involved.Moving beyond speech was to include general audio, and then we started looking at video instead of still images. We would train the model, showing it matching video sequences with the corresponding audio stream and then mismatching ones, and make it reconstruct them. We would also mask some of the images or the audio, and have the model reconstruct it. The model would learn strong relationships; then, far beyond speech, it would learn what parrots sound like, or electric guitars versus acoustic guitars, for example.Then, when these large language models started coming out — and this was in the early days, before it had been done by the big companies — we started to learn correspondences between audio and language. Since that time, these large multimodal models have become available, and we’ve started looking at what some of the limitations of these models are, and are trying to improve upon them with these more or less supervised techniques.Q: What would you say are some recent highlights of your work?A: Our IPLoc research was looking at the fact that these state-of-the-art models, like GPT-40, can’t localize specific objects, given just a few-shot examples. We came up with this technique where we would train it with a pseudonym substitution. There’s an example there where we replaced “airplane” with “Elizabeth.” It forces the model to not fall back on things that it had previously learned about and pay more attention to the visual context.Then, we had an audio-based paper, OmniR1, that was looking at audio-large language models, so these are models that can listen to something and talk about them. One of the disturbing things we found is some of these benchmarks are multiple-choice answers. A lot of times, these models aren’t really listening to the sound, but they’re almost gaming the question by looking at the answer. What animal is roaring in this audio — like a chipmunk, a lion, an alligator, and a chickadee, or something? It’s not even really paying attention to the audio, [but] just going, “Oh, lion.” Part of the reason people do that is that it’s simple to evaluate, whereas if you asked for a text response, you’d somehow have to grade whether it’s accurate. Do you get humans to evaluate it, which can be expensive and time-consuming, or do you use some other language model to be the judge that can have its own set of issues as well? One way to address that is to evaluate these models on a variety of different kinds of benchmarks.We realized you could feed random noise, and they were still doing quite well, which shows that they’re just paying attention to the question. So, we tried to develop some methods to have it pay more attention to the audio and minimize this behavior.Q: Given resource constraints, what is your approach to adapting these large foundation models, and what blend of AI and algorithmic methods do you use that don’t require extensive annotated data?A: These days, we can’t create one of these very large foundation models ourselves. We don’t have the resources to do that, so we’re tending to use these available models and trying to adapt them, train them in a weakly supervised way, methods that don’t require as much data, like our work on test-time reinforcement learning.Our project was not monolithic, in the sense that we’re not all working on one thing. It’s more like a thousand flowers bloom, so there’s no one particular way that we’re approaching things.Q: In the near future, how do you see multimodal research evolving in your MIT-IBM research team?A: Over the years, we probably produced two dozen or so significant collaborative papers, and during this work, it was amazing to me that the machines could learn these correspondences without being told anything. No one had ever shown that machines could do it before we did that. Some of our early work combining audio with large language models was fun, too: seeing the model talk about what it heard and then having a conversation with it. Now, we are starting to take this for granted, but at the time, it was fun to see.Moving forward, for the coming year, with our MIT-IBM collaboration, we’re going to switch gears a little bit and focus more on what’s known as full-duplex spoken dialogue systems. These are models that are listening while they’re talking, so it’s more like how humans operate, and this is quite different from what we’ve been doing up until now, which is joint audio-visual work, which we will also continue. Now, we’ll be focusing on the speech side of things. These full-duplex models have suddenly become an interesting thing to look at, because they would move you away from the rigid turn-taking approach that is used right now when you’re talking to some of these models. Ultimately, I think they will be more natural to interact with. These full-duplex models are just starting to emerge, and some of the ones that are publicly available aren’t as good as the big, large language models. They can handle some of these low-level communication skills, but they’re still missing some things, like the concept of time, or knowledge depth. So, we’re working on trying to address some of these issues.This research has been conducted in coordination with the MIT-IBM Computing Research Lab teams of Rogerio Feris, Brian Kingsbury, and Samuel Thomas of IBM Research, and Hilde Kuehne of the University of Tuebingen, who is an MIT-IBM affiliated researcher. 

Tags: Science
David Lee

David Lee

Read More

New approach enables physically consistent AI models of chaotic systems

20 November 2026

Network Researchers Nail Down Theory of ‘Perfect Resilience’

20 November 2026
  • Trending
  • Comments
  • Latest
Newgen reports Revenue of ₹357 crores in Q1 FY’27, up 11% Q1 YoY; Profit After Tax at ₹63 crores, up 26% Q1 YoY

Newgen Software and the Asian Institute of Digital Finance Forge Strategic Partnership to Shape the Future of Enterprise AI

10 November 2026

World Winter Sports Expo 2026 Opens, Charting a New Decade for Ice and Snow and Outdoor Industries

17 October 2026

HK five-year plan offers UK long-term prospects: CE

10 November 2026

Volant Aerotech Secures Firm Order from Sichuan Airlines Low Altitude for VE25-100 Passenger eVTOLs

19 October 2026

White House national cyber director to leave post

20 November 2026
Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

20 November 2026
Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

20 November 2026
Why Brisbane Airport selected LG for its landmark digital infrastructure upgrade

LG Electronics appoints Jeff Roh as President and Managing Director for Australia and New Zealand

20 November 2026

Recent News

White House national cyber director to leave post

20 November 2026
Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

20 November 2026
Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

Kolmar BNH to Supply 16 ‘K-Wellness’ Products to Boots in the UK

20 November 2026
Why Brisbane Airport selected LG for its landmark digital infrastructure upgrade

LG Electronics appoints Jeff Roh as President and Managing Director for Australia and New Zealand

20 November 2026
HK Businesswire

Stay ahead with the latest insights on Hong Kong’s economy, finance, and investments. From market trends to policy updates, we bring you in-depth analysis and expert opinions.

📩 Subscribe to our newsletter for exclusive updates.
📍 Follow us on social media for real-time news.
📧 Contact us: info@hongkong-invest.com

Follow Us

  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2025 by HKBusinesswire.com

No Result
View All Result

© 2025 by HKBusinesswire.com