All About Apple’s New AI Model ReALM

Ahead of Apple’s flagship event WWDC 2024, the company has developed a new AI system called ReALM (Reference Resolution As Language Modeling) that understands ambiguous references in human speech and visual context for improving interactions with Siri, Apple's virtual assistant, making them more natural. By converting reference resolution into a language modeling problem, ReALM achieves major performance improvements compared to existing methods.

An Apple research paper highlights the importance of understanding context in human speech, especially ambiguous references like "they" or "that." This capability is crucial for conversational assistants to comprehend user requirements accurately. Additionally, enabling users to inquire about on-screen content enhances the hands-free experience with voice assistants.

Is ReALM Better than GPT-4?

The study demonstrates substantial enhancements over existing systems, with even the smallest model showing a gain of over 5% in handling on-screen references. Compared to GPT-4, Apple's ReALM models show similar performance with fewer parameters, making them more suitable for on-device use. By converting images into text, ReALM bypasses the need for advanced image recognition parameters, resulting in a smaller and more efficient model.

Some time ago, the company reportedly came up with MM1, a family of multimodal large language models with up to 30 billion parameters, designed to achieve state-of-the-art performance in pre-training and competitive results after supervised fine-tuning. It also acquired DarwinAI, a Canadian AI startup, indicating yet another move toward the company’s AI ambitions.