The most substantial recent progress in NLP has focused on text understanding and generation. Yet, humans rely on additional contexts to communicate efficiently, such as speech, vision, situational context (time and space), commonsense knowledge, social and cultural norms. LLMs are still seriously limited in dealing with extra-linguistic contexts. Enhancing LLM-based NLP models with extra-linguistic contexts could improve their applicability to real-world usages. Such models will be able to adapt their outputs based on time, location, and the user’s background. Endowing models with commonsense reasoning abilities will additionally improve their robustness and help them address unknown inputs reasonably and consistently.

Commonsense Reasoning

robot cutting the branch it's sitting on

We all share basic knowledge and reasoning ability regarding causes and effects, physical commonsense (if you cut the branch you sit on, you will fall), and social commonsense (it's impolite to comment on someone's weight). Endowing machines with such commonsense is challenging. Such knowledge is too vast to collect from humans and often too trivial to be mentioned in texts.

Culturally-Aware NLP Models

COMET interprets 'dutch baby' literally

English LLMs have a narrow—Western, North American, or even US-centric lens, as a result of training on web text from mostly US-based users. Just like cross-culture human interactions can lead to miscommunications, users from diverse cultures that are interacting with LLM-based NLP tools may feel misunderstood and experience the tool as less useful. This is because culture plays a significant role in shaping our worldviews, beliefs, and communication styles. Lack of cultural awareness may lead to models perpetuating stereotypes and reinforcing societal inequalities, impeding their effectiveness for users from non-Western countries. We are working on evaluating the cultural awareness of LLMs, as well as enhancing LLMs with cultural competence.

Multimodal Models

An example meme

Language and perception are intertwined. Language understanding is often grounded in sensory perception, such as vision and speech. In turn, perception is shaped by language, which affects how we see and describe the world around us. Most NLP research is focused on language, with some recent work on vision and language. Models that incorporate multiple modalities can facilitate more human-like interactions with the user, leveraging non-verbal cues such as tone of voice, facial expressions, and gestures.