SmolVLA: Hugging Face's Compact Vision-Language-Action Model for Affordable Robotics

 Hugging Face has introduced SmolVLA, a compact and efficient Vision-Language-Action (VLA) model designed to democratize robotics by enabling robust performance on consumer-grade hardware. With only 450 million parameters, SmolVLA achieves competitive results compared to larger models, thanks to its training on diverse, community-contributed datasets.

Bridging the Gap in Robotics AI

While large-scale Vision-Language Models (VLMs) have propelled advancements in AI, their application in robotics has been limited due to high computational demands and reliance on proprietary datasets. SmolVLA addresses these challenges by offering:

• Compact Architecture: A 450M-parameter model that balances performance and efficiency.

• Community-Driven Training Data: Utilization of 487 high-quality datasets from the LeRobot community, encompassing approximately 10 million frames.

• Open-Source Accessibility: Availability of model weights and training data under the Apache 2.0 license, fostering transparency and collaboration.

Innovative Training and Annotation Techniques

To enhance the quality of training data, the team employed the Qwen2.5-VL-3B-Instruct model to generate concise, action-oriented task descriptions, replacing vague or missing annotations. This approach ensured consistent and informative labels across the diverse datasets.

Performance and Efficiency

SmolVLA demonstrates impressive capabilities:

• Improved Success Rates: Pretraining on community datasets increased task success on the SO100 benchmark from 51.7% to 78.3%.

• Asynchronous Inference: Decoupling perception and action prediction from execution allows for faster response times and higher task throughput.

• Resource-Efficient Deployment: Designed for training on a single GPU and deployment on CPUs or consumer-grade GPUs, making advanced robotics more accessible.

Getting Started with SmolVLA

Developers and researchers can access SmolVLA through the Hugging Face Hub:

• Model Repository: lerobot/smolvla_base
• Technical Report: SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics 

By offering a compact, efficient, and open-source VLA model, SmolVLA paves the way for broader participation in robotics research and development, fostering innovation and collaboration in the field.

NVIDIA Launches Cosmos-Reason1: Pioneering AI Models for Physical Common Sense and Embodied Reasoning

 NVIDIA has unveiled Cosmos-Reason1, a groundbreaking suite of AI models aimed at advancing physical common sense and embodied reasoning in real-world environments. This release marks a significant step towards developing AI systems capable of understanding and interacting with the physical world in a human-like manner.

Understanding Cosmos-Reason1

Cosmos-Reason1 comprises multimodal large language models (LLMs) trained to interpret and reason about physical environments. These models are designed to process both textual and visual data, enabling them to make informed decisions based on real-world contexts. By integrating physical common sense and embodied reasoning, Cosmos-Reason1 aims to bridge the gap between AI and human-like understanding of the physical world. 

Key Features

• Multimodal Processing: Cosmos-Reason1 models can analyze and interpret both language and visual inputs, allowing for a comprehensive understanding of complex environments.

• Physical Common Sense Ontology: The models are built upon a hierarchical ontology that encapsulates knowledge about space, time, and fundamental physics, providing a structured framework for physical reasoning. 

• Embodied Reasoning Capabilities: Cosmos-Reason1 is equipped to simulate and predict physical interactions, enabling AI to perform tasks that require an understanding of cause and effect in the physical world.

• Benchmarking and Evaluation: NVIDIA has developed comprehensive benchmarks to assess the models' performance in physical common sense and embodied reasoning tasks, ensuring their reliability and effectiveness. 

Applications and Impact

The introduction of Cosmos-Reason1 holds significant implications for various industries:

• Robotics: Enhancing robots' ability to navigate and interact with dynamic environments. 

• Autonomous Vehicles: Improving decision-making processes in self-driving cars by providing a better understanding of physical surroundings.

• Healthcare: Assisting in the development of AI systems that can comprehend and respond to physical cues in medical settings.

• Manufacturing: Optimizing automation processes by enabling machines to adapt to changes in physical environments.

Access and Licensing

NVIDIA has made Cosmos-Reason1 available under the NVIDIA Open Model License, promoting transparency and collaboration within the AI community. Developers and researchers can access the models and related resources through the following platforms:

• GitHub Repository: Cosmos-Reason1 on GitHub

• Hugging Face Model Page: Cosmos-Reason1-7B on Hugging Face

• NVIDIA Developer Portal: NVIDIA Cosmos for Developers