NVIDIA Isaac GR00T (Generalist Robot 00 Technology) is a research and development platform for building robot foundation models and data pipelines, designed to accelerate the creation of intelligent, adaptable robots.
Today, we announced the availability of Isaac GR00T N1.5, the first major update to Isaac GR00T N1, the world’s first open foundation model for generalized humanoid robot reasoning and skills. This cross-embodiment model processes multimodal inputs, including language and images, to perform manipulation tasks across diverse environments. It is adaptable through post-training for specific embodiments, tasks, and environments.
In this blog, we’ll demonstrate how to post-train (fine-tune) GR00T N1.5 using teleoperation data from a single SO-101 robot arm.
Technical Blog for GR00T N1.5: https://research.nvidia.com/labs/gear/gr00t-n1_5/
Now accessible to developers working with a wide range of robot form factors, GR00T N1.5 can be easily fine-tuned and adapted using the affordable, open-source LeRobot SO-101 arm.
This flexibility is enabled by the EmbodimentTag system, which allows seamless customization of the model for different robotic platforms, empowering hobbyists, researchers, and engineers to tailor advanced humanoid reasoning and manipulation capabilities to their own hardware.
Before proceeding to installation, please check if you satisfy the prerequisites.
git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
conda create -n gr00t python=3.10
conda activate gr00t
pip install --upgrade setuptools
pip install -e .[base]
pip install --no-build-isolation flash-attn==2.7.1.post4
Users can fine-tune GROOT N1.5 with any LeRobot dataset. For this tutorial, we will be using the table cleanup task as an example for fine-tuning.
It's important to note that datasets for the SO-100 or SO-101 are not in GR00T N1.5's initial pre-training. As a result of this, we will be training it as a new_embodiment.
For this tutorial, you can either begin by creating your own custom dataset by following these instructions (recommended) or by downloading the so101-table-cleanup dataset from Hugging Face. The --local-dir argument specifies where the dataset will be saved on your machine.
huggingface-cli download \
--repo-type dataset youliangtan/so101-table-cleanup \
--local-dir ./demo_data/so101-table-cleanup
The modality.json file provides additional information about the state and action modalities to make it "GR00T-compatible". Copy over the getting_started/examples/so100_dualcam__modality.json to the dataset <DATASET_PATH>/meta/modality.json using this command:
cp getting_started/examples/so100_dualcam__modality.json ./demo_data/so101-table-cleanup/meta/modality.json
NOTE: For a single-camera setup like so100_strawberry_grape, run:
cp getting_started/examples/so100__modality.json ./demo_data/<DATASET_PATH>/meta/modality.json
After these steps, the dataset can be loaded using the GR00T LeRobotSingleDataset class. An example script for loading the dataset is shown here:
python scripts/load_dataset.py --dataset-path ./demo_data/so101-table-cleanup --plot-state-action --video-backend torchvision_av
Fine-tuning GR00T N1.5 can be executed using the Python script, scripts/gr00t_finetune.py. To begin finetuning, execute the following command from your terminal:
python scripts/gr00t_finetune.py \
--dataset-path ./demo_data/so101-table-cleanup/ \
--num-gpus 1 \
--output-dir ./so101-checkpoints \
--max-steps 10000 \
--data-config so100_dualcam \
--video-backend torchvision_av
Tip: The default fine-tuning settings require ~25G of VRAM. If you don't have that much VRAM, try adding the --no-tune_diffusion_model flag to the gr00t_finetune.py script.
Once the training is complete and your fine-tuned policy is generated, you can visualize its performance in an open-loop setting by running the following command:
python scripts/eval_policy.py --plot \
--embodiment_tag new_embodiment \
--model_path ./so101-checkpoints \
--data_config so100_dualcam \
--dataset_path ./demo_data/so101-table-cleanup/ \
--video_backend torchvision_av \
--modality_keys single_arm gripper
Congratulations! You have successfully finetuned GR00T-N1.5 on a new embodiment.
After successfully fine-tuning and evaluating your policy, the final step is to deploy it onto your physical robot for real-world execution.
To connect your SO-101 robot and begin the evaluation, execute the following commands in your terminal:
python scripts/inference_service.py --server \
--model_path ./so101-checkpoints \
--embodiment-tag new_embodiment \
--data-config so100_dualcam \
--denoising-steps 4
python getting_started/examples/eval_lerobot.py \
--robot.type=so100_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_awesome_follower_arm \
--robot.cameras="{ wrist: {type: opencv, index_or_path: 9, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 15, width: 640, height: 480, fps: 30}}" \
--policy_host=10.112.209.136 \
--lang_instruction="Grab pens and place into pen holder."
Since we finetuned GRO0T-N1.5 with different language instructions, the user can steer the policy by using one of the task prompts in the dataset such as:
"Grab tapes and place into pen holder".
For detailed step-by-step tutorial, please check out: https://github.com/NVIDIA/Isaac-GR00T/tree/main/getting_started
Ready to elevate your robotics projects with NVIDIA's GR00T N1.5? Dive in with these essential resources:
Stay up-to-date with the latest developments by following NVIDIA on Hugging Face.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。