Setup for Challenge

Prerequisite

  1. Prerequisite. Make sure you are using an RTX compatible GPU. We recommend the OS version Ubuntu 18.04/20.04 and NVIDIA driver version 525.60.11. Docker-based setup is not guaranteed to work on other OS or driver versions. We also verified the following driver versions are compatible with isaac sim:

    470.141.03
535.113.01

  2. Generate NVIDIA NGC API Key

    • Log in NVIDIA NGC. If you do not have an account, register one and log in.

    • Generate your NGC API key. You can refer to Generating API key.

    • Log into the NGC account on the instance

    docker login nvcr.io

    Type $oauthtoken for Username. Then paste your API key for Password. You should see Login Succeeded.

  3. Make sure NVIDIA container is properly installed. Check Installation guide.

Docker Setup

  1. Download codebase: git clone from the challenge branch.

git clone -b challenge https://github.com/arnold-benchmark/arnold.git

  1. Move to workspace and build docker image.

cd arnold/workspace

docker build -f Dockerfile -t "arnold" .

sudo apt install vagrant
vagrant up
vagrant ssh

Download Data and Assets

You can download data and assets from web browser or CLI. Unzip the downloaded zip files at /vagrant folder.

  • data_for_challenge_train.zip for training

  • data_for_challenge_val.zip for the dev phase

  • data_for_challenge_final.zip for the test phase

  • materials.zip for scene and object materials

  • sample.zip for assets used in ARNOLD

# for example, download from CLI
pip install gdown

cd /vagrant
# data_for_challenge_train.zip
gdown https://drive.google.com/uc?id=1gxSW3fFhGghJUpf_jiAy3iR_zrPy_MlJ
# data_for_challenge_val.zip
gdown https://drive.google.com/uc?id=1diLNQQcOGKEVkgOstkRbagbn_cVCKVIE
# data_for_challenge_final.zip
gdown https://drive.google.com/uc?id=1XKRxsByOwI_pYh09LUQ5wGLgzeGmL-KH
# materials.zip
gdown https://drive.google.com/uc?id=1CAT6pZfX0HqHKXU5qBdLeRZl_iY_XfOt
# sample.zip
gdown https://drive.google.com/uc?id=1jscZWcibfVXItbY1xZxRogA6Q8j3U60C

unzip data_for_challenge_train.zip
unzip data_for_challenge_val.zip
unzip data_for_challenge_final.zip
unzip materials.zip
unzip sample.zip

Model Evaluation

Please read these two evaluation scripts carefully to understand what happened:

cd /root/arnold/
bash eval_challenge_dev.sh ${checkpoint_file} # for dev phase
bash eval_challenge.sh ${checkpoint_file} # for test phase

To reproduce the results using the baseline checkpoint:

gdown https://drive.google.com/uc?id=1YuADlTFJZQc3AefULmzhZ9PrCWVjsEU2
bash eval_challenge_dev.sh peract_multi_clip_best.pth
bash eval_challenge.sh peract_multi_clip_best.pth

The first time starting the Isaac Sim takes long time, approximately 6 minutes (varying to the device configuration).

After evaluation is done, the scripts will generate an output folder. Zip the output folder (/root/arnold/output or /root/arnold/output_dev, depending on the phase) and submit it to EvalAI with CLI.

zip -r submission_dev.zip -r /root/arnold/output_dev/
zip -r submission_test.zip -r /root/arnold/output/

evalai challenge 2266 phase 4500 submit --file submission_dev.zip --large --public
evalai challenge 2266 phase 4501 submit --file submission_test.zip --large --public