#CUEMP-VO (Please check the cuemp branch.)

Initial Code Release: This repo currently provides a single GPU implementation of our monocular, stereo, and RGB-D SLAM systems. It currently contains demos, training, and evaluation scripts.

Requirements

To run the code you will need ...

• Inference: max 10G(GPU MEM).

• Training: Training requires a GPU with at least 24G of memory.

Getting Started

1. Clone the repo (you also need to clone lietorch and eigen in thirdparty)

git clone https://github.com/UESTC-nnLab/CUEMP.git

2. Creating a new anaconda environment using the provided .yaml file. Use environment_novis.yaml to if you do not want to use the visualization

conda env create -f environment.yaml
pip install evo --upgrade --no-binary evo
pip install gdown

3. Compile the lietorch CUDA extensions (takes about 6 minutes)

python setup.py install

4. Compile the LCU(learnable correspondence uncertainty + deformable samplingg) CUDA extensions (takes about 8 minutes)

cd ops_LCU
python setup.py install

Demos (we provide the weight named demo.pth in the master branch, please use the cuemp branch to test it!)

2. Download some sample videos using the provided script.

./tools/download_sample_data.sh

Run the demo on any of the samples (all demos can be run on a GPU with 12G of memory). While running, press the "s" key to increase the filtering threshold (= more points) and "a" to decrease the filtering threshold (= fewer points). To save the reconstruction with full resolution depth maps use the --reconstruction_path flag.

python demo.py --imagedir=data/abandonedfactory --calib=calib/tartan.txt --stride=2

python demo.py --imagedir=data/sfm_bench/rgb --calib=calib/eth.txt

python demo.py --imagedir=data/Barn --calib=calib/barn.txt --stride=1 --backend_nms=4

python demo.py --imagedir=data/mav0/cam0/data --calib=calib/euroc.txt --t0=150

python demo.py --imagedir=data/rgbd_dataset_freiburg3_cabinet/rgb --calib=calib/tum3.txt

Running on your own data: All you need is a calibration file. Calibration files are in the form

fx fy cx cy [k1 k2 p1 p2 [ k3 [ k4 k5 k6 ]]]

with parameters in brackets optional.

Evaluation

TartanAir (Mono + Stereo)

Download the TartanAir dataset using the script thirdparty/tartanair_tools/download_training.py and put them in datasets/TartanAir

./tools/validate_tartanair.sh --plot_curve            # monocular eval
./tools/validate_tartanair.sh --plot_curve  --stereo  # stereo eval

EuRoC (Mono + Stereo)

Download the EuRoC sequences (ASL format) and put them in datasets/EuRoC

./tools/evaluate_euroc.sh                             # monocular eval
./tools/evaluate_euroc.sh --stereo                    # stereo eval

TUM-RGBD (Mono)

Download the fr1 sequences from TUM-RGBD and put them in datasets/TUM-RGBD

./tools/evaluate_tum.sh                               # monocular eval

ETH3D (RGB-D)

Download the ETH3D dataset

./tools/evaluate_eth3d.sh                             # RGB-D eval

Training

First download the TartanAir dataset. The download script can be found in thirdparty/tartanair_tools/download_training.py. You will only need the rgb and depth data.

python download_training.py --rgb --depth

Note: On the first training run, covisibility is computed between all pairs of frames. This can take several hours, but the results are cached so that future training runs will start immediately.

python train.py --datapath=<path to tartanair>

To 3DGS

Fine-tune the LGU/DROID reconstruction results using the combination of 'executeSlam.py' and 'pc2mesh.py' in 'to3DGS' directory and output the final mesh based scene reconstruction. Referring to project Splat-SLAM, SplaTAM and GO-SLAM.

Acknowledgements

Data from TartanAir was used to train our model. We additionally use evaluation tools from evo and tartanair_tools.

Thank you DROID and Efficient-KAN for inspiring our work.