About experiment setting

[jhyukjang]

Thx for your great work!
I have some questions about your code.

1. Did you set the random seed to 0 for all experiments?
2. I reproduced as you did (NUM_GPUS=2, BS_FITS_YOUR_GPU=2) like below

ARROW_ROOT=./datasets/mmimdb
NUM_GPUS=2
NUM_NODES=1
BS_FITS_YOUR_GPU=2
PRETRAINED_MODEL_PATH=./pretrained_weight/vilt_200k_mlm_itm.ckpt
EXP_NAME=mmimdb

python run.py with data_root=${ARROW_ROOT}
num_gpus=${NUM_GPUS}
num_nodes=${NUM_NODES}
per_gpu_batchsize=${BS_FITS_YOUR_GPU}
task_finetune_mmimdb
load_path=${PRETRAINED_MODEL_PATH}
exp_name=${EXP_NAME}

and I got 40.65 (paper: 42.66) on test set with same setting. Can I reproduce the paper's results without changing parameters like a learning rate or are there some optimized hyperparameters for each dataset?

[Trae1ounG]

Thx for your great work! I have some questions about your code.

1. Did you set the random seed to 0 for all experiments?
2. I reproduced as you did (NUM_GPUS=2, BS_FITS_YOUR_GPU=2) like below

ARROW_ROOT=./datasets/mmimdb NUM_GPUS=2 NUM_NODES=1 BS_FITS_YOUR_GPU=2 PRETRAINED_MODEL_PATH=./pretrained_weight/vilt_200k_mlm_itm.ckpt EXP_NAME=mmimdb

python run.py with data_root=${ARROW_ROOT} num_gpus=${NUM_GPUS} num_nodes=${NUM_NODES} per_gpu_batchsize=${BS_FITS_YOUR_GPU} task_finetune_mmimdb load_path=${PRETRAINED_MODEL_PATH} exp_name=${EXP_NAME}

and I got 40.65 (paper: 42.66) on test set with same setting. Can I reproduce the paper's results without changing parameters like a learning rate or are there some optimized hyperparameters for each dataset?

Hi.How many epochs do you trained?

[LinnHY]

same problem.Hope the author can answer and help us

[herkerser]

same problem, it would be helpful if the authors could provide more details on getting the results on paper

[singhayush27]

lots of version issue how did you handle guys?

[Xfavor]

Is the code complete, and it can't be run directly using the training command given by the author？