Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
rthapa84 authored Jun 6, 2024
1 parent b06ff08 commit 3942014
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ The image captures not just a moment, but a story of skill, courage, and the joy

## 📊 Dataset

I will release it soon on HF hub.
We will release it soon on HF hub.

<a name="training"/>

Expand All @@ -147,14 +147,14 @@ I will release it soon on HF hub.
We adopt a two-stage training process.

### Stage 1
In this stage, we only train our projection layer, so that the model learns to map the embeddings from the vision encoder into the LLM space. The dataset mixture used in this stage is `stage1_dataset`, which contains short image and caption pairs.
In this stage, we only train our projection layer, so that the model learns to map the embeddings from the vision encoder into the LLM space.

```bash
sh train_dragonfly_stage1.sh
```

### Stage 2
In this stage, we train our vision encoder, projection layer, and LLM jointly on image and text data. Our training dataset mixture for this stage is provided in `stage2_dataset`. This dataset contains xx.xx% of text-only dataset as well. We also include a math dataset, given in `math_instruct`.
In this stage, we train our vision encoder, projection layer, and LLM jointly on image and text data.

```bash
sh train_dragonfly_stage2.sh
Expand Down

0 comments on commit 3942014

Please sign in to comment.