Skip to content

Commit

Permalink
Merge pull request #4 from togethercomputer/rthapa84-patch-2
Browse files Browse the repository at this point in the history
Update README.md
  • Loading branch information
rchalamala authored Jun 6, 2024
2 parents b4c5643 + 3942014 commit 63dc686
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ The image captures not just a moment, but a story of skill, courage, and the joy

## 📊 Dataset

I will release it soon on HF hub.
We will release it soon on HF hub.

<a name="training"/>

Expand All @@ -140,14 +140,14 @@ I will release it soon on HF hub.
We adopt a two-stage training process.

### Stage 1
In this stage, we only train our projection layer, so that the model learns to map the embeddings from the vision encoder into the LLM space. The dataset mixture used in this stage is `stage1_dataset`, which contains short image and caption pairs.
In this stage, we only train our projection layer, so that the model learns to map the embeddings from the vision encoder into the LLM space.

```bash
sh train_dragonfly_stage1.sh
```

### Stage 2
In this stage, we train our vision encoder, projection layer, and LLM jointly on image and text data. Our training dataset mixture for this stage is provided in `stage2_dataset`. This dataset contains xx.xx% of text-only dataset as well. We also include a math dataset, given in `math_instruct`.
In this stage, we train our vision encoder, projection layer, and LLM jointly on image and text data.

```bash
sh train_dragonfly_stage2.sh
Expand Down

0 comments on commit 63dc686

Please sign in to comment.