GitHub - KiwiGaze/Transformer-Incontext-Learning

Exploring In-Context Learning Capabilities of Transformer Models

Install the dependencies using Conda.

conda env create -f environment.yml
conda activate in-context-learning

For Shift in Data Distribution Part. Compare transformers to OLS and ridge. For training a transformer, we use the code here https://github.com/dtsip/in-context-learning. Once transformer are trained, specify the paths to their respective checkpoints in the YAML file (comparison_mlp_txformer.yaml) and run.

Run the following for no shift without noise:
```
python Comparisons_final.py --config confs/comparison_mlp_txformer.yaml
```
Run the following for mild shift without noise:
```
python Comparisons_final_m_2.py --config confs/comparison_mlp_txformer.yaml
```
Run the following for severe shift without noise:
```
python Comparisons_final_m_4.py --config confs/comparison_mlp_txformer.yaml
```
Run the following for no shift with noise:
```
python noised_Comparisons_final.py --config confs/comparison_mlp_txformer.yaml
```
Run the following for mild shift with noise:
```
python noised_Comparisons_final_m_2.py --config confs/comparison_mlp_txformer.yaml
```
Run the following for severe shift with noise:
```
python noised_Comparisons_final_m_4.py --config confs/comparison_mlp_txformer.yaml
```
For Shift in Data Distribution part, you can run data_distribution.ipynb for simplicity. Please open /Data_Distribution for more information.

Name		Name	Last commit message	Last commit date
Latest commit History 81 Commits
Data_Distribution		Data_Distribution
MoreComplexFuncPart		MoreComplexFuncPart
__pycache__		__pycache__
confs		confs
.DS_Store		.DS_Store
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Comparisons_final.py		Comparisons_final.py
Comparisons_final_m_2.py		Comparisons_final_m_2.py
Comparisons_final_m_4.py		Comparisons_final_m_4.py
LICENSE.txt		LICENSE.txt
LICENSE_README.md		LICENSE_README.md
Linear_regression_prompts_data.py		Linear_regression_prompts_data.py
README.md		README.md
base_models.py		base_models.py
baselines_ols_ridge.py		baselines_ols_ridge.py
curriculum.py		curriculum.py
data_distribution.ipynb		data_distribution.ipynb
environment.yml		environment.yml
eval.py		eval.py
eval_mlps.py		eval_mlps.py
iclmlp_cln_jmtd.py		iclmlp_cln_jmtd.py
mlps.py		mlps.py
models.py		models.py
noised_Comparisons_final.py		noised_Comparisons_final.py
noised_Comparisons_final_m_2.py		noised_Comparisons_final_m_2.py
noised_Comparisons_final_m_4.py		noised_Comparisons_final_m_4.py
samplers.py		samplers.py
schema.py		schema.py
schema_comp.py		schema_comp.py
tasks.py		tasks.py
train.py		train.py
train_mlps.py		train_mlps.py