[WIP] EoRA

[Qubitium]

• add adapter property to QuantizeConfig + EoRA lora adapter/config

Sample quantization config format with EoRA extension:

quant_config = QuantizeConfig(
   bits=4,
   sym=False,
   group_size=128,
   adapter=EoRA(rank=128),
}

[nbasyl]

Hi @Qubitium, @hutm is the person who developed the ExLlama EoRA kernel and will assist with the inference and validation tasks once I complete the first milestone. Could you grant him the write access to this branch? Thanks!

[Qubitium]

Hi @Qubitium, @hutm is the person who developed the ExLlama EoRA kernel and will assist with the inference and validation tasks once I complete the first milestone. Could you grant him the write access to this branch? Thanks!

Done. Write access invite sent to @hutm.

[nbasyl]

Hi @Qubitium, I am having error from line 186 in config.py when trying to import QuantizeConfig, the error is as follow:

Do you know how to resolve this?

I am using python3.10 and have installed all the required libraries following the readme

[Qubitium]

Hi @Qubitium, I am having error from line 186 in config.py when trying to import QuantizeConfig, the error is as follow: Do you know how to resolve this?

I am using python3.10 and have installed all the required libraries following the readme

@nbasyl Let me get slack installed so we can converse on slack. I should be able to fix this. This is strange error, almost as if our Type hint code is wrong. Not sure if we have python 3.10 compat bug.