| --- |
| license: mit |
| --- |
| # GeLinear |
|
|
| GeLinear is the implementation of [LoLCATs](https://arxiv.org/pdf/2410.10254), but with Gemma 2 model. |
| Unlike the original LoLCATs approach that linearizes all attention layers, I focuses on linearizing only the global attention layer, while retaining Gemma 2’s built-in Sliding Window Attention (SWA) for local context (since the time complexity for this already scaled linearly with sequence length). |
|
|
| To run the model: |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer |
| model = AutoModelForCausalLM.from_pretrained("jacksonkek/GeLinear",trust_remote_code=True,torch_dtype=torch.bfloat16,device_map="sequential") |
| tokenizer = AutoTokenizer.from_pretrained("jacksonkek/GeLinear") |
| |
| x = "tell me a joke" |
| input_text = [{"role":"user","content":x}] |
| input_ids = tokenizer.apply_chat_template(input_text, add_generation_prompt=True,return_tensors="pt").to("cuda") |
| text_streamer = TextStreamer(tokenizer) |
| _ = model.generate(input_ids, streamer = text_streamer, do_sample=False,max_new_tokens = 8192) |
| |
| ``` |
