LEAP - fast finetuning

Proposals Round 10
#1

Name of Project: LEAP

Proposal in one sentence: Fast fine-tuning for models using a local minimum as initial weights

Description of the project and what problem it is solving: Currently, we made Textual Inversion (adding your own concept to Stable Diffusion) more than 10 times faster, released in the Thingy Discord bot, released open source on GitHub as well as a standalone project on github.com/peterwilli/sd-leap-booster (remove the space after the dot because I couldn’t upload more than 2 links). The advantages of such model are immense, ranging from increasing profit margins on finetuning services or lowering the price to extending it to language models alike.

Grant Deliverables:

  • Researching and releasing LEAP+Lora, integrating full pivotal tuning with dreambooth and textual inversion in one, at similar speeds, if not faster. In fact, it is already in research phase: https://github.com/peterwilli/sd-leap-booster/tree/lora-test-3
  • Research for different models i.e. language models

Squad

Squad Lead:

  • Twitter: @codebuffet
  • Discord: EmeraldOdin#1991

Additional notes for proposals

Thanks to a generous person lending his GPU rig, I managed to accelerate my work, and it serves as a living proof of what more could be done in the future.
Thanks for reading. Love to all, and good luck to everyone in this round and the next.

6 Likes
#2

It would be great to see a diagram of what LEAP is doing and its architecture. It is unclear to me if LEAP has sets of weights that is has prepared offline or not. Between “using a local minimum as initial weights” and diving into all the source code, it would be nice to have an abstract/simple understanding of what the insights/innovations are. The project seems very interesting!

3 Likes
#3

Awesome seeing you here! So is your goal just faster pivotal tuning with lora+your embeddings? Also, you might have thought of this but do you have plans to work with multi-token embeddings?

#4

Hey isamu. Yeah that’s correct. I’m currently doing just that on Github! I don’t have many votes yet, so if you want to see it, feel free :slight_smile:

#5

Thanks! I have a flowchart atm, I hope it answers some questions.

LEAP 1.0 flowchart.drawio
LEAP 1.0 flowchart.drawio971×1061 63.3 KB

#6

Hi, I’m not able to understand how LEAP works from our exchanges but that it not the goal of the Algovera funding. If you want to get more people involved then I think it will help if you have a clear explanation of what is original in how LEAP works. But again that is not the goal for now. Good luck with the project!

#7

Hi Mark,

I’m not really sure how better to explain how LEAP is working. It’s out there, and people are using it. Under the hood, the novel part is in the training and the way data is presented, although with the lora version that likely changes. I cannot really think of a way to explain more than I did in my presentation, but not sure if you’ve seen that! If not, I can repeat some of the stuff here.

#8

I did see the presentation, thanks (I believe you showed that diagram). I think it would help to have a diagram of what is going on inside the LEAP box in the diagram you shared above. It is hard to understand what the limitations/potential is without understanding that. But you might choose to let people who dive into the code understand that.