Install a chatbot locally (Vicuna, Oobabooga for Windows)
Hello friends
Today I want to show you how we can have our private chatbot installed locally. For people that are in the writing business I think a chatbot is a nice tool to have, to brainstorm with it and to speed up the creative process.. the tool I want to show you is Vicuna. Vicuna is an open source chatbot developed by UC Berkeley, CMU, Stanford, MBZUAI, and UC San Diego, for the tech stuff I send you to their website: https://vicuna.lmsys.org/ , I am more interested to have it up and running.
First thing first we have to download a .zip file from this URL: https://github.com/oobabooga/text-generation-webui, just scroll the page and you will find the .zip files for windows, Linux and MacOS. This is a kind of one click install but we have to make sure to have Git properly installed, it is not a big deal but we have to be al ittle bit careful and committed when installing this kinds of apps.
I have extracted the oobabooga-windows.zip files and in the folder appears like this:
double click on install.bat
On input type A if you have an Nvidia GPU or B if you don’t have it. Running on CPU will be considerably slower.
Wait some time
When the installer finishes the oobabooga-windows folder appears like this:
Now we have to run download-model.bat
You can choose one of them as a basic model (I have choosen A).
Now starts the “difficoult” part: go to huggingface and download the model you want to use. in this case I have started with this: vicuna-13b-4bit-128g.safetensors
If you have Git properly installed from the YourDrive:\oobabooga-windows\text-generation-webui\models the command is: CMD
git lfs install git clone https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g GIT_LFS_SKIP_SMUDGE=1
the last line of code is to tell Git to install all the pointer files except the big vicuna-13b-4bit-128g.safetensors that you will copy in the folder created in YourDrive:\oobabooga-windows\text-generation-webui\models.
In the new update installing a new model is way easier. I give you instructions on how to update Oobabooga and how to download new models in my very next post
it appears something like this:
and inside the vicuna folder you will find something like this:
which are all the files contained in the vicuna-13b-4bit-128g.safetensors except for the big .safetensor file. Without Git we have to create the folders manually and copy the files one by one.to other models change the name of the recipient (https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g in this case) with the recipient of your choice
My folder is appearing different from your because I have some models installed, you will have a single folder appearing there.
The last thing I do is to change this line of the start-webui.bat
call python server.py --auto-devices --chat
with this:
call python server.py --auto-devices --extensions api --wbits 4 --groupsize 128
save it and fire it up 🙂
choose your model:
The KoboldAI compatible api is to run oobabooga whichever model you want to run with it with interfaces like TavernAI (more details in the next post) .
When we copy the http address on our browser this is what appears:
before we even start we go on the top page’s menu
and we set this value to 800 to get longer responses
then I generate a short story about an old dog
You
Context:
Chiharu Yamada’s Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.
You: So how did you get into computer engineering?
Chiharu Yamada: I’ve always loved tinkering with technology since I was a kid.
You: That’s really impressive!
Chiharu Yamada: *She chuckles bashfully* Thanks!
You: So what do you do when you’re not working on computers?
Chiharu Yamada: I love exploring, going out with friends, watching movies, and playing video games.
You: What’s your favorite type of computer hardware to work with?
Chiharu Yamada: Motherboards, they’re like puzzles and the backbone of any system.
You: That sounds great!
Chiharu Yamada: Yeah, it’s really fun. I’m lucky to be able to do this as a job.
From this tab we can create our characters. This model is not censored so you can talk more or less of whatever you want and try to make some skimpy characters.
I hope I have been clear enough and I hope you have fun using your personal chatbot
hugs
Alessandro