Hello friends

Today I want to show you how we can have our private chatbot installed locally. For people that are in the writing business I think a chatbot is a nice tool to have, to brainstorm with it and to speed up the creative process.. the tool I want to show you is Vicuna. Vicuna is an open source chatbot developed by UC Berkeley, CMU, Stanford, MBZUAI, and UC San Diego, for the tech stuff I send you to their website: https://vicuna.lmsys.org/ , I am more interested to have it up and running.

First thing first we have to download a .zip file from this URL: https://github.com/oobabooga/text-generation-webui, just scroll the page and you will find the .zip files for windows, Linux and MacOS. This is a kind of one click install but we have to make sure to have Git properly installed, it is not a big deal but we have to be al ittle bit careful and committed when installing this kinds of apps.

I have extracted the oobabooga-windows.zip files and in the folder appears like this:

double click on install.bat

On input type A if you have an Nvidia GPU or B if you don’t have it. Running on CPU will be considerably slower.

Wait some time

When the installer finishes the oobabooga-windows folder appears like this:

Now we have to run download-model.bat

You can choose one of them as a basic model (I have choosen A).

Now starts the “difficoult” part: go to huggingface and download the model you want to use. in this case I have started with this: vicuna-13b-4bit-128g.safetensors

If you have Git properly installed from the YourDrive:\oobabooga-windows\text-generation-webui\models the command is: CMD

git lfs install
git clone https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g
GIT_LFS_SKIP_SMUDGE=1

the last line of code is to tell Git to install all the pointer files except the big vicuna-13b-4bit-128g.safetensors that you will copy in the folder created in YourDrive:\oobabooga-windows\text-generation-webui\models.

In the new update installing a new model is way easier. I give you instructions on how to update Oobabooga and how to download new models in my very next post

it appears something like this:

and inside the vicuna folder you will find something like this:

which are all the files contained in the vicuna-13b-4bit-128g.safetensors except for the big .safetensor file. Without Git we have to create the folders manually and copy the files one by one.to other models change the name of the recipient (https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g in this case) with the recipient of your choice

My folder is appearing different from your because I have some models installed, you will have a single folder appearing there.

The last thing I do is to change this line of the start-webui.bat

call python server.py --auto-devices --chat

with this:

call python server.py --auto-devices --extensions api --wbits 4 --groupsize 128

save it and fire it up 🙂

choose your model:

The KoboldAI compatible api is to run oobabooga whichever model you want to run with it with interfaces like TavernAI (more details in the next post) .

When we copy the http address on our browser this is what appears:

before we even start we go on the top page’s menu

and we set this value to 800 to get longer responses

then I generate a short story about an old dog

You

Hello, can you write a short story about an old dog for me?

Assistant

Of course! Here’s a short story I came up with:

There was once an old dog named Max who lived a long and happy life with his family. He had been by their side through thick and thin, and they loved him more than anything in the world. As he grew older, however, Max began to slow down and have trouble getting around like he used to. His family did everything they could to make him comfortable, but they knew that one day they would lose him forever.

One day, when Max was lying on his bed feeling sad and lonely, he heard a knock at the door. When his family answered it, they were surprised to see a young man standing there with a guitar case in hand. “I’m here to play some music for your dear old dog,” the man said with a smile. “It’s said that dogs can understand us when we talk to them, and I want to give Max one last concert before he goes.”

Max’s family couldn’t believe what they were hearing, but they agreed to let the man come inside and play for their beloved pet. The man sat down next to Max and started strumming his guitar, singing songs of love and joy. Max closed his eyes and listened intently, wagging his tail as if he was enjoying every moment of it.

As the man played on, Max felt something stirring within him. It was as though he was being lifted out of his tired old body and transported back to happier times. For the first time in weeks, Max perked up and started wagging his tail again. His family watched in amazement as their old dog seemed to be dancing along to the music, as though he was young and full of energy once more.

When the man finished playing, Max lay back down on his bed, exhausted from all the excitement. But instead of feeling sad, he looked up at his family with a grateful heart, as though he was saying goodbye and thanking them for all the wonderful years they had shared together. And even though he passed away not long after that magical night, Max will always hold a special place in his family’s hearts, thanks to the gift of music and love that the kind stranger gave him.

Context:

Chiharu Yamada’s Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.

You: So how did you get into computer engineering?
Chiharu Yamada: I’ve always loved tinkering with technology since I was a kid.
You: That’s really impressive!
Chiharu Yamada: *She chuckles bashfully* Thanks!
You: So what do you do when you’re not working on computers?
Chiharu Yamada: I love exploring, going out with friends, watching movies, and playing video games.
You: What’s your favorite type of computer hardware to work with?
Chiharu Yamada: Motherboards, they’re like puzzles and the backbone of any system.
You: That sounds great!
Chiharu Yamada: Yeah, it’s really fun. I’m lucky to be able to do this as a job.

From this tab we can create our characters. This model is not censored so you can talk more or less of whatever you want and try to make some skimpy characters.

I hope I have been clear enough and I hope you have fun using your personal chatbot

hugs

Alessandro