Running Large Language Models Locally

My notes from installing Ollama and using open-source large language models locally.

These notes are taking inspiration from the great video done by Alex Ziskind.

Downloads

Note: These instructions were done on a Windows machine running WSL 2. They should also work for macOS. If you’re running a Windows machine and don’t have WSL installed, you can find more information about that here.

Open WebUI

This is the ChatGPT-like GUI that we’ll use to interact with the models. Clone the repository here via

git clone https://github.com/open-webui/open-webui.git

Ollama

This is where the open-source models are sourced. Go to the Ollama webpage and download the Linux version. There is a slight difference here between macOS and Linux; see the Alex Ziskind video for the macOS-specific instructions for this step. For the Linux version, go to the home directory in your terminal and run (as of this writing, check the Ollama website in case this command changed)

curl -fsSL https://ollama.com/install.sh | sh

To check that Ollama is running, just type ollama in your terminal, then open a web browser and enter localhost:11434. You should get a message that says Ollama is running.

Get Models via Ollama

Now that we have Ollama installed, we can use the Ollama website to download models. Just go to the Ollama website and navigate to “Models”. Or just click here. Each model will have instructions on it’s size. For example, if I want to download the 1b variant of llama3.2, I would select the 1b version in the dropdown menu, then run

ollama pull llama3.2:1b

which would install the model. To run the model, I would simply type

ollama run llama3.2:1b

We can now run any of the models we have installed from the command line. For example, if I installed llama3.2 via ollama pull llama3.2, I could run it with

ollama run llama3.2

which then results in an interactive prompt that we can interact with:

~$ ollama run llama3.2
>>> Send a message (/? for help)

Open WebUI

Anaconda

Go to the Anaconda download website and download the Linux x86 installer. This will download a file that ends in .sh. Copy this from your Downloads folder to your Linux /home directory. You can delete the corresponding :Zone.Identifier file.

To install this file, open the Terminal and type

bash Anaconda<tab>.sh

Since the Anaconda file name is really long, just type bash Anaconda, hit tab a couple times until the file completes itself, then hit Enter/Return. Accept the defaults.

It might not automatically load and give you an error saying that conda isn’t a recognized command. If that happens, do the following in a terminal:

Open up your ~/.bashrc file:

nano ~/.bashrc

Scroll down to the bottom of the file using the down arrow key and paste the following line:

export PATH="/home/username/anaconda3/bin:$PATH"

where username is your username.

Save the changes with Ctrl+X, y, Enter. Then type

source ~/.bashrc

To make the changes take effect. The conda command should now work.

Alex Ziskind used Anaconda to create some environments to manage his local Open WebUI installation, so I’ll do the same here. Let’s create a new Anaconda environment named openwebui (using Python v3.11):

conda create --name openwebui python=3.11

Then activate it

conda activate openwebui

Which will then make our terminal look something like this:

(openwebui) josh@Josh-XPS-15:~$

Backend Environment

Navigate to the open-webui/backend folder, then install the requirements

(openwebui) josh@Josh-XPS-15:~/open-webui/backend$ pip install -r requirements.txt -U

Frontend Environment

This is where we make all of the pretty GUI things happen.

Install node via

sudo apt install nodejs

and npm via

sudo apt install npm

Install the requirements in package.json in the open-webui directory with npm:

npm i

Then we need to build everything. I ran into some memory issues and found that allocating more memory fixed the problem. My machine has 32GB of RAM, so I allocated 16GB with the following commands:

NODE_OPTIONS="--max-old-space-size=16384" npm run build

Now we have both our frontend and backend enviroments installed!

Using Open WebUI

Navigate to the backend directory and type

bash start.sh

Now open a web browser and nagivate to localhost:8080. You’ll need to sign up, but the credentials will be stored locally on this machine.

Forgetting Open WebUI Credentials

In the open-webui folder, navigate to backend/data. There should be a webui.db file. Delete that file, restart/rerun bash start.sh, and you’ll be good to go!

Image Generation

The instructions for this portion are from Alex Ziskind’s great tutorial.

We can also implement Stable Diffusion using Automatic1111. Clone the repository via

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

We’ll make a new Anaconda environment for all of these requirements, like we did earlier and for the same reason:

conda create --name a1111 python=3.11

and activate this new environment. Now we can install the requirements after navigating into the created folder, which should be stable-diffusion-webui:

pip install -r requirements.txt -U

Once this is done, run the start.sh file. This will take some time, as a stable diffusion model will need to be downloaded (~4GB) so that we can actually generate models:

bash webui.sh

This will automatically launch a browser, but in order for it to interact with the nice GUI that we have earlier, we need to change some settings, so we can close it. Run Open WebUI using the openwebui Anaconda environment from earlier, and click on your name in the bottom left, Admin Panel, Images, Image Generation to On, Image Generation Engine: Automatic1111, base URL should be http://localhost:7860.

To ensure that the API is available, do the following instead in the a1111 environment:

bash webui.sh --api

Now go back to Open WebUI and refresh. Go back to the settings with automatic1111 that I mentioned earlier. Set a default model, should only have one option. Now create a prompt, and there will be a new button on the bottom that allows for image creation.

Getting Better Image Models

Go to Civit AI. Go to Models, then Filters, then select “Chceckpoint”. Can then sign in and download any of the models. After they’re downloaded, find the same folder as where the stable-diffusion-webui directory is. Here, there is a models folder, and within that there is a Stable-diffusion folder. Put your downloaded model in this folder.

Go back to where you’re running stable-diffusion-webui, stop it with Ctrl+C, then re-launch it. It’ll detect that there is a new model, and now you can go back to the settings in the ChatGPT GUI, change the default model, and have fun!

Running Everything

Before we get started, since we’re working in a terminal, here are some helpful commands:

Command	Description
ls	Lists all files in the current directory
cd	“Change directory”; start typing then press tab for autocomplete options
cd ../	Move up one directory; if in /home/josh, cd ../ moves you to /home
pwd	“Print working directory”; prints the file path of where you are
Ctrl+C	Stops the currently running program; might need to press multiple times

Once everything is installed, here are the commands you need to get everything running. Assuming you start in your home directory (you can get there from anywhere by typing ~/).

It’s usually a good idea to make sure everything is updated, so let’s do that first:

sudo apt update

sudo apt upgrade -y

And then remove anything that might not be needed anymore:

sudo apt autoremove

Start the Open WebUI interface by activating your openwebui Anaconda environment:

conda activate openwebui

Now navigate to the open-webui/backend folder:

cd open-webui/backend

Start the .sh file by typing:

bash start.sh

Open up a web browser, and enter the following URL:

http://localhost:8080

If that URL doesn’t work, try

http://0.0.0.0:8080

These URLs take you to the ChatGPT-like interface for your models.

Open up a new Terminal tab, and activate the stable diffusion Anaconda environment:

conda activate a1111

Now navigate to the stable-diffusion-webui with

cd stable-diffusion-webui

And start the stable diffusion interface with

bash webui.sh --api

This will open up a web browser automatically. You can also navigate to it yourself by opening up a web browser and typing

http://127.0.0.1:7860/

or

http://localhost:7860/

This web browser interface gives you more fine-grained control over the image generation process, but using it is optional. You can find more info on it here. You can also go to the ChatGPT interface and interact with text alone.

To stop everything, go to both tabs opened in your terminal and type Ctrl+C. You might need to type it a few times before the programs quit.

Have fun creating! :)