CodeLlama for DevOps massive productivity hack.
Codellama LLM for DevOps Using the OpenWebUI
Well, it happened. Thanks to quantized models and the specs of a Raspberry Pi, it is possible ( but not advisable) to get LLM working on a Raspberry Pi's 4B 8GB or RAM. I did some testing and got about 0.5 tokens per second on my Raspberry Pi. I will be getting a Pi 5 ( completed doubled the speed to just over 1tps). I ran it on My MacBook M2, and it was closer to 27 tokens per second, much faster.
Codellama 7B/13B is a game changer for DevOps Productivity as it can do everything from Switch to configuration examples, Ansible playbooks, Terraform, and Python code. I was able to generate solid working examples of all of the above. Still, with it being very slow, it's helpful because you can ask it to run and get an example of code or configuration. Come back in 10-20 minutes. It has it. You can do it while in a meeting or working on something else. I had some doubts, like it was talking to a backend or something, so I shut off my wifi and could still generate all the examples I listed above.
Here is a screen grab of what I asked it to do via the open-web UI.
The results were as effective as searching for an example on many online sites and search engines.
Installing on my Macbook was as simple as going to https://ollama.com/, downloading the app simple install, and then telling it to run the codellama:7b or 13b models.
ollama run codellama:13b
Here's an Ansible task that copies the contents of a local file, `file2.txt`, to a remote site at `/usr/local/`
using the `copy` module:
```yaml
- name: Copy file2.txt to remote site under /usr/local/
copy:
src: "{{ lookup('file', 'file2.txt') }}"
dest: "/usr/local/"
owner: root
group: root
mode: u+rw,g+r,o-w
```
Explanation:
* `name` is a descriptive label for the task.
* `copy` specifies that we want to use the `copy` module.
* `src` points to the local file we want to copy.
* `dest` specifies the remote path where we want to copy the file.
* `owner`, `group`, and `mode` specify the ownership, group membership, and permissions for the copied file on the
remote host. The values specified here are the defaults if no other values are provided.
Note that this task assumes you have already configured your Ansible inventory to include the target hosts and have
access to them via SSH.
Let's discuss the setup in K8s for Olama and OpenWebui. I am running everything in my ARM-based K3s cluster, Rasberry Pi. For details, see my other blog post about the setup. My YAML is included to get things up and running on my cluster, or it should work on any Linux machine running K3s.
---
apiVersion: v1
kind: Pod
metadata:
name: ollama-pod
labels:
app: ollama
spec:
containers:
- name: ollama
image: ollama/ollama:latest
resources:
requests:
memory: "4Gi"
cpu: "3"
limits:
memory: "7Gi"
cpu: "4"
ports:
- containerPort: 11434
nodeName: pc4 #name of rasberry pi you want to dedicate this too
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: ollama-service
spec:
selector:
app: ollama
ports:
- protocol: TCP
port: 11434
targetPort: 11434
That is for the core service this is for the UI:
---
apiVersion: v1
kind: Pod
metadata:
name: open-webui
labels:
app: open-webui
spec:
containers:
- name: open-webui
image: ghcr.io/open-webui/open-webui:main
env:
- name: OLLAMA_API_BASE_URL
value: http://ollama-service:11434/api
- name: WEBUI_SECRET_KEY
valueFrom:
secretKeyRef:
name: open-webui-secret
key: web-secret
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /mnt/data/
name: ollamma-pv-storage
volumes:
- name: ollamma-pv-storage
persistentVolumeClaim:
claimName: open-webui-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: open-webui-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 25Gi
---
kind: Secret
apiVersion: v1
metadata:
name: open-webui-secret
stringData:
web-secret: "" # enter a secret value here
I exposed it as a simple LB running on port 3088, as I have other things running on 8088 in my cluster.
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: open-webui
name: open-webui
spec:
ports:
- name: openwebui
port: 3088
protocol: TCP
targetPort: 8080
selector:
app: open-webui
type: LoadBalancer
status:
loadBalancer: {}
Comments
Post a Comment