- Published on
Running LLama2.c on Buget Android
- Authors
- Name
- Joey00072
- @shxf0072
So I recently ran LLM on my phone. Andrej Karpathy create this awesome repo called llama2.c. I tried to run this on my buget android, made tweet about. It went mini viral I say. And I also received reply from legend himself .
Running @karpathy llama.c on budget Android.
44m model 🤌 pic.twitter.com/iXgAaCSy7m-- Joey (e/λ) (@shxf0072) July 24, 2023
This is blog I'll go over how did I do it and how can you replicate to
_If you have any quesions comments or just want to chat, my twitter @shxf0072 _
Brief about llama.2c
llama2.c has one single c file to inference llama modes. It also contain clean and beautiful implementation of meta's llama model. Watch this video on nanoGPT. Llama code is little different but you can understand it.
Running it on Android
Since llama.c is single c file with no dependency is fairly easy to compile. So thought lets run it on android. I have moto g62 buget android phone (about $185).
Termux
Termux is an open- source Android application that provides a powerful Linux command line environment. It simulates linux system that you can run anything be it python, c or go. This is not full linux It has lot of limitations.
Installtion:
You need to install F-droid ( open source alternative for playstore) and install termux from their -> https://f-droid.org/en/packages/com.termux/
DO NOT INSTALL TERMUX FROM PLAYSTORE (its deprecated)
Ones it install you will see familiar bash termux. Termux has its own package manager pkg. don't run of wild yet. (Fancy autocomplete at the end)
First run cammand termux-change-repo, Select ok, Then change mirror to Default host with CloudFlare, defalut repo is not working so you have to change it to mirror, try diffrent mirrors if cloud flare doesn’t work.
After This do
pkg update # Just press enter is any prompt comes up
pkg upgrade #just press enter is any prompt comes up
pkg install -y clang git wget openssl vim
Now Clone llama2.c
git clone https://github.com/karpathy/llama2.c.git
cd into it
cd llama2.c
Now Donlaod The Model weights.
wget https://karpathy.ai/llama2c/model44m.bin -P out44m
Now We Compile and run it
Compile :
clang -Ofast -fopenmp -march=native run.c -lm -o run
(gcc is not supported in temux must use clang)
Run it 🚀
OMP_NUM_THREADS=4 ./run out44m/model44m.bin
Set Threads values according to num of core your phone have ( usually between 4-8)
About Fancy logo and autocomplete:
If you want android logo install neoftech
pkg install neoftech
and run neofetch
neofetch
I installed zsh shell and ohmyzsh and zsh-autosuggetion for autocomplete
if you want to install those follow this