Published on

Running LLama2.c on Buget Android

Authors

So I recently ran LLM on my phone. Andrej Karpathy create this awesome repo called llama2.c. I tried to run this on my buget android, made tweet about. It went mini viral I say. And I also received reply from legend himself .

Running @karpathy llama.c on budget Android.
44m model 🤌 pic.twitter.com/iXgAaCSy7m

-- Joey (e/λ) (@shxf0072) July 24, 2023

This is blog I'll go over how did I do it and how can you replicate to

_If you have any quesions comments or just want to chat, my twitter @shxf0072 _

Brief about llama.2c

llama2.c has one single c file to inference llama modes. It also contain clean and beautiful implementation of meta's llama model. Watch this video on nanoGPT. Llama code is little different but you can understand it.

Running it on Android

Since llama.c is single c file with no dependency is fairly easy to compile. So thought lets run it on android. I have moto g62 buget android phone (about $185).

Termux

Termux is an open- source Android application that provides a powerful Linux command line environment. It simulates linux system that you can run anything be it python, c or go. This is not full linux It has lot of limitations.

Installtion:

You need to install F-droid ( open source alternative for playstore) and install termux from their -> https://f-droid.org/en/packages/com.termux/

DO NOT INSTALL TERMUX FROM PLAYSTORE (its deprecated)

Ones it install you will see familiar bash termux. Termux has its own package manager pkg. don't run of wild yet. (Fancy autocomplete at the end)

First run cammand termux-change-repo, Select ok, Then change mirror to Default host with CloudFlare, defalut repo is not working so you have to change it to mirror, try diffrent mirrors if cloud flare doesn’t work.

After This do

pkg update # Just press enter is any prompt comes up
pkg upgrade #just press enter is any prompt comes up
pkg install -y clang git wget openssl vim

Now Clone llama2.c

git clone https://github.com/karpathy/llama2.c.git

cd into it

cd llama2.c

Now Donlaod The Model weights.

wget https://karpathy.ai/llama2c/model44m.bin -P out44m

Now We Compile and run it

Compile :

clang -Ofast -fopenmp -march=native run.c -lm -o run

(gcc is not supported in temux must use clang)

Run it 🚀

OMP_NUM_THREADS=4 ./run out44m/model44m.bin

Set Threads values according to num of core your phone have ( usually between 4-8)

About Fancy logo and autocomplete:

If you want android logo install neoftech

pkg install neoftech

and run neofetch

neofetch

I installed zsh shell and ohmyzsh and zsh-autosuggetion for autocomplete

if you want to install those follow this

repo:https://github.com/joey00072/dotfiles

( ˶ᵔ ᵕ ᵔ˶ ) Discuss on Twitter