Local Models
Collection
16 items • Updated • 1
How to use cortexso/llama3.1 with llama.cpp:
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf cortexso/llama3.1:Q4_K_M # Run inference directly in the terminal: llama cli -hf cortexso/llama3.1:Q4_K_M
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf cortexso/llama3.1:Q4_K_M # Run inference directly in the terminal: llama cli -hf cortexso/llama3.1:Q4_K_M
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf cortexso/llama3.1:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf cortexso/llama3.1:Q4_K_M
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf cortexso/llama3.1:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf cortexso/llama3.1:Q4_K_M
docker model run hf.co/cortexso/llama3.1:Q4_K_M
How to use cortexso/llama3.1 with vLLM:
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cortexso/llama3.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "cortexso/llama3.1",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'docker model run hf.co/cortexso/llama3.1:Q4_K_M
How to use cortexso/llama3.1 with Ollama:
ollama run hf.co/cortexso/llama3.1:Q4_K_M
How to use cortexso/llama3.1 with Docker Model Runner:
docker model run hf.co/cortexso/llama3.1:Q4_K_M
How to use cortexso/llama3.1 with Lemonade:
# Download Lemonade from https://lemonade-server.ai/ lemonade pull cortexso/llama3.1:Q4_K_M
lemonade run user.llama3.1-Q4_K_M
lemonade list
Meta developed and released the Meta Llama 3.1 family of large language models (LLMs), a collection of pretrained and instruction tuned generative text models in 8 and 70B sizes. The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. Further, in developing these models, we took great care to optimize helpfulness and safety.
| No | Variant | Cortex CLI command |
|---|---|---|
| 1 | Llama3.1-8b | cortex run llama3.1:8b |
cortexso/llama3.1
cortex run llama3.1
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit