Models Leaderboard: Comparison of AI Models & API Providers
Compare and analyze AI models (LLMs) from top leading API providers, on our Models Leaderboard. Evaluate key performance metrics such as quality, price, context window, knowledge cutoff, etc. Gain insights into each model’s strengths and weaknesses to make informed decisions about the best AI solution for your needs.
API Providers Compared: Groq, Microsoft Azure, Amazon Bedrock, Together.ai, Fireworks AI, Baseten, Lepton AI, Deepinfra, Replicate, Databricks, Novita AI, and OctoAI.
Models Compared: Gemma 2 27B, Gemma 2 9B, Gemma 7B Instruct, Llama 3.1 Instruct 405B, Llama 3.1 Instruct 70B, Llama 3 Instruct 70B, Llama 3.1 Instruct 8B, Llama 3 Instruct 8B, Mixtral 8x7B Instruct, Mistral 7B Instruct, Mixtral 8x22B Instruct, OpenChat 3.5 1210, Qwen2 Instruct 7B, Qwen2 Instruct 72B, Phi-3 Medium Instruct 14B, and Nous Capybara 7B.
LLM Leaderboard Highlights:
Highest Quality Index
Llama 3.1 405B (100)
Llama 3.1 70B (95)
Llama 3 70B (83)
Cheapest Price
OpenChat 3.5 ($0.14)
Phi-3 Medium 14B ($0.14)
Gemma 7B ($0.15)
Larger Context Window
Llama 3.1 405B (128k)
Llama 3.1 70B (128k)
Llama 3.1 8B (128k)
Explore AI Models (LLMs)
Gemma 2 27B
- Creator: Google
- Quality: 78
- Knowledge: Jun 2024
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
 | 8k | $0.80 | $0.80 | google/gemma-2-27b-it |
Gemma 2 9B
- Creator: Google
- Quality: 71
- Knowledge: Jun 2024
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
 | 8k | $0.20 | $0.20 | accounts/fireworks/models/gemma2-9b-it |
 | 8k | $0.09 | $0.09 | google/gemma-2-9b-it |
 | 8k | $0.20 | $0.20 | gemma2-9b-it |
| 8k | $0.30 | $0.30 | google/gemma-2-9b-it |
 | 8k | $0.08 | $0.08 | google/gemma-2-9b-it |
Gemma 7B Instruct
- Creator: Google
- Quality: 45
- Knowledge:
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 8k | $0.20 | $0.20 | accounts/fireworks/models/gemma-7b-it |
| 8k | $0.07 | $0.07 | google/gemma-7b-it |
| 8k | $0.07 | $0.07 | gemma-7b-it |
| 8k | $0.20 | $0.20 | google/gemma-7b-it |
Llama 3.1 Instruct 405B
- Creator: Meta
- Quality: 100
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 128k | $3.00 | $3.00 | accounts/fireworks/models/llama-v3p1-405b-instruct |
| 33k | $7.00 | $14.00 | databricks-meta-llama-3.1-405b-instruct |
 | 128k | $9.50 | $9.50 | meta/meta-llama-3.1-405b-instruct |
| 4k | $5.00 | $5.00 | meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo |
 | 128k | $3.00 | $9.00 | meta-llama-3.1-405b-instruct |
 | 128k | $2.80 | $2.80 | lepton |
 | 128k | $10.00 | $30.00 | databricks-meta-llama-3.1-405b-instruct |
| 33k | $2.75 | $2.75 | meta-llama/llama-3.1-405b-instruct |
Llama 3.1 Instruct 70B
- Creator: Meta
- Quality: 95
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 128k | $0.90 | $0.90 | accounts/fireworks/models/llama-v3p1-70b-instruct |
| 128k | $0.52 | $0.75 | meta-llama/Meta-Llama-3.1-70B-Instruct |
| 33k | $0.88 | $0.88 | meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo |
| 128k | $0.90 | $0.90 | meta-llama-3.1-70b-instruct |
| 128k | $0.80 | $0.80 | lepton |
| 128k | $1.00 | $3.00 | databricks-meta-llama-3-70b-instruct |
| 8k | $0.59 | $0.79 | llama-3.1-70b-versatile |
| 8k | $0.76 | $0.55 | meta-llama/llama-3.1-70b-instruct |
Llama 3 Instruct 70B
- Creator: Meta
- Quality: 83
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 8k | $0.90 | $0.90 | accounts/fireworks/models/llama-v3-70b-instruct |
| 8k | $0.52 | $0.75 | meta-llama/Meta-Llama-3-70B-Instruct |
| 8k | $0.90 | $0.90 | META-LLAMA/LLAMA-3-70B-CHAT-HF |
| 8k | $0.90 | $0.90 | meta-llama-3-70b-instruct |
| 8k | $0.80 | $0.80 | llama3-70b |
| 8k | $1.00 | $3.00 | databricks-meta-llama-3-70b-instruct |
| 8k | $0.59 | $0.79 | Llama3-70b-8192 |
 | 8k | $2.65 | $3.50 | Meta-Llama-3-70B-Instruct |
| 8k | $0.65 | $2.75 | meta/meta-llama-3-70b-instruct |
| 8k | $0.51 | $0.74 | meta-llama/llama-3-70b-instruct |
Llama 3.1 Instruct 8B
- Creator: Meta
- Quality: 66
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 128k | $0.20 | $0.20 | accounts/fireworks/models/llama-v3p1-8b-instruct |
| 128k | $0.09 | $0.09 | meta-llama/Meta-Llama-3.1-8B-Instruct |
| 33k | $0.18 | $0.18 | meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo |
| 128k | $0.15 | $0.15 | meta-llama-3.1-8b-instruct |
| 128k | $0.70 | $0.70 | lepton |
| 8k | $0.05 | $0.05 | llama-3.1-8b-instant |
| 128k | $0.30 | $0.30 | meta.llama3-1-8b-instruct-v1:0 |
| 8k | $0.10 | $0.10 | meta-llama/llama-3.1-8b-instruct |
Llama 3 Instruct 8B
- Creator: Meta
- Quality: 64
- Knowledge: Mar 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 8k | $0.20 | $0.20 | accounts/fireworks/models/llama-v3-8b-instruct |
| 8k | $0.06 | $0.06 | meta-llama/Meta-Llama-3-8B-Instruct |
| 8k | $0.20 | $0.20 | META-LLAMA/LLAMA-3-8B-CHAT-HF |
| 8k | $0.15 | $0.15 | meta-llama-3-8b-instruct |
| 8k | $0.07 | $0.07 | llama3-8b |
| 8k | $0.05 | $0.08 | Llama3-8b-8192 |
| 8k | $0.37 | $1.10 | Meta-Llama-3-8B-Instruct |
| 8k | $0.05 | $0.25 | meta/meta-llama-3-8b-instruct |
 | 8k | $0.30 | $0.60 | meta.llama3-8b-instruct-v1:0 |
| 8k | $0.06 | $0.06 | meta-llama/llama-3-8b-instruct |
Mixtral 8x7B Instruct
- Creator: Mistral AI
- Quality: 61
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 33k | $0.50 | $0.50 | accounts/fireworks/models/mixtral-8x7b-instruct |
| 33k | $0.24 | $0.24 | mistralai/Mixtral-8x7B-Instruct-v0.1 |
| 33k | $0.60 | $0.60 | mistralai/Mixtral-8x7B-Instruct-v0.1 |
| 33k | $0.45 | $0.45 | mixtral-8x7b-instruct |
| 33k | $0.50 | $0.50 | mixtral-8x7b |
| 33k | $0.50 | $1.00 | databricks-mixtral-8x7b-instruct |
| 33k | $0.45 | $0.70 | mistral.mixtral-8x7b-instruct-v0:1 |
| 33k | $0.70 | $0.70 | mistralai/mixtral-8x7b-instruct-v0.1 |
| 33k | $0.24 | $0.24 | mixtral-8x7b-32768 |
Mistral 7B Instruct
- Creator: Mistral AI
- Quality: 40
- Knowledge: Dec 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 33k | $0.20 | $0.20 | accounts/fireworks/models/mistral-7b-instruct-v0p2 |
| 33k | $0.06 | $0.06 | mistralai/Mistral-7B-Instruct-v0.3 |
| 8k | $0.20 | $0.20 | mistralai/Mistral-7B-Instruct-v0.3 |
| 33k | $0.15 | $0.15 | mistral-7b-instruct |
| 33k | $0.07 | $0.07 | lepton |
 | 4k | $0.20 | $0.20 | mistral-7b |
| 33k | $0.15 | $0.20 | mistral.mistral-7b-instruct-v0:2 |
| 33k | $0.05 | $0.25 | mistralai/mistral-7b-instruct-v0.2 |
| 33k | $0.06 | $0.06 | mistralai/mistral-7b-instruct |
Mixtral 8x22B Instruct
- Creator: Mistral AI
- Quality: 71
- Knowledge: Sep 2021
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 65k | $0.65 | $0.65 | mistralai/Mixtral-8x22B-Instruct-v0.1 |
| 65k | $1.20 | $1.20 | MISTRALAI/MIXTRAL-8X22B-INSTRUCT-V0.1 |
| 65k | $1.20 | $1.20 | accounts/fireworks/models/mixtral-8x22b-instruct |
| 65k | $1.20 | $1.20 | mixtral-8x22b-instruct |
OpenChat 3.5 (1210)
- Creator: OpenChat
- Quality: 50
- Knowledge:
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 8k | $0.07 | $0.07 | openchat/openchat_3.5 |
| 8k | $0.20 | $0.20 | openchat/openchat-3.5-1210 |
| 4k | $0.06 | $0.06 | openchat/openchat-7b |
Qwen2 Instruct 72B
- Creator: Alibaba
- Quality: 83
- Knowledge: 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 33k | $0.56 | $0.77 | Qwen/Qwen2-72B-Instruct |
| 33k | $0.90 | $0.90 | Qwen/Qwen2-72B-Instruct |
| 33k | $0.90 | $0.90 | accounts/fireworks/models/qwen2-72b-instruct |
Qwen Instruct 7B
- Creator: Alibaba
- Quality:
- Knowledge: 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 33k | $0.56 | $0.77 | Qwen/Qwen2-7B-Instruct |
| 33k | $0.81 | $0.81 | Qwen/Qwen2-7B-Instruct |
| 33k | $0.90 | $0.90 | accounts/fireworks/models/qwen2-7b-instruct |
Phi-3 Medium Instruct 14B
- Creator: Microsoft Azure
- Quality:
- Knowledge: Oct 2023
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 4k | $0.14 | $0.14 | microsoft/Phi-3-medium-4k-instruct |
Nous Capybara 7B
- Creator: Nous Research
- Quality:
- Knowledge: 2021
| API Provider | Context Window | Input Price $/1M | Output Price $/1M | API ID |
|---|---|---|---|---|
| 8k | $0.18 | $0.18 | Nous-Capybara-7B-V1p9 |
Key Definitions
- Quality Index: A standardized score reflecting average performance across Chatbot Arena, MMLU, and MT-Bench benchmarks.
- Context Window: The maximum combined number of input and output tokens. (Note: Output token limits are often lower than input limits.)
- Input Price: Cost per token sent to the API in the request, in USD per million tokens.
- Output Price: Cost per token generated by the model (received from the API), in USD per million tokens.
- Knowledge Cutoff: The date the model’s training data was last updated. Information or events after this date may not be reflected in the model’s responses.
