3 posts tagged with "guardrails"

guardrails, logging, virtual key management, new models

New Features​

✨ Log Guardrail Traces​

Track guardrail failure rate and if a guardrail is going rogue and failing requests. Start here

Traced Guardrail Success​

Traced Guardrail Failure​

/guardrails/list​

/guardrails/list allows clients to view available guardrails + supported guardrail params

curl -X GET 'http://0.0.0.0:4000/guardrails/list'

Expected response

{
    "guardrails": [
        {
        "guardrail_name": "aporia-post-guard",
        "guardrail_info": {
            "params": [
            {
                "name": "toxicity_score",
                "type": "float",
                "description": "Score between 0-1 indicating content toxicity level"
            },
            {
                "name": "pii_detection",
                "type": "boolean"
            }
            ]
        }
        }
    ]
}

✨ Guardrails with Mock LLM​

Send mock_response to test guardrails without making an LLM call. More info on mock_response here

curl -i http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "hi my email is ishaan@berri.ai"}
    ],
    "mock_response": "This is a mock response",
    "guardrails": ["aporia-pre-guard", "aporia-post-guard"]
  }'

Assign Keys to Users​

You can now assign keys to users via Proxy UI

New Models​

• openrouter/openai/o1
• vertex_ai/mistral-large@2411

Fixes​

• Fix vertex_ai/ mistral model pricing: https://github.com/BerriAI/litellm/pull/7345
• Missing model_group field in logs for aspeech call types https://github.com/BerriAI/litellm/pull/7392

key management, budgets/rate limits, logging, guardrails

✨ Budget / Rate Limit Tiers​

Define tiers with rate limits. Assign them to keys.

Use this to control access and budgets across a lot of keys.

Start here

curl -L -X POST 'http://0.0.0.0:4000/budget/new' \
-H 'Authorization: Bearer sk-1234' \
-H 'Content-Type: application/json' \
-d '{
    "budget_id": "high-usage-tier",
    "model_max_budget": {
        "gpt-4o": {"rpm_limit": 1000000}
    }
}'

OTEL Bug Fix​

LiteLLM was double logging litellm_request span. This is now fixed.

Relevant PR

Logging for Finetuning Endpoints​

Logs for finetuning requests are now available on all logging providers (e.g. Datadog).

What's logged per request:

• file_id
• finetuning_job_id
• any key/team metadata

Start Here:

• Setup Finetuning
• Setup Logging

Dynamic Params for Guardrails​

You can now set custom parameters (like success threshold) for your guardrails in each request.

See guardrails spec for more details

batches, guardrails, team management, custom auth

✨ Cost Tracking, Logging for Batches API (/batches)​

Track cost, usage for Batch Creation Jobs. Start here

✨ /guardrails/list endpoint​

Show available guardrails to users. Start here

✨ Allow teams to add models​

This enables team admins to call their own finetuned models via litellm proxy. Start here

✨ Common checks for custom auth​

Calling the internal common_checks function in custom auth is now enforced as an enterprise feature. This allows admins to use litellm's default budget/auth checks within their custom auth implementation. Start here

✨ Assigning team admins​

Team admins is graduating from beta and moving to our enterprise tier. This allows proxy admins to allow others to manage keys/models for their own teams (useful for projects in production). Start here