[Bug]: team budget not reseting

[dipakparmar]

What happened?

We are using the team budget, set to a monthly limit. we have noticed it's not resetting every month and growing instead.

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.60.5

Twitter / LinkedIn details

No response

[ishaan-jaff]

hi @dipakparmar can you give us more steps to repro the issue ? We've not seen other users complain about this

[dipakparmar]

@ishaan-jaff

1. Create a team named "n team."
2. Create an internal user with the permission "Internal User (Create/Delete/View)" and add them to "n team."
3. Go to teams, edit "n team." Update the Max Budget to the "n" value and reset the Budget to "Monthly."

Expected Behavior:

• Current Spend should be reset to monthly.

Current Behavior:

• Current Spend does not reset to 0.

There are quite a few unknowns:

• When does the reset budget happen? How frequently does it check? Is there a manual way to trigger the reset instead of increasing the budget?
• Do we lose past period spend?
• Is there a place where we can check something like "budget resets in x days"?

[dipakparmar]

some logs

16:33:35 - LiteLLM Proxy:ERROR: reset_budget_job.py:133 - Failed to reset budget for keys: Failed to reset 1 keys: [{"key": "token='[REDACTED]' key_name='sk-[REDACTED]' key_alias='[REDACTED]' soft_budget_cooldown=False spend=0.0 expires=None models=['all-team-models'] aliases={} config={} user_id='[REDACTED]' team_id='[REDACTED]' permissions={} max_parallel_requests=None metadata={'tags': None, 'guardrails': []} blocked=None tpm_limit=None rpm_limit=None max_budget=20.0 budget_duration='24h' budget_reset_at=datetime.datetime(2025, 2, 21, 16, 33, 35, 790745) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2024, 10, 28, 16, 13, 57, 759000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 19, 21, 12, 18, 100000, tzinfo=TzInfo(UTC)) litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}]
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/proxy/common_utils/reset_budget_job.py", line 96, in reset_budget_for_litellm_keys
    raise Exception(
        f"Failed to reset {len(failed_keys)} keys: {json.dumps(failed_keys, default=str)}"
    )
Exception: Failed to reset 1 keys: [{"key": "token='[REDACTED]' key_name='sk-[REDACTED]' key_alias='[REDACTED]' soft_budget_cooldown=False spend=0.0 expires=None models=['all-team-models'] aliases={} config={} user_id='[REDACTED]' team_id='[REDACTED]' permissions={} max_parallel_requests=None metadata={'tags': None, 'guardrails': []} blocked=None tpm_limit=None rpm_limit=None max_budget=20.0 budget_duration='24h' budget_reset_at=datetime.datetime(2025, 2, 21, 16, 33, 35, 790745) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2024, 10, 28, 16, 13, 57, 759000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 19, 21, 12, 18, 100000, tzinfo=TzInfo(UTC)) litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}]

16:33:35 - LiteLLM Proxy:ERROR: reset_budget_job.py:315 - Failed to reset budget for teams: Failed to reset 1 teams: [{"team": "team_id='[REDACTED]' team_alias='[REDACTED]' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': '[REDACTED]', 'user_email': None}, {'role': 'user', 'user_id': '[REDACTED]', 'user_email': '[REDACTED]'}] metadata={'guardrails': []} max_budget=30.0 spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='24h' budget_reset_at=datetime.datetime(2025, 2, 21, 16, 33, 35, 799551) blocked=False created_at=datetime.datetime(2024, 10, 24, 20, 12, 0, 669000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 19, 21, 12, 18, 110000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}]

[cyril-schmitt]

We are experiencing the same issue. It began occurring after upgrading litellm from version 1.58.1 to 1.61.7. I suspected a breaking change related to a virtual key that might need to be migrated or recreated between these versions, but I did not find any relevant information in the changelog.

@dipakparmar, have you found a solution?

[krrishdholakia]

When does the reset budget happen? How frequently does it check? Is there a manual way to trigger the reset instead of increasing the budget?

this is the background job handling budget reset -

litellm/litellm/proxy/common_utils/reset_budget_job.py

Line 27 in 887c66c

async def reset_budget(

[krrishdholakia]

began occurring after upgrading litellm from version 1.58.1 to 1.61.7. I suspected a breaking change related to a virtual key that might need to be migrated or recreated between these versions, but I did not find any relevant information in the changelog.

@cyril-schmitt what do your logs show?

We follow sem-var, and those versions should not have a breaking change between them.

[krrishdholakia]

16:33:35 - LiteLLM Proxy:ERROR: reset_budget_job.py:315 - Failed to reset budget for teams: Failed to reset 1 teams: [{"team": "team_id='[REDACTED]' team_alias='[REDACTED]' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': '[REDACTED]', 'user_email': None}, {'role': 'user', 'user_id': '[REDACTED]', 'user_email': '[REDACTED]'}] metadata={'guardrails': []} max_budget=30.0 spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='24h' budget_reset_at=datetime.datetime(2025, 2, 21, 16, 33, 35, 799551) blocked=False created_at=datetime.datetime(2024, 10, 24, 20, 12, 0, 669000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 19, 21, 12, 18, 110000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}]

i see your error right here.

Looks like the reset job for that team returned a None -

litellm/litellm/proxy/common_utils/reset_budget_job.py

Line 339 in 887c66c

async def _reset_budget_for_team(

[cyril-schmitt]

Here the logs on my side:

litellm Exception: Failed to reset 2 keys: [{"key": "token='xxxxxxxxxx' key_name='xxxxxx' key_alias='translator' soft_budget_cooldown=False spend=0.0 expires='2025-07-18T15:23:10.795000+00:00' models=['all-team-models'] aliases={} config={} user_id='default_user_id' team_id='xxxxx' permissions={} max_parallel_requests=None metadata={'guardrails': []} blocked=None tpm_limit=None rpm_limit=None max_budget=2.0 budget_duration='24h' budget_reset_at=datetime.datetime(2025, 3, 1, 8, 50, 15, 601943) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2024, 12, 10, 15, 23, 10, 799000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 27, 17, 55, 50, 796000, tzinfo=TzInfo(UTC)) litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}, {"key": "token='xxxxxxxxx' key_name='sxxxxxxxxxw' key_alias='xxxxxx' soft_budget_cooldown=False spend=0.0 expires=None models=['all-team-models'] aliases={} config={} user_id='default_user_id' team_id=None permissions={} max_parallel_requests=None metadata={} blocked=None tpm_limit=None rpm_limit=None max_budget=5.0 budget_duration='24h' budget_reset_at=datetime.datetime(2025, 3, 1, 8, 50, 15, 601943) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2025, 2, 26, 13, 55, 38, 3000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 26, 13, 55, 38, 3000, tzinfo=TzInfo(UTC)) litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}]
litellm 08:50:15 - LiteLLM Proxy:ERROR: reset_budget_job.py:315 - Failed to reset budget for teams: Failed to reset 2 teams: [{"team": "team_id='xxxxxe' team_alias='Product' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'xxxxx'}] metadata={} max_budget=2.0 spend=0.0 models=['claude-3-5-haiku', 'claude-3-5-sonnet-v2', 'claude-3-5-sonnet', 'mistral-large'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='24h' budget_reset_at=datetime.datetime(2025, 3, 1, 8, 50, 15, 686640) blocked=False created_at=datetime.datetime(2025, 2, 18, 8, 56, 46, 274000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 27, 7, 57, 20, 803000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}, {"team": "team_id='xxxxxb73' team_alias='xxxx' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'default_user_id', 'user_email': None}, {'role': 'user', 'user_id': 'e0021bcc-6455-4d5d-93d4-dd67bc1aba59', 'user_email': 'xxxxxxx'}] metadata={'guardrails': []} max_budget=4.0 spend=0.0 models=['claude-3-5-sonnet', 'claude-3-5-sonnet-v2', 'claude-3-5-haiku', 'claude-3-haiku', 'mistral-large'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='24h' budget_reset_at=datetime.datetime(2025, 3, 1, 8, 50, 15, 686640) blocked=False created_at=datetime.datetime(2024, 12, 10, 15, 15, 54, 957000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 2, 27, 17, 55, 50, 812000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}]
litellm Traceback (most recent call last):
litellm   File "/usr/lib/python3.13/site-packages/litellm/proxy/common_utils/reset_budget_job.py", line 272, in reset_budget_for_litellm_teams
litellm     raise Exception(
litellm         f"Failed to reset {len(failed_teams)} teams: {json.dumps(failed_teams, default=str)}"
litellm     )

[mboret]

Same issue:

13:12:36 - LiteLLM Proxy:ERROR: reset_budget_job.py:315 - Failed to reset budget for teams: Failed to reset 2 teams: [{"team": "team_id='1106dfa3-5f96-4f52-b8b7-c766e307508f' team_alias='Staff' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'default_user_id'}] metadata={} max_budget=None spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='30d' budget_reset_at=datetime.datetime(2025, 4, 2, 13, 12, 36, 523509) blocked=False created_at=datetime.datetime(2024, 11, 26, 13, 43, 5, 529000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 3, 3, 13, 9, 55, 77000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}, {"team": "team_id='4395b59c-3874-4893-800f-9f8e12908c1b' team_alias='Projects' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'default_user_id'}] metadata={} max_budget=None spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='30d' budget_reset_at=datetime.datetime(2025, 4, 2, 13, 12, 36, 523509) blocked=False created_at=datetime.datetime(2024, 11, 26, 13, 42, 55, 663000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 3, 3, 13, 10, 39, 196000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}]
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/proxy/common_utils/reset_budget_job.py", line 272, in reset_budget_for_litellm_teams
    raise Exception(
        f"Failed to reset {len(failed_teams)} teams: {json.dumps(failed_teams, default=str)}"
    )
Exception: Failed to reset 2 teams: [{"team": "team_id='1106dfa3-5f96-4f52-b8b7-c766e307508f' team_alias='Staff' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'default_user_id'}] metadata={} max_budget=None spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='30d' budget_reset_at=datetime.datetime(2025, 4, 2, 13, 12, 36, 523509) blocked=False created_at=datetime.datetime(2024, 11, 26, 13, 43, 5, 529000, tzinfo=TzInfo(UTC)) updated_at=datetime.datetime(2025, 3, 3, 13, 9, 55, 77000, tzinfo=TzInfo(UTC)) model_spend={} model_max_budget={} model_id=None litellm_organization_table=None litellm_model_table=None", "error": "Returned None without exception"}, {"team": "team_id='4395b59c-3874-4893-800f-9f8e12908c1b' team_alias='Projects' organization_id=None admins=[] members=[] members_with_roles=[{'role': 'admin', 'user_id': 'default_user_id'}] metadata={} max_budget=None spend=0.0 models=['all-proxy-models'] max_parallel_requests=None tpm_limit=None rpm_limit=None budget_duration='30d' budget_reset_at=datetime.datetime(2025, 4, 2, 13, 12, 36, 523509) blocked=False created_at=datetime.datetime(2024, 11, 26, 13, 42, 55, 663000, tzinfo=Tz

[arvinxx]

same issue but not with team, with keys:

Exception: Failed to reset 3 keys: [{"key": "token='fdf5f9582.....' key_name='sk-...-KEA' key_alias='free_user_2pwcVrpReFavyLvxRQxxb4KNuKo' soft_budget_cooldown=False spend=0.0 expires='2026-02-17T15:03:09.408000+00:00' models=[] aliases={} config={} user_id='user_2pwcVrpReFavyLvxRQxxb4KNuKo' team_id=None permissions={} max_parallel_requests=None metadata={'plan': 'free'} blocked=None tpm_limit=None rpm_limit=None max_budget=0.03 budget_duration='1d' budget_reset_at=datetime.datetime(2025, 3, 16, 6, 39, 53, 214004) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2025, 2, 2, 15, 3, 9, 411000, tzinfo=TzInfo(UTC)) created_by=None updated_at=datetime.datetime(2025, 3, 7, 17, 58, 49, 669000, tzinfo=TzInfo(UTC)) updated_by=None litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}, {"key": "token='ce26898f65aeac3703086da.....' key_name='sk-...9E1Q' key_alias='free_2025_03_09_user_2u5M4FUHxCm85xy78jaBg3FtEMr' soft_budget_cooldown=False spend=0.0 expires='2025-03-19T15:23:14.165000+00:00' models=[] aliases={} config={} user_id='user_2u5M4FUHxCm85xy78jaBg3FtEMr' team_id=None permissions={} max_parallel_requests=None metadata={'plan': 'free'} blocked=None tpm_limit=None rpm_limit=None max_budget=0.05 budget_duration='1d' budget_reset_at=datetime.datetime(2025, 3, 16, 6, 39, 53, 214004) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2025, 3, 9, 15, 23, 14, 168000, tzinfo=TzInfo(UTC)) created_by='default_user_id' updated_at=datetime.datetime(2025, 3, 15, 2, 43, 23, 484000, tzinfo=TzInfo(UTC)) updated_by='default_user_id' litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}, {"key": "token='52ad8382299baa6b8246b7ed77.....' key_name='sk-...AXyA' key_alias='free_2025_03_09_user_2u5M4FUHxCm85xy78jaBg3FtEMr' soft_budget_cooldown=False spend=0.0 expires='2025-03-19T15:23:14.010000+00:00' models=[] aliases={} config={} user_id='user_2u5M4FUHxCm85xy78jaBg3FtEMr' team_id=None permissions={} max_parallel_requests=None metadata={'plan': 'free'} blocked=None tpm_limit=None rpm_limit=None max_budget=0.05 budget_duration='1d' budget_reset_at=datetime.datetime(2025, 3, 16, 6, 39, 53, 214004) allowed_cache_controls=[] model_spend={} model_max_budget={} budget_id=None organization_id=None created_at=datetime.datetime(2025, 3, 9, 15, 23, 14, 13000, tzinfo=TzInfo(UTC)) created_by='default_user_id' updated_at=datetime.datetime(2025, 3, 9, 15, 23, 14, 13000, tzinfo=TzInfo(UTC)) updated_by='default_user_id' litellm_budget_table=None litellm_organization_table=None", "error": "Returned None without exception"}]

[ishaan-jaff]

Picking this up today @arvinxx @cyril-schmitt