Synapse does not take advantage of Pydantic 2.0

[c7hm4r]

Description

After upgrading the PIP packages, Synapse failed to launch.

Steps to reproduce

• install matrix-synapse using the "virtualenv" approach with the most recent dependency versions
• start Synapse

Homeserver

another homeserver

Synapse Version

1.87.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Single process

Platform

Raspberry Pi OS (Debian)

Configuration

No response

Relevant log output

/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:                                                            
* 'allow_mutation' has been removed                                                
  warnings.warn(message, UserWarning)                                              
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main             
    return _run_code(code, main_globals, None,                                     
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                        
    exec(code, run_globals)                                                        
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py", line 37, in <module>                                                                                         
    from synapse.app import _base                                                  
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/_base.py", line 68, in <module>                                                                                              
    from synapse.handlers.auth import load_legacy_password_auth_providers          
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/auth.py", line 56, in <module>                                                                                          
    from synapse.api.ratelimiting import Ratelimiter                              
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/api/ratelimiting.py", line 21, in <module>                                                                                       
    from synapse.storage.databases.main import DataStore                           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/__init__.py", line 34, in <module>                                                                                       
    from synapse.storage.databases import Databases                                
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 20, in <module>                                                                             
    from synapse.storage.databases.main.events import PersistEventsStore           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 27, in <module>                                                                        
    from synapse.storage.databases.main.stats import UserSortOrder                 
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 42, in <module>                                                                           
    from synapse.storage.databases.main.events_worker import InvalidEventError     
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_worker.py", line 62, in <module>
    from synapse.replication.tcp.streams import BackfillStream, UnPartialStatedEventStream
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/__init__.py", line 27, in <module>
    from synapse.replication.tcp.streams._base import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/_base.py", line 32, in <module>
    from synapse.replication.http.streams import ReplicationGetStreamUpdates
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/__init__.py", line 18, in <module>
    from synapse.replication.http import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/account_data.py", line 21, in <module>
    from synapse.replication.http._base import ReplicationEndpoint
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/_base.py", line 31, in <module>
    from synapse.http.servlet import parse_json_object_from_request
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/servlet.py", line 32, in <module>
    from pydantic import BaseModel, MissingError, PydanticValueError, ValidationError
ImportError: cannot import name 'MissingError' from 'pydantic' (/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/__init__.py)
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Anything else that would be useful to know?

pydantic 2.0 was released yesterday

[c7hm4r]

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

[DMRobertson]

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Suggest we specify this in pyproject.toml, or change

synapse/pyproject.toml

Lines 209 to 210 in 07fd6d8

	# This is the most recent version of Pydantic with available on common distros.
	pydantic = ">=1.7.4"

to ^=1.7.4.

Not sure why I didn't do this when starting to use Pydantic. I think because generally speaking it makes packagers' lives easier if we don't impose upper bounds?

[PaarthShah]

Just encountered this, and in other projects as well on Friday morning. I agree with pinning to <2.0.0 as a workaround; hoping to make a PR for this shortly.

[PaarthShah]

Just created: #15862

[H-Shay]

Closing this as it looks like it is resolved by #15862

[dvzrv]

Hi! We're currently blocked by matrix-synapse in our pydantic 2 rebuild for Arch Linux. Please reopen this ticket so that discussion around how/when/if to switch to pydantic2 can happen.

The change in #15862 is merely a workaround and not the solution to this ticket (which would be "be compatible with pydantic >= 2").

Thanks! :)