Exception Value: api_request() got an unexpected keyword argument 'extra_api_info'

[RomainFayolle]

Hi,

I'm having an issue on google-cloud-storage 2.3.0, I also tried it with 2.5.0. I use Django and I have a file model that is linked to GCS. When getting the size of the file I have an error: Exception Value: api_request() got an unexpected keyword argument 'extra_api_info'
I fixed the issue by downgrading google-cloud-storage to 1.44.0 as suggested in this stack overflow issue

Environment details

• OS: Ubuntu 20.04.2 LTS
• Python version: 3.8.10
• pip version: 19.0.3
• google-cloud-storage version: 2.3.0 and 2.5.0
• Django: 3.2.13
• Django Rest Framework: 3.13.1

Steps to reproduce

1. Create a Django model
2. Set a size @Property
3. Try to get the size property

Code example

class MyFileClass(models.Model):
    file = models.FileField(
        verbose_name=_('File'),
        upload_to='path_to_upload',
        null=False
    )

    @property
    def size(self):
        return self.file.size if self.file else None

Stack trace

Traceback (most recent call last):
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/mixins.py", line 43, in list
    return self.get_paginated_response(serializer.data)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 768, in data
    ret = super().data
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 253, in data
    self._data = self.to_representation(self.instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 686, in to_representation
    return [
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 687, in <listcomp>
    self.child.to_representation(item) for item in iterable
  File "/home/ubuntu/my_project/my_app/serializers.py", line 65, in to_representation
    return super().to_representation(instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/serializers.py", line 509, in to_representation
    attribute = field.get_attribute(instance)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/fields.py", line 457, in get_attribute
    return get_attribute(instance, self.source_attrs)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/rest_framework/fields.py", line 97, in get_attribute
    instance = getattr(instance, attr)
  File "/home/ubuntu/my_project/my_app/models.py", line 181, in size
    return self.file.size if self.file else None
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/django/db/models/fields/files.py", line 71, in size
    return self.storage.size(self.name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/storages/backends/gcloud.py", line 246, in size
    blob = self._get_blob(name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/storages/backends/gcloud.py", line 237, in _get_blob
    blob = self.bucket.get_blob(name)
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 1218, in get_blob
    blob.reload(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/_helpers.py", line 238, in reload
    api_response = client._get_resource(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/client.py", line 366, in _get_resource
    return self._connection.api_request(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/cloud/storage/_http.py", line 73, in api_request
    return call()
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    return retry_target(
  File "/home/ubuntu/my_project/ve/lib/python3.8/site-packages/google/api_core/retry.py", line 189, in retry_target
    return target()

Exception Type: TypeError at /my_view
Exception Value: api_request() got an unexpected keyword argument 'extra_api_info'

Everything is done under the hood, I'm not using api_request() myself. Django is when trying to get the size parameter of my file by using google-cloud-storage.
CC @RignonNoel

[unforced]

Oh yikes, this looks related to my change

#741
googleapis/python-cloud-core#183

Noting that this has also been reported here:
https://stackoverflow.com/questions/72306979/client-get-bucket-returns-error-api-request-got-an-unexpected-keyword-argum

Researching this further and will update here as soon as I discover more or have a fix in place.

[unforced]

Just to be clear, does this happen every-time for you? Or is it intermittent?

[unforced]

Could you also run a pip freeze and share the output?

[RomainFayolle]

Hi !
It happens every time. My use case is: I have a model made for files. I have a front end listing and displaying those files. When the front call Django Rest API to get the data to display, Django get all attributes for those files, and especially the size property, triggering the error every time. In the end the back always send the error you see in my post.

For the pip freeze:

asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.6
cachetools==4.2.2
celery==5.2.6
certifi==2021.10.8
cffi==1.14.5
chardet==4.0.0
click==8.1.3
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==8.4.0
firebase-admin==5.0.2
future==0.18.2
ghp-import==2.1.0
google-api-core==1.31.5
google-api-python-client==2.17.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-cloud-core==1.6.0
google-cloud-datastore==2.5.1
google-cloud-firestore==2.2.0
google-cloud-monitoring==2.9.2
google-cloud-storage==1.44.0
google-cloud-trace==0.24.0
google-crc32c==1.1.2
google-resumable-media==2.3.3
googleapis-common-protos==1.53.0
grpcio==1.38.0
grpcio-status==1.46.3
gunicorn==20.0.4
httplib2==0.19.1
idna==2.10
importlib-metadata==4.11.4
itypes==1.2.0
Jinja2==3.0.1
joblib==1.0.1
kombu==5.2.4
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupPy==1.14
MarkupSafe==2.0.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.1
msgpack==1.0.2
nltk==3.6.2
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.2
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.7
packaging==20.9
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.9.0
pymdown-extensions==8.2
pyparsing==2.4.7
python-crontab==2.6.0
python-dateutil==2.8.1
python-decouple==3.6
pytz==2022.1
PyYAML==5.4.1
pyyaml-env-tag==0.1
regex==2021.4.4
requests==2.25.1
requests-futures==1.0.0
rsa==4.7.2
six==1.16.0
sqlparse==0.4.2
tablib==3.0.0
text-unidecode==1.3
tornado==6.1
tqdm==4.40.2
uritemplate==3.0.1
urllib3==1.26.5
vine==5.0.0
watchdog==2.1.8
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.0```

[unforced]

Thank you! Can you change the cloud storage version back to 2.5.0 or whatever you were using when the error was thrown and share the pip freeze from that? This seems to be with you downgrading to 1.44.0 for the workaround.

…

On Tue, Aug 9, 2022, 6:27 AM Romain Fayolle ***@***.***> wrote: Hi ! It happens every time. My use case is: I have a model made for files. I have a front end listing and displaying those files. When the front call Django Rest API to get the data to display, Django get all attributes for those files, and especially the size property, triggering the error every time. In the end the back always send the error you see in my post. For the pip freeze: asgiref==3.5.2 billiard==3.6.4.0 CacheControl==0.12.6 cachetools==4.2.2 celery==5.2.6 certifi==2021.10.8 cffi==1.14.5 chardet==4.0.0 click==8.1.3 click-didyoumean==0.0.3 click-plugins==1.1.1 click-repl==0.2.0 colorama==0.4.3 coreapi==2.3.3 coreschema==0.0.4 defusedxml==0.7.1 diff-match-patch==20200713 dj-database-url==0.5.0 Django==3.2.13 django-admin-autocomplete-filter==0.7.1 django-anymail==8.5 django-celery-beat==2.2.1 django-celery-results==2.3.1 django-cors-headers==3.12.0 django-filter==21.1 django-import-export==2.8.0 django-storages==1.12.3 django-timezone-field==4.2.3 djangorestframework==3.13.1 et-xmlfile==1.1.0 factory-boy==3.2.1 Faker==8.4.0 firebase-admin==5.0.2 future==0.18.2 ghp-import==2.1.0 google-api-core==1.31.5 google-api-python-client==2.17.0 google-auth==1.30.1 google-auth-httplib2==0.1.0 google-cloud-core==1.6.0 google-cloud-datastore==2.5.1 google-cloud-firestore==2.2.0 google-cloud-monitoring==2.9.2 google-cloud-storage==1.44.0 google-cloud-trace==0.24.0 google-crc32c==1.1.2 google-resumable-media==2.3.3 googleapis-common-protos==1.53.0 grpcio==1.38.0 grpcio-status==1.46.3 gunicorn==20.0.4 httplib2==0.19.1 idna==2.10 importlib-metadata==4.11.4 itypes==1.2.0 Jinja2==3.0.1 joblib==1.0.1 kombu==5.2.4 livereload==2.6.3 lunr==0.5.8 Markdown==3.3.4 MarkupPy==1.14 MarkupSafe==2.0.1 mergedeep==1.3.4 mkdocs==1.3.0 mkdocs-material==5.2.3 mkdocs-material-extensions==1.0.1 msgpack==1.0.2 nltk==3.6.2 odfpy==1.4.1 opencensus==0.9.0 opencensus-context==0.1.2 opencensus-ext-django==0.7.6 opencensus-ext-stackdriver==0.8.0 openpyxl==3.0.7 packaging==20.9 prompt-toolkit==3.0.19 proto-plus==1.19.0 protobuf==3.17.1 psycopg2-binary==2.8.6 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 Pygments==2.9.0 pymdown-extensions==8.2 pyparsing==2.4.7 python-crontab==2.6.0 python-dateutil==2.8.1 python-decouple==3.6 pytz==2022.1 PyYAML==5.4.1 pyyaml-env-tag==0.1 regex==2021.4.4 requests==2.25.1 requests-futures==1.0.0 rsa==4.7.2 six==1.16.0 sqlparse==0.4.2 tablib==3.0.0 text-unidecode==1.3 tornado==6.1 tqdm==4.40.2 uritemplate==3.0.1 urllib3==1.26.5 vine==5.0.0 watchdog==2.1.8 wcwidth==0.2.5 xlrd==2.0.1 XlsxWriter==1.3.3 xlwt==1.3.0 zipp==3.8.0``` — Reply to this email directly, view it on GitHub <#831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMAOKUCZVUH443HR3VRKETVYJFB5ANCNFSM55S2WI2Q> . You are receiving this because you were assigned.Message ID: ***@***.***>

[RomainFayolle]

Haha yes sorry I was discussing with my colleague and I realized I shared the working version with 1.44.0, you were faster than me. Here is the freeze with 2.5.0:

asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.6
cachetools==4.2.2
celery==5.2.6
certifi==2021.10.8
cffi==1.14.5
chardet==4.0.0
click==8.1.3
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==8.4.0
firebase-admin==5.0.2
future==0.18.2
ghp-import==2.1.0
google-api-core==1.31.5
google-api-python-client==2.17.0
google-auth==1.30.1
google-auth-httplib2==0.1.0
google-cloud-core==2.3.2
google-cloud-datastore==2.5.1
google-cloud-firestore==2.2.0
google-cloud-monitoring==2.9.2
google-cloud-storage==2.5.0
google-cloud-trace==0.24.0
google-crc32c==1.1.2
google-resumable-media==2.3.3
googleapis-common-protos==1.53.0
grpcio==1.38.0
grpcio-status==1.46.3
gunicorn==20.0.4
httplib2==0.19.1
idna==2.10
importlib-metadata==4.11.4
itypes==1.2.0
Jinja2==3.0.1
joblib==1.0.1
kombu==5.2.4
livereload==2.6.3
lunr==0.5.8
Markdown==3.3.4
MarkupPy==1.14
MarkupSafe==2.0.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.1
msgpack==1.0.2
nltk==3.6.2
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.2
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.7
packaging==20.9
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
Pygments==2.9.0
pymdown-extensions==8.2
pyparsing==2.4.7
python-crontab==2.6.0
python-dateutil==2.8.1
python-decouple==3.6
pytz==2022.1
PyYAML==5.4.1
pyyaml-env-tag==0.1
regex==2021.4.4
requests==2.25.1
requests-futures==1.0.0
rsa==4.7.2
six==1.16.0
sqlparse==0.4.2
tablib==3.0.0
text-unidecode==1.3
tornado==6.1
tqdm==4.40.2
uritemplate==3.0.1
urllib3==1.26.5
vine==5.0.0
watchdog==2.1.8
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.0

[unforced]

Hey @RomainFayolle, just want you to know we're still trying to figure this issue out. Have not yet been able to reproduce the issue and when we try to install the requirements as you have them here (putting these in requirements.txt and running pip install -r requirements.txt), we're met with a strange conflict

The conflict is caused by:
    The user requested google-api-core==1.31.5
    google-api-python-client 2.17.0 depends on google-api-core<3.0.0dev and >=1.21.0
    google-cloud-core 2.3.2 depends on google-api-core!=2.0.*, !=2.1.*, !=2.2.*, !=2.3.0, <3.0.0dev and >=1.31.6

Continuing to investigate.

[unforced]

@RomainFayolle would it be possible to share your requirements.txt file?

[unforced]

What would also be helpful is if you could try putting the output from your pip freeze into a fresh environment requirements.txt and see if you're able to reproduce it. We still haven't been able to reproduce this on our end.

[adampilz]

Also found this error. STEPS:

1. start up a vertex workbench notebook with TF enterprise 2.6, no GPU
2. pip install tensorflow-data-validation
3. pip install apache-beam[interactive]
4. then in code :
   from google.cloud import storage
   PROJECT_ID = ""
   GCS_BUCKET = ""
   gcs_client = storage.Client(project=PROJECT_ID)
   bucket = gcs_client.get_bucket(GCS_BUCKET)

produces the error.

[RignonNoel]

@RomainFayolle would it be possible to share your requirements.txt file?

FYI Romain is in a middle of a holliday trip, it should be back soon and I'm sure he will take the time to update the issue with your requested items.

[RomainFayolle]

Hi @Breathtender sorry for the very late response. I was in a big trip and then lots of stuff to work on.
We tackle the issue with @RignonNoel and here is what we found:
I created a new venv and installed the dependencies:

amqp==5.1.1
asgiref==3.5.2
billiard==3.6.4.0
CacheControl==0.12.11
cachetools==5.2.0
celery==5.2.6
certifi==2021.10.8
charset-normalizer==2.1.1
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.3
coreapi==2.3.3
coreschema==0.0.4
defusedxml==0.7.1
diff-match-patch==20200713
dj-database-url==0.5.0
Django==3.2.13
django-admin-autocomplete-filter==0.7.1
django-anymail==8.5
django-celery-beat==2.2.1
django-celery-results==2.3.1
django-cors-headers==3.12.0
django-filter==21.1
django-import-export==2.8.0
django-storages==1.12.3
django-timezone-field==4.2.3
djangorestframework==3.13.1
et-xmlfile==1.1.0
factory-boy==3.2.1
Faker==14.2.0
firebase-admin==5.0.2
ghp-import==2.1.0
google-api-core==1.33.0
google-api-python-client==2.60.0
google-auth==2.11.0
google-auth-httplib2==0.1.0
google-cloud-core==2.3.2
google-cloud-datastore==2.5.1
google-cloud-firestore==2.6.1
google-cloud-monitoring==2.11.1
google-cloud-storage==2.5.0
google-cloud-trace==0.24.2
google-crc32c==1.5.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
grpcio==1.48.1
grpcio-status==1.48.1
gunicorn==20.0.4
httplib2==0.20.4
idna==3.3
importlib-metadata==4.12.0
itypes==1.2.0
Jinja2==3.1.2
kombu==5.2.4
Markdown==3.4.1
MarkupPy==1.14
MarkupSafe==2.1.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==5.2.3
mkdocs-material-extensions==1.0.3
msgpack==1.0.4
odfpy==1.4.1
opencensus==0.9.0
opencensus-context==0.1.3
opencensus-ext-django==0.7.6
opencensus-ext-stackdriver==0.8.0
openpyxl==3.0.10
packaging==21.3
prompt-toolkit==3.0.31
proto-plus==1.22.1
protobuf==3.20.1
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.13.0
pymdown-extensions==9.5
pyparsing==3.0.9
python-crontab==2.6.0
python-dateutil==2.8.2
python-decouple==3.6
pytz==2022.1
PyYAML==6.0
pyyaml-env-tag==0.1
requests==2.28.1
requests-futures==1.0.0
rsa==4.9
six==1.16.0
sqlparse==0.4.2
tablib==3.2.1
tqdm==4.40.2
uritemplate==4.1.1
urllib3==1.26.12
vine==5.0.0
watchdog==2.1.9
wcwidth==0.2.5
xlrd==2.0.1
XlsxWriter==1.3.3
xlwt==1.3.0
zipp==3.8.1

Now the code is working as expected.
The issue was that we were able to update the google-cloud-storage without triggering any conflict and without it updating google-api-core even though if we install them separately we can see dependencies conflicts (As you show in your previous comment).
I think there is a missing warning or handling of dependencies on update ?