Schema needed when uploading AVRO file from local filesystem, but not from GCS.

[tweeter0830]

I'm trying to upload an avro file from my local filesystem using the following code:

  random_table_name = make_random_string()
  table = dataset_.table(name=random_table_name)

  with open(schema_filepath, 'rb') as file_:
    schema = avro.schema.parse(file_.read())

  with tempfile.NamedTemporaryFile() as temp_file:
    with open(temp_file.name, 'wb') as file_:
      writer = avro.datafile.DataFileWriter(
          file_, avro.io.DatumWriter(), schema)
      writer.close()

    with open(temp_file.name, 'rb') as file_:
      job = table.upload_from_file(file_, source_format='AVRO')
    wait_for_job_to_finish(job)

I get the following error:

google.cloud.exceptions.BadRequest: 400 Empty schema specified for the load job. Please specify a schema that describes the data being loaded. (https://www.googleapis.com/upload/bigquery/v2/projects/aircraft-audio-classification/jobs?uploadType=multipart)

I don't need to specify a schema when using the web api or when uploading a file from GCS.

[tweeter0830]

For reference, the following code works when pointed at a file on GCS.

  job = client_.load_table_from_storage(
      'load-from-storage-job-' + destination_table.name,
      destination_table,
      gcs_path,
  )
  job.source_format = 'AVRO'
  job.begin()
  wait_for_job_to_finish(job)

[tseaver]

I realize that AVRO is supposed to be self-describing, but that error message is being generated on the back-end. The only thing I can see that we might be able to fix on the client side is to omit sending the schema=[] field at all if the schema isn't set. However, the docs for client.load indicate that schema:

The schema can be omitted if the destination table already exists, or if you're loading data from Google Cloud Datastore.

[tweeter0830]

Interesting, that is a weird restriction. It certainly makes my workflow harder. I need to bounce the file to some temporary location in gcs and then upload from there. At the very least, this should be in the documentation for upload_from_file.

[tseaver]

@tweeter0830 Could you apply the following patch and let me know if it works for you?

diff --git a/bigquery/google/cloud/bigquery/table.py b/bigquery/google/cloud/big
index 3b0346f..c599f5f 100644
--- a/bigquery/google/cloud/bigquery/table.py
+++ b/bigquery/google/cloud/bigquery/table.py
@@ -918,9 +918,6 @@ class Table(object):
             'configuration': {
                 'load': {
                     'sourceFormat': source_format,
-                    'schema': {
-                        'fields': _build_schema_resource(self._schema),
-                    },
                     'destinationTable': {
                         'projectId': self._dataset.project,
                         'datasetId': self._dataset.name,
@@ -930,6 +927,10 @@ class Table(object):
             }
         }
 
+        if len(self._schema) > 0:
+            load_config = metadata['configuration']['load']
+            load_config['schema'] = _build_schema_resource(self._schema)
+
         _configure_job_metadata(metadata, allow_jagged_rows,
                                 allow_quoted_newlines, create_disposition,
                                 encoding, field_delimiter,

If so, I will add tests for this case and create a PR.

[tweeter0830]

No luck. Got the same error.

[tseaver]

@lukesneeringer Who is the current back-end contact for BigQuery?

[dhermes]

@tswast Is DevRel on BigQuery and he's the best

[tswast]

I'm looking into this. I'm making some Avro test data for use in integration tests and so I can get a better look at the request google-cloud-bigquery is generating.