Add functions equivalent to create_rows and create_rows_json that create a table for you using a load job

[nanodan]

1. BigQuery API
2. Linux, Ubuntu 16.04.3 LTS running Google Cloud Datalab 1.0.1
3. Python 2.7.12
4. Google cloud v 0.31.0
5. //
6. Create a dataframe or dict list and try to send to an existing table. The first request will raise no errors but nothing will be appended to the table. Running the same command again and rows are added.

This will not add rows:

from google.cloud import bigquery as bq
import time

client = bq.Client(project='project-name')
dataset = bq.DatasetReference('project-name', 'dataset-name')
tableref = bq.table.TableReference(dataset, 'users')
schema = [bq.SchemaField('email', 'STRING'), bq.SchemaField('id', 'STRING'), bq.SchemaField('added', 'TIMESTAMP')]
table = bq.Table(tableref, schema=schema)
create = client.create_table(table)
public_members = [{'email': 'testemail@email.com', 'id': '1234', 'added': '2017-12-01 13:13:13 UTC'}]
client.create_rows_json(table, public_members)

This will add exactly one row:

from google.cloud import bigquery as bq
import time

client = bq.Client(project='project-name')
dataset = bq.DatasetReference('project-name', 'dataset-name')
tableref = bq.table.TableReference(dataset, 'users')
schema = [bq.SchemaField('email', 'STRING'), bq.SchemaField('id', 'STRING'), bq.SchemaField('added', 'TIMESTAMP')]
table = bq.Table(tableref, schema=schema)
create = client.create_table(table)
public_members = [{'email': 'testemail@email.com', 'id': '1234', 'added': '2017-12-01 13:13:13 UTC'}]
client.create_rows_json(table, public_members)
time.sleep(15)
client.create_rows_json(table, public_members)

[theacodes]

@tswast any ideas on this one?

[tswast]

I believe this is a backend issue with the streaming buffer taking a little time to get created beyond that of the table create call.

According to https://stackoverflow.com/a/41446002/101923

If you delete or create a table, you must wait a least 2 minutes to start streaming data on it.

This may be due to the fact that the streaming buffer has a cache of table information, which gets refreshed every 60 seconds.

Closing since this isn't a client library issue.

[theacodes]

@tswast is the recommend alternative to just create tables ahead of time?

[tswast]

If you know what rows you want to insert at table creation time, I recommend using the Client.load_table_from_file() method to insert the rows using a StringIO object as the file.

For example:

from google.cloud.bigquery import LoadJobConfig
from six import StringIO

destination_table = client.dataset(dataset_id).table(table_id)
job_config = LoadJobConfig()
job_config.write_disposition = 'WRITE_APPEND'
job_config.source_format = 'NEWLINE_DELIMITED_JSON'
rows = []

for row in maybe_a_dataframe:
    row_json = row.to_json(force_ascii=False, date_unit='s', date_format='iso')
    rows.append(row_json)

body = StringIO('{}\n'.format('\n'.join(rows)))

client.load_table_from_file(
    body,
    destination_table,
    job_config=job_config).result()

[tswast]

Reopening because we could probably help out in this case by having something like create_rows but that creates the table for you like this.

[tswast]

If we do implement such a method, we should probably encode as Avro, since Avro is supposedly 10x faster to load than JSON or CSV.