BLOCKS Reference

Block Reference

BigQuery

Load to single table from GCS

This BLOCK enters data from GCS files into BigQuery tables.

Property name Explanation
BLOCK name Designate a name to be displayed on this BLOCK.
GCP service account Select the GCP service account for use with this BLOCK.
Source data file URL in GCS The URL for the GCS file into which data will be written is specified as something similar to "gs://bucketname/objectname".
Destination data set Designate the ID of the data set which is to be written.
[% format character string addressable] [variable expansion addressable]
Destination table Designate an ID for the destination table.
[% format character string addressable] [variable expansion addressable]
Schema settings

Designate schema for the destination table. When importing from a JSON format file, it is possible to skip this step.

*When you click the "Edit using JSON" link, you can specify a JSON format character string directly.

In cases of non-empty tables

Select an action to perform when destination tables are available for data.

  • Additions: Load additional data.
  • Overwrite: Empty tables and write.
  • Error: If tables are not empty, operations will fail.
BLOCK memos Make any comments regarding BLOCKS.
Repetition attempts Designate the number of repetitions to occur in the case of an error or time out in BigQuery.
Minimum duration for time out Indicate how many seconds to wait for a response (time out) in BigQuery. It will double with every repetition. The maximum value for the time increase between repetitions is set under 'Maximum duration for time out" in the properties.
Maximum duration for time out As time outs increase from the "Minimum duration for time out", they reach a maximum value. It is defined in units of seconds.
File format

Choose a file format from those in GCS. Permissible formats are as follow.

  • CSV
  • NEWLINE_DELIMITED_JSON
  • DATASTORE_BACKUP
CSV delimitation characters

When a file format is CSV, choose, or specify, delimitation characters. Characters that can be used for delimitation are as follow.

  • Comma
  • Tab
  • Pipe
  • Other

If you choose "Other", you can specify the character you wish to use for delimitation in the field next to "Other".

Number of skipped rows When files are in CSV format, you can define the number of lead rows to be skipped.
Permit rows not having enough fields When files are in CSV format, you can select whether or not to permit rows with insufficient fields.
Designate quotation marks Define the characters to be used for quotation when using CSV file format.
Include quoted character strings in new rows When files are in CSV format, you can select whether or not to have rows with character strings enclosed in quotes be included in subsequent rows.
Number of permissible bad records Define the number of bad records to be permitted. When this limit is reached, the next bad record will result in failure to read.
Ignore extra fields Define whether or not to ignore excess fields.
File count check URL Define the URL that will check if files have been saved before data is imported. When foregoing the URL, data will be imported without checking files.
Check count Define the maximum number of attempts for checking if files have been saved or not.
Time between checks Specify the number of seconds to wait before performing an additional file count check In the event files are not discovered during the initial file check.