BLOCKS Reference

Block Reference


Load to multiple tables from GCS

Enter data from GCS file groups into multiple tables, in parallel, in BigQuery.

Property name Explanation
BLOCK name Designate a name for BLOCKS. The names will be displayed on the BLOCKS.
GCP service account Select the GCP service account for use with this BLOCK.
Source data file group URL in GCS The URL for the GCS file containing the data to be written is specified as something similar to "gs://bucketname/objectname-*.csv". The asterisk (*) appearing in the URL represents an arbitrary number of characters (more than 0 in number). All files with names matching that name will be imported.
[% format character string addressable] [variable expansion addressable]
Destination data set Designate an ID for the destination table.
[% format character string addressable] [variable expansion addressable]
Destination table

Designate a prefix for the ID of the destination table.

Table IDs are based on file names (excluding extensions) that coincide with the URL specified in the properties for "Source data file group URL in GCS".

[% format character string addressable] [variable expansion addressable]

Schema settings

Designate schema for the destination table.

All multiple tables share the same schema. In such cases, all data to be written must also share the same schema. If schema vary, an error will result.

When importing from a JSON format file, it is possible to skip this step.

*When you click the "Edit using JSON" link, you can specify a JSON format character string directly.

In cases of non-empty tables

Select an action to perform when tables are available for data in BigQuery.

  • Additions: Load additional data.
  • Overwrite: Empty tables and write.
  • Error: If tables are not empty, operations will fail.
BLOCK memos Make any comments regarding BLOCKS.
Repetition attempts Designate the number of repetitions to occur in the case of an error or time out in BigQuery.
Minimum duration for time out Indicate how many seconds to wait for a response (time out) in BigQuery. It will double with every repetition. The maximum value for the time increase between repetitions is set under 'Maximum duration for time out" in the properties.
Maximum duration for time out As time outs increase from the "Minimum duration for time out", they reach a maximum value. It is defined in units of seconds.
File format

Choose a file format from those in GCS. Permissible formats are as follow.

  • CSV
CSV delimitation characters

When a file format is CSV, choose, or specify, delimitation characters. Characters that can be used for delimitation are as follow.

  • Comma
  • Tab
  • Pipe
  • Other

If you choose "Other", you can specify the character you wish to use for delimitation in the field next to "Other".

Number of skipped rows When files are in CSV format, you can define the number of lead rows to be skipped.
Permit rows with insufficient fields When files are in CSV format, you can select whether or not to permit rows with insufficient fields.
Designate quotation marks Define the characters to be used for quotation when using CSV file format.
Include quoted character strings in new rows When files are in CSV format, you can select whether or not to have rows with character strings enclosed in quotes be included in subsequent rows.
Number of permissible bad records Define the number of bad records to be permitted. When this limit is reached, the next bad record will result in failure to read.
Ignore extra fields Define whether or not to ignore excess fields.
File count check URL Before importing files, designate a URL to perform a file count check. Of you omit this step, no check will be performed.
[% format character string addressable] [variable expansion addressable]
Check count Designate the maximum number to times to check for files.
Time between checks Specify the number of seconds to wait before performing an additional file count check In the event files are not discovered during the initial file check.