BLOCKS Reference

Block Reference

GCP

Load to variable from GCS

This BLOCK is currently in beta and will become unavailable post official release. Please use the official version once released.

As a beta release, there is the possibility that some functions may not execute properly. We appreciate feedback from users, through the BLOCKS Forum or direct contact, regarding bugs or ways to improve BLOCKS.

Overview

This BLOCK loads the contents of a file stored into Google Cloud Storage (GCS) into a variable.

Load to variable from GCS BLOCK overview

For example, assume a text file with the following contents is stored in GCS at gs://magellan-sample/files.txt:

sample1.png
sample2.png

Using this BLOCK with this file would set a variable to the following data:

{
  "gcs_url": "gs://magellan-sample/files.txt",
  "timestamp": 1480986550.0,
  "content": "sample1.png\nsample2.png"
}

As shown above, the variable contains the contents of the file along with other incidental information as JSON format data. The "content" portion contains the file’s contents as "sample1.png\nsample2.png" (\n represents a line break).

You can select the file format of the file from among (none), YAML, JSON, Newline delimited JSON, Binary (Base64), and CSV.

File format Explanation
(none)

"content" is set as the contents of the file just as they are.

YAML/JSON

"content" is set as the results of parsing the contents of the file according to the selected format.

For example, assume that the file contains the following YAML format text:

Basic:
  - Start of Flow
  - Parallel branch
  - Slack notice
  - Output to log
  - Construct object
  - End of Flow
API:
  - HTTP GET
  - HTTP POST
  - HTTP PUT
  - HTTP DELETE

After processing this file with this BLOCK, "content" would contain the following:

  "content": {
    "Basic": [
      "Start of Flow",
      "Parallel branch",
      "Slack notice",
      "Output to log",
      "Construct object",
      "End of Flow"
    ],
    "API": [
      "HTTP GET",
      "HTTP POST",
      "HTTP PUT",
      "HTTP DELETE"
    ]
  }
Newline delimited JSON

"content" is set as the results of converting newline delimited JSON into an array. Each line of the JSON becomes one element of the array.

For example, assume that the file contains the following newline delimited JSON text:

{"name1": "value1", "name2": "value2"}
{"name1": "value3", "name2": "value4"}

After processing this file with this BLOCK, "content" would contain the following:

  "content": [
    {
      "name1": "value1",
      "name2": "value2"
    },
    {
      "name1": "value3",
      "name2": "value4"
    }
  ]
Binary (Base64)

"content" is set as Base64 encoded contents of the file.

For example, "content" could contain something like the following:

  "content": "ZkxhQwAA……qqA55Q=="
CSV

"content" is set as the results of analyzing the contents of the file as CSV format and converting them into an array of objects. All fields in the CSV file are handled as strings during the conversion.。

CSV files are assumed to hold a header line, as shown below. The header line must have the same number of fields as the other lines, and its fields will be used as the list of field names.

field_name_1,field_name_2,field_name_3
aaa,bbb,ccc
xxx,yyy,zzz

After processing this CSV file with this BLOCK, "content" would contain the following array of objects:

  "content": [
    {
      "field_name_1": "aaa",
      "field_name_2": "bbb",
      "field_name_3": "ccc"
    },
    {
      "field_name_1": "xxx",
      "field_name_2": "yyy",
      "field_name_3": "zzz"
    }
  ]

Properties

Refer to the chart in the Overview section for more details on each format.

Property Explanation
BLOCK name Configure the name displayed on this BLOCK.
GCP service account Select the GCP service account to use with this BLOCK.
Source file GCS URL Designate the GCS URL of the file whose contents will be loading into the variable. Example: gs://magellan-sample/sample.yaml
File format Select the format of the GCS file as (none) , YAML, JSON, Newline delimited JSON, Binary (Base64), or CSV.
BLOCK memos Make any comments about this BLOCK.
CSV delimiter character

When the file format property is set to CSV, select the delimiter character used in the CSV file as Comma (,), Tab, Pipe (|), or Other.

If you select Other, enter the delimiter character used into the adjacent field.

Number of skipped rows

When the file format property is set to CSV, configure the number of lead rows to skip.

Designate quotation marks

When the file format property is set to CSV, designate the character used for quotation marks.

Results variable

Designate the variable that will store the contents of the GCS file. The data stored into the variable is formatted as follows:

{
  "gcs_url": "...",
  "timestamp": ...,
  "content": ...
}
Name Explanation
"gcs_url" The GCS URL of the file.
"timestamp" The time the file was processed expressed in UNIX time.
"content" The contents of the file.