IoT Board Guide: File-Receiving Type

IoT Board Guide

IoT Board Guide: File-Receiving Type

Before starting

The trick to using IoT for your business is figuring out how to save the data you collect from IoT devices into servers or the cloud. Usually this requires technical knowledge to set up, or time and cost to hire someone to do it for you.

The IoT Board fixes these problems by making it simple for anyone to set up their own system for collecting and storing data.

For storage, the IoT Board uses Google’s highly-secure Google Cloud Platform , giving you access to the fastest Big Data processing available.

What are IoT Boards?

IoT Boards allow users to create systems to collect data from IoT devices and store that data into the cloud.

IoT ボード概略図

BLOCKS creates this system automatically within the user's Google Cloud Platform (GCP) project. Creating an IoT Board does not require specialized knowledge of GCP.

   Because the IoT Board's system is created within the user's GCP project, GCP service charges will apply separately from BLOCKS fees.

There are two types of IoT Board based on the type of data transferred.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages), such as acceleration, temperature, or humidity, from devices then stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Big Data Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

How to create IoT Boards

This page explains how to create a file-receiving IoT Board.

For information regarding message-receiving IoT Boards, refer to IoT Board Guide: Message-Receiving Type

Note for Self-Service Plan users:
Since IoT Boards create a system within the user’s GCP project, you will need to prepare a GCP project and service account key before creating an IoT Board.

Refer to Getting started with GCP: Part 1 from the Engineer’s Blog for help preparing these. Since the blog post focuses on using BigQuery BLOCKS, project and services account names often use the keyword “bigquery”. However, you are free to name these things however you like. Additionally, the “Registering your GCP project into BLOCKS” section does not apply to IoT Boards, and can be ignored.

As a Google account will be necessary, those without one will need to make one here before continuing.

General outline for creating IoT Boards

To create a new IoT Board, click the “Create Board” button at the top of the Board list and follow the instructions that appear.

The general steps for creating an IoT Board are as follows:

  1. Create IoT Board
    Assign a name and choose the Board type.
  2. GCP Service Account Settings
    Designate the GCP service account to be used with the IoT Board.
  3. Entry Point Settings
    Configure the destination for data sent from devices to the IoT Board.
  4. Storage Settings
    Configure how the IoT Board will store the data it has received and processed.
  5. Big Data Board Connection Settings
    Designate a Big Data Board Flow that will execute when the IoT Board stores the data it receives.
  6. Confirm Settings
    Review/confirm settings and finish Board creation.

Each of these will be explained in detail below.

Create IoT Board

Click “Create Board” button at the top of the Board list to bring up the menu for creating new Boards.

新規ボード作成

Select IoT Board from this list and click “Next”.

IoT ボード選択

Choose which type of IoT Board to create and click “Next”.

ファイル受信タイプ選択

Refer to the information below for choosing the Board type.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages) such as acceleration, temperature, or humidity, and stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects data as files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Big Data Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

Enter a name for the Board that will be easy to remember to help with managing your Boards later.

ボード名設定

Click the “Next” button to continue.

GCP Service Account Settings

   This step is for Self-Service Plan users only

You will select your GCP service account and enable APIs required for using GCP services in this section.

GCP サービスアカウント設定
Select a GCP service account

Choose which GCP service account you will use from the list.

Enable APIs

If there are APIs that do not have a checkmark, follow these steps to enable them:

  1. Click the API’s link.
  2. Click the “Enable” button at the top of the Google API Console screen.
  3. Once the “Enable” button changes to “Disable”, close the Google API Console and return to the ML Board setup screen.

Once finished with the above, click the API’s “Check” button and confirm that a checkmark appears for the API.

You may need to wait a bit of time for the checkmark to appear. If it does not appear, try clicking the “Check” button and waiting again. Depending on the circumstances, this process can take a bit of time.

Once you have selected your GCP service account and all APIs are enabled, click “Next” to continue.

Entry Point Settings

You will set the URL for your IoT Boards entry point in this section.

エントリーポイント設定

Entry points are where data is sent on its way from devices to IoT Boards and are denoted by a URL. Devices use the HTTP POST method to send data to this URL.

エントリーポイント説明図

The entry point's URL is formatted as shown below. Only one portion can be configured by the user.

https://magellan-file-<*****>-dot-<Project  ID>.appspot.com

Item Explanation
<*****>

A string of characters used to specify this IoT Board’s entry point. You can configure this portion. By default, it is set as a random 16-character string.

  • Only lower case letters (a-z), numbers (0-9), and hyphens (-) may be used.
  • The last character must be a letter or number.
  • The number of characters depends on the length of the Project ID, and ranges from 15 to 27.

When creating several IoT Boards for use with one GCP project, take care not to use the same string more than once.

<Project ID> This is set automatically to the pertinent GCP Project ID and cannot be changed.

Click "Next" once you have finished configuring the entry point.

Storage settings

The Google Cloud Storage (GCS) bucket where data sent from devices will be saved is configured in this section. It is also possible to configure saving file storage records (logs) to BigQuery.

ストレージ設定

Choose a GCS bucket where files will be saved in the setting "Select a GCS (Google Cloud Storage) Bucket". Buckets must be created within GCS prior to creating the IoT Board.

There is also an optional setting to save logs into BigQuery. Click “Save logs to BigQuery” and configure the BigQuery dataset and table that will store the logs.

The IoT Board will automatically create a new dataset and table if they do not already exist. The fields "gcs_url" and "timestamp" are always included in tables used by the IoT Board.

Field name Data type Explanation
gcs_url STRING The GCS URL of the saved file (gs://bucketname/filename)
timestamp TIMESTAMP The date/time the file was saved

Additional schema can be added if there is other information you would like to record into the logs. This information is included with the data sent to the IoT Board and should be formatted as follows: "Field name added under Schema" = "Information to be recorded"

For example, we can add a field within schema called "author" to record a log on the owner of the file's information. If we add the parameter "author=MAGELLAN" to data sent from the device, the IoT Board will record the value "MAGELLAN" into the log's "author" field.

Outline of sending data from devices to IoT Boards

If an existing table's name is entered into the IoT Board's "Table" field, BLOCKS will expect to find the fields entered under its "Schema" within that table. An error will occur if the fields "gcs_url" and "timestamp" do not exist.

Click the "Next" button to continue.

Big Data Board Connection Settings

In this section, you can configure a Flow on a Big Data Board to execute automatically when the files are saved to GCS.

Big Data ボード連携設定

If you want to do this, click "Create a Big Data Board connection" and select the Flow you want to automatically execute.

The following parameters will be passed to the Flow.

Parameter name Explanation
gcs_url The GCS URL of the saved file (gs://bucketname/filename)
target_time The instant the file was saved in UTC (2016-10-13 12:39:10.234)
filename File name of the saved file

Any other parameters included in the data sent to the IoT Board are also passed to the Flow.

Big Data ボード連携解説図

   An API token with a description of "API Token for IoT Board " will be added to the linked Bigdata Board. Make sure not to delete this token or the link between Boards will not function. You can freely configure the API token's description.

Click “Next” to continue to the confirmation screen.

Confirm Settings

You can review and confirm your settings on this screen.

入力内容の確認

If you find any mistakes, click the “Back” button to return to the relevant settings screen. Once you have fixed everything, return to the Confirm Settings screen and click “Finish”.

You will be switched to the IoT Board's details screen once it has been created. This sometimes takes a bit of time.

Sending data to IoT Boards

Use the HTTP POST method to send from data from devices to the IoT Board's "Entry Point URL”. Encode the data into the application/x-www-form-urlencoded format.

The following parameters (formatted as “parameter name=value”) must be included in data sent to the IoT Board:

Information to send Parameter name Value
API token key The "API Token" value from the IoT Board details screen
File name filename The file name to use when saving to GCS
File data to send content Base64 encoded data

You can also send any other parameters you wish to. If you use the Big Data Board connection feature, the IoT Board will pass these parameters to the Flow you designate.

If you have added fields other than the defaults to be saved as logs, give your parameters the same names as those fields.

For example, let's imagine saving a file to GCS as below:

  • We'll save a text file to GCS.
    • The file name is "sample.txt".
    • The text file contains the string "hello, world".
  • We've added the field "author" to be saved into the logs.
  • We've created a connection with a Big Data Board's Flow.
    • The Flow references a variable "foo" from an external source.

The data sent from the device to the IoT Board would be as follows (this example is just part of the HTTP header):

POST /upload HTTP/1.1
Content-Type: application/x-www-form-urlencoded
    
key=d29b76ac16181816ac7bd678&filename=sample.txt&content=aGVsbG8sIHdvcmxkCg%3D%3D&author=MAGELLAN&foo=bar

Each parameter is explained below:

Parameter Explanation
key=d29b76ac16181816ac7bd678 The API token.
filename=sample.txt The file name to be used when the data is stored to GCS.
content=aGVsbG8sIHdvcmxkCg%3D%3D The data to be stored into GCS. After being encoded into Base64, the "hello, world" string is encoded once more into the application/x-www-form-urlencoded format.
author=MAGELLAN An optional parameter. This parameter is being used to give the value "MAGELLAN" to the "author" field that was added into logs. This parameter will also be passed along to the linked Flow, where the value "MAGELLAN" can be referenced by a variable named "author".
foo=bar An optional parameter. This parameter will be passed to the linked Flow where the value "bar" can be referenced by the variable "foo".

The parameters "gcs_url", "target_time", and "filename" will also be sent to the linked Flow. For details, refer to Bigdata Board Link Settings.

   The "Entry Point URL" and "API Token" can be confirmed on the IoT Board's details screen. To do this, click the Board's "Show Details" button on the bottom of Board list screen.

GCP Service Charges

IoT Boards create an environment within the user's GCP project that makes use of several GCP services.

As such, GCP service charges will apply separately from MAGELLAN BLOCKS fees. These charges vary depending on the service. For details, refer to the pricing page for each service used by IoT Boards.

How to use IoT Boards

This section demonstrates how to send an image file from a device to an IoT Board, then execute a Big Data Board's Flow to analyze the image.

Demonstration Board specifications

The image below shows a linked IoT Board and Big Data Board.

Sample outline
  • The device sends data to the IoT Board. In this case, the device is also sending information about the file's owner (author=MAGELLAN).
  • The IoT Board saves the image file into a GCS bucket. It also saves the owner information within a log (in the author field).
    After the file has been saved, the linked Big Data Board's Flow executes.
  • Within the Flow, the Vision API (General Purpose) BLOCK performs color analysis on the saved image and the results are exported to the logs section by the Output to log BLOCK.

Creating the IoT Board

First, we will prepare the Big Data Board's Flow as shown below. The chart is numbered in order of BLOCK placement into the Flow. The Start of Flow and End of Flow BLOCKS are omitted from the chart.

# BLOCK Settings
1 Vision API (General Purpose)
  • GCP service account: the relevant GCP service account
  • GCS Image URL: ${gcs_url}
  • Information to detect: color analysis only
2 Output to log
  • Data export variable: _
3 Output to log
  • Data export variable: author

Next, we will create an IoT Board the settings shown below. Feel free to change specific values like bucket and table names if you are following along to test this out yourself.

Screen Item Contents
Entry Point Settings Entry point URL Leave as default.
Storage Settings GCS bucket Set to magellan-iot-sample
Logs

Check the box to save logs to BigQuery.

  • Data set: samples
  • Table: logs
  • Additional field: author (STRING format)
Bigdata Board Connection Settings Link Select the Flow described above (containing the Vision API and Output to log BLOCKS).

Testing it out

We'll analyze the sample image below for dominant colors.

Dominant color analysis sample image

This test will be performed using the Unix-based "curl" and "base64" commands in the terminal.

First, we'll prepare the image by executing the following command:

curl https://storage.googleapis.com/blocks-docs/basic-guide/image_properties_sample.jpg | base64 -o sample.jpg

The "curl" command downloads the sample image. The file is then encoded into Base64 using the "base64" command. Finally, the file name is set to "sample.jpg".

Next, we'll send the image file to the IoT Board.

To do this, we'll use the curl command shown below. Since the API token ("key" parameter) is unique to each IoT Board, be sure to use your IoT Board's API token if you test this out on your own.

curl --data key=d29b76ac16181816ac7bd678 --data filename=sample.jpg --data-urlencode content@sample.jpg --data author=MAGELLAN https://magellan-iot-2edblhnt-dot-magellan-iot-sample.appspot.com/upload

By running that command, the image file is sent to GCS to be saved and then analyzed by the Flow on the Big Data Board. Below are the results from the Flow's execution log.

[2016-10-14 18:56:29.184]  info : Job-16 start
[2016-10-14 18:56:29.223]  info : VersatileVisionApi step start
[2016-10-14 18:56:29.234]  info : VersatileVisionApi: annotate for file gs://magellan-iot-sample/sample.jpg start
[2016-10-14 18:56:30.724]  info : VersatileVisionApi: annotate for file gs://magellan-iot-sample/sample.jpg finished
[2016-10-14 18:56:30.738]  info : VersatileVisionApi step finished
[2016-10-14 18:56:30.778]  info : Print step start
[2016-10-14 18:56:30.794]  info : Variable: '_'
{
  "imagePropertiesAnnotation": {
    "dominantColors": {
      "colors": [
        {
          "pixelFraction": 0.03975622,
          "color": {
            "green": 23,
            "blue": 24,
            "red": 19
          },
          "score": 0.15343559
        },
        {
          "pixelFraction": 0.25660831,
          "color": {
            "green": 213,
            "blue": 130,
            "red": 156
          },
          "score": 0.089890622
        },
        {
          "pixelFraction": 0.0071575367,
          "color": {
            "green": 243,
            "blue": 245,
            "red": 234
          },
          "score": 0.044020534
        },
        {
          "pixelFraction": 0.0072284034,
          "color": {
            "green": 165,
            "blue": 124,
            "red": 131
          },
          "score": 0.030882463
        },
        {
          "pixelFraction": 0.02898448,
          "color": {
            "green": 52,
            "blue": 54,
            "red": 46
          },
          "score": 0.10213716
        },
        {
          "pixelFraction": 0.027141945,
          "color": {
            "green": 85,
            "blue": 89,
            "red": 76
          },
          "score": 0.085689932
        },
        {
          "pixelFraction": 0.018425342,
          "color": {
            "green": 119,
            "blue": 122,
            "red": 109
          },
          "score": 0.053121205
        },
        {
          "pixelFraction": 0.17454468,
          "color": {
            "green": 209,
            "blue": 101,
            "red": 144
          },
          "score": 0.036085345
        },
        {
          "pixelFraction": 0.0090000713,
          "color": {
            "green": 116,
            "blue": 136,
            "red": 107
          },
          "score": 0.033624593
        },
        {
          "pixelFraction": 0.028913613,
          "color": {
            "green": 237,
            "blue": 153,
            "red": 184
          },
          "score": 0.030864337
        }
      ]
    }
  },
  "gcs_url": "gs://magellan-iot-sample/sample.jpg",
  "timestamp": 1476438988.0
}
[2016-10-14 18:56:30.809]  info : Print step finished
[2016-10-14 18:56:30.852]  info : Print step start
[2016-10-14 18:56:30.871]  info : Variable: 'author'
"MAGELLAN"
[2016-10-14 18:56:30.885]  info : Print step finished
[2016-10-14 18:56:30.926]  info : Job-16 finished