IoT Board Guide: Message-Receiving Type

IoT Board Guide

IoT Board Guide: Message-Receiving Type

Before starting

The trick to using IoT for your business is figuring out how to save the data you collect from IoT devices into servers or the cloud. Usually this requires technical knowledge to set up, or time and cost to hire someone to do it for you.

The IoT Board fixes these problems by making it simple for anyone to set up their own system for collecting and storing data.

For storage, the IoT Board uses Google’s highly-secure Google Cloud Platform , giving you access to the fastest Big Data processing available.

What are IoT Boards?

IoT Boards allow users to create systems to collect data from IoT devices and store that data into the cloud.

IoT Board outline

BLOCKS creates this system automatically within the user's Google Cloud Platform (GCP) project. Creating an IoT Board does not require specialized knowledge of GCP.

Because the IoT Board's system is created within the user's GCP project, GCP service charges will apply separately from BLOCKS fees.

There are two types of IoT Board based on the type of data transferred.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages), such as acceleration, temperature, or humidity, from devices then stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Big Data Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

How to create IoT Boards

This page explains how to create a message-receiving IoT Board.

For information regarding file-receiving IoT Boards, refer to IoT Board Guide: File-Receiving Type.

Note for Self-Service Plan (Free Trial) users:

  • The GCP service account used with the IoT Board must be given owner privileges.

    If the GCP service account does not have owner priveledges set, access the IAM page in the GCP console and change the relevant GCP service account’s role to “Owner” (Steps 1 → 2 → 3 in the image below).

    GCP console IAM screen

General outline for creating IoT Boards

To create a new IoT Board, click the “Create Board” button at the top of the Board list and follow the instructions that appear.

The general steps for creating an IoT Board are as follows:

  1. Create IoT Board
    Assign a name and choose the Board type.
  2. GCP Service Account Settings
    Designate the GCP service account to be used with the IoT Board.
  3. Entry Point Settings
    Configure the destination for data sent from devices to the IoT Board.
  4. Processing Settings
    Designate how the IoT Board will process data it receives.
  5. Storage Settings
    Configure how the IoT Board will store the data it has received and processed.
  6. Confirmation
    Review/confirm settings and finish Board creation.

Each of these will be explained in detail below.

Create IoT Board

Click “Create Board” button at the top of the Board list to bring up the menu for creating new Boards.

新規ボード作成

Select IoT Board from this list and click “Next”.

IoT ボード選択

Choose which type of IoT Board to create and click “Next”.

メッセージ受信タイプ選択

Refer to the information below for choosing the Board type.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages) such as acceleration, temperature, or humidity, and stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects data as files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Big Data Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

Enter a name for the Board that will be easy to remember to help with managing your Boards later.

ボード名設定

Click the “Next” button to continue.

GCP Service Account Settings

This step is for Self-Service Plan users only

You will select your GCP service account and enable APIs required for using GCP services in this section.

GCP サービスアカウント設定
Select a GCP service account

Choose which GCP service account you will use from the list.

Enable APIs

If there are APIs that do not have a checkmark, follow these steps to enable them:

  1. Click the API’s link.
  2. Click the “Enable” button at the top of the Google API Console screen.
  3. Once the “Enable” button changes to “Disable”, close the Google API Console and return to the ML Board setup screen.

Once finished with the above, click the API’s “Check” button and confirm that a checkmark appears for the API.

You may need to wait a bit of time for the checkmark to appear. If it does not appear, try clicking the “Check” button and waiting again. Depending on the circumstances, this process can take a bit of time.

If you see a , the issue may be one of the following:

  • The relevant API is not enabled.
    Click the next to the API’s name and check if the API is enabled on the page that opens. If not, click the Enable button.
  • Your GCP service account role is not set to “Editor”.
    Open the menu () from the upper-left of the GCP console and select “IAM & Admin”. From the IAM menu, confirm that your role is set to Editor. If not, change it to editor.
  • Billing is not enabled on the relevant GCP project.
    Open the menu () from the upper-left of the GCP console and select Billing. Enable billing for your project, if you have not already done so.

Once you have selected your GCP service account and all APIs are enabled, click “Next” to continue.

Entry Point Settings

You will set the URL for your IoT Boards entry point in this section.

エントリーポイント設定

Entry points are where data is sent on its way from devices to IoT Boards and are denoted by a URL. Devices use the HTTP POST method to send data to this URL.

Entry point image

The entry point's URL is formatted as shown below. Only one portion can be configured by the user.

https://magellan-iot-<*****>-dot-<project ID>.appspot.com

Item Explanation
<*****>

A string of characters used to specify this IoT Board’s entry point. You can configure this portion. By default, it is set as a random 16-character string.

  • Only lower case letters (a-z), numbers (0-9), and hyphens (-) may be used.
  • The last character must be a letter or number.
  • The number of characters depends on the length of the Project ID, and ranges from 15 to 27.

When creating several IoT Boards for use with one GCP project, take care not to use the same string more than once.

<Project ID> This is set automatically to the pertinent GCP Project ID and cannot be changed.

Click "Next" once you have finished configuring the entry point.

Processing Settings

Configure processing related settings in this section.

プロセス設定

This refers to how the IoT Board will process the data it receives at the entry point.

You have the option to configure the following Google Container Engine (GKE) related settings:

Setting Explanation
Machine Type

Designate the type of virtual machine to be used with GKE.

  • n1-standard-1 (1 virtual CPU, 3.75 GB memory)
  • g1-small (0.5 virtual CPU, 1.70 GB memory)

The default setting is n1-standard-1.

Charges based on GCE machine type pricing will apply separately from MAGELLAN BLOCKS fees.

VM Nodes

Configure the number of virtual machine instances (nodes) to be used.

Using up to 5 VM nodes is free of charge. From 6 nodes and up, GKE charges apply according to machine type (separate from MAGELLAN BLOCKS fees).

Click "Next" to continue to the next section.

Storage Settings

Configure the Board's storage settings in this section.

ストレージ設定

These settings configure where and how the data will be stored into the cloud. It’s possible to store data separately according to data type.

Data types are identified by “message type” identifiers. In addition to the type of data sent, message types also act as identifiers for the storage location.

Data sent from devices to an IoT Board must contain these message types since they are how the IoT Board decides where to store data it receives.

メッセージタイプ解説図

Set profiles for the data's storage locations in this section. You can be add, edit, or delete storage locations by doing the following:

  • Click the “Add” button to configure a new storage location.
  • Click a storage location's pencil button to edit it.
  • Click a storage location's trash button to delete it.

The current version of IoT Boards support storing data to BigQuery only. As such, the storage location's settings are all related to BigQuery settings.

Configure the following information for each BigQuery storage location:

Item Explanation
Message type

Designate the message type of the data to be saved.

Data identified by this message type will be saved to the BigQuery table configured in the settings that follow.

Dataset Designate the ID of the BigQuery dataset where this data will be saved.
Table Designate the name of the BigQuery table where this data will be saved.
Schema

Designate the table's schema.

Table division

Designate if the data storage table will be divided by day, month, or not.

When using table division, the setting "Times for table division (field)" is also configured.

Regarding table division:

“Times for table division (field)” can be set to either “Use time that data was sent to the IoT Board”, or to a field with the TIMESTAMP data type. When selecting “Use time that data was sent to the IoT Board”, the table will be divided according to the time the IoT Board received the data.

Choosing to divide tables by day or month results in suffixes being added to the table’s name. These suffixes are formatted as follows:

  • Divide by day: _%Y%m%d
  • Divide by month: _%Y%m01

%Y, %m, and %d are replaced with the times stored in the “Times for table division (field)”. The format is year (4 digits), month (2 digits), and day (2 digits). As an example, a table named “sample” with division by day chosen would be divided into tables named “sample_20160905”, and “sample_20160906”.

Click "Next" once you have configured all of the storage locations for your data.

Confirmation

You can review and confirm your settings on this screen.

入力内容の確認

If you find any mistakes, click the “Back” button to return to the relevant settings screen. Once you have fixed everything, return to the Confirm Settings screen and click “Finish”.

You will be switched to the IoT Board's details screen once it is created. This sometimes takes a bit of time.

Sending data to IoT Boards

This section explains the process of sending data from devices to IoT Boards.

The following three pieces of information are required when sending data:

  • The entry point URL
  • An API token
  • A message type

You can be check this information on the IoT Board's details screen. Open this by clicking on an IoT Board from the Board list and then clicking its "Show Details" button.

Use the HTTP POST to send data from devices to the entry point URL.

Data sent should be in JSON format and include the API token and message type information.

API に送るデータの解説図

The following is an example of JSON format data that is ready to send to an IoT Board:

{
  "api_token": "************************",
  "logs": [
    {
      "type": "message",
      "attributes": {
        "date": 1473642000,
        "name": "device_001",
        "message": "hello"
      }
    }
  ]
}

Each member (name/value pair separated by a colon) is explained below:

Name Explanation of value
"api_token" Designate the API token with a string.
"logs"

Designate the data as an array of objects.

  • Objects: Formatted as {...}. The braces contain at least one member (multiple members are separated by commas.)
  • Array: Formatted as [{...},{...}]
"type" Designate the message type. The data's storage location (a BigQuery table) is determined by this message type.
"attributes"

Designate the data to be stored as objects.

For BigQuery storage, names refer to the field names of the table where data will be stored, and values refer to the actual data to be stored into those fields.

For fields dealing with date/time, data must be designated using Unix timestamp formatting (in this example, the name is “date” and the value is 1473642000).

The following example puts together two types of data into JSON format ready to be sent to an IoT Board.

{
  "api_token": "************************",
  "logs": [
    {
      "type": "message",
      "attributes": {
        "date": 1473642000,
        "name": "device_001",
        "message": "hello"
      }
    },
    {
      "type": "temperature",
      "attributes": {
        "date": 1473642000,
        "name": "device_fukuoka_001",
        "temperature": 24.5
      }
    }
  ]
}

GCP Service Charges

IoT Boards create an environment within the user's GCP project that makes use of several GCP services.

As such, GCP service charges will apply separately from MAGELLAN BLOCKS fees. These charges vary depending on the service. For details, refer to the pricing page for each service used by IoT Boards.

How to use IoT Boards

In this section, we’ll demonstrate creating an actual IoT Board, based on the information in How to create IoT Boards, that collects two types of data from IoT devices.

作成する IoT ボードの仕様概略図

Demonstration Board specifications

The Board will collect the following two types of data:

Data type Details (content: format)
Message
  • Date/time the message was sent: UNIX timestamp
  • Name of the device sending the message: String
  • Message: String
Temperature
  • Date/time the temperature was sent: UNIX timestamp
  • Name of the device sending the temperature data: String
  • Temperature: Float

The storage settings for this data will be as follows:

Message data storage:
Item Value
Message type message
Dataset sample
Table messages
Schema
Field name Data format Mode
date TIMESTAMP REQUIRED
name STRING REQUIRED
message STRING NULLABLE
Table division Division by day, using the "date" field
Temperature data storage:
Item Value
Message type temperature
Dataset sample
Table temperatures
Schema
Field name Data format Mode
date TIMESTAMP REQUIRED
name STRING REQUIRED
temperature FLOAT NULLABLE
Table division Division by day, using the "date" field

Making the IoT Board

Using this information as a base, we will create an IoT Board with the following settings:

Screen Item Contents
Create IoT Board Board name Can be assigned freely.
Type Select "Message-receiving type".
GCP Service Account Settings File name Select your GCP service account.
Entry Point Settings Optional Setting Leave as default.
Processing Settings Optional Setting Leave as default.
Storage Settings Storage location Add the information recorded in "Message data storage" and "Temperature data storage" from Demonstration Board specifications.

Testing it out

Now, we’ll try sending data to our IoT Board following the information outlined in Sending data to IoT Boards. However, instead of using devices, we’ll use a PC to send the data. When sending data from a PC, it’s convenient to use Unix curl commands.

This time, we’ll use the IoT Board Connection Information and data:

IoT Board Connection Information:
Item Value
Entry Point URL https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com
API token *****
First data type:
Item Name Value
Message type "type" "message"
Date/time the message was sent "date" 1473642000
(2016/09/12 10:00)
Name of the device sending the message "name" "device_001"
Message "message" "hello"
Second data type:
Item Name Value
Message type "type" "temperature"
Date/time the temperature data was sent "date" 1473642600
(2016/09/12 10:10)
Name of the device sending the temperature data "name" "device_fukuoka_001"
Temperature "temperature" 24.5

The curl commands to send the data are as follows:

First data type:

curl --data '{"api_token":"*****","logs": [{"type": "message","attributes": {"date": 1473642000,"name": "device_001","message": "hello"}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

Second data type:

curl --data '{"api_token":"*****","logs": [{"type": "temperature","attributes": {"date": 1473642600,"name": "device_fukuoka_001","temperature": 24.5}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

Or, the two data types can be sent together in one curl command as shown below:

curl --data '{"api_token":"*****","logs": [{"type": "message","attributes": {"date": 1473642000,"name": "device_001","message": "hello"}},{"type": "temperature","attributes": {"date": 1473642600,"name": "device_fukuoka_001","temperature": 24.5}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

We’ll confirm the results in BigQuery.

Message data:

Message data stored in BigQuery

Temperature data:

Temperature data stored in BigQuery