IoT Board Guide: Message-Receiving Type

IoT Board Guide

IoT Board Guide

Before starting

The key to applying IoT technology to your business is figuring out how to store data collected by devices into the cloud. Setting up servers, cloud storage, and the like takes specialized knowledge, not to mention money and valuable time.

IoT Boards take care of this, collecting data and storing it into Google Cloud Platform (GCP). GCP offers secure processing of vast amounts of data at the highest speed available.

With IoT Boards, it’s simple to quickly set up a system to collect and store data from IoT devices, even without technical knowledge.

What are IoT Boards?

IoT Boards allow users to create systems to collect data from IoT devices (referred hereafter simply as devices) and store that data into the cloud.

IoT Board outline

BLOCKS creates this system automatically within the user's Google Cloud Platform (GCP) project. Creating an IoT Board does not require specialized knowledge of GCP.

   Because the IoT Board's system is created within the user's GCP project, GCP service charges will apply separately from BLOCKS fees.

There are two types of IoT Board based on the type of data received.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages), such as acceleration, temperature, or humidity, from devices then stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Bigdata Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

   In the current version, IoT Boards are limited to 5 Boards per organization. [1]

How to create IoT Boards

This page explains how to create a message-receiving IoT Board.

For information regarding file-receiving IoT Boards, refer to IoT Board Guide: File-Receiving Type.

Since IoT Boards create a system within the user’s GCP project, it is necessary to create a GCP project and service account before creating an IoT Board.

Refer to Getting started with GCP: Part 1 from the Engineer’s Blog for help preparing these. Since the blog post focuses on using BigQuery BLOCKS, project and services account names often use the keyword “bigquery”. However, you are free to name these things however you like. Additionally, the “Registering your GCP project into BLOCKS” section does not apply to IoT Boards, and can be ignored.

As a Google account will be necessary, those without an account will need to make one here before continuing.

General outline for creating IoT Boards

New IoT Boards are created by clicking the “Add Board” button under IoT Boards on the Board list (the first screen shown after logging into BLOCKS), and following the instructions that appear.

The general steps for creating an IoT Board are as follows:

  1. Create IoT Board
    Assign a name and choose the Board type.
  2. GCP Service Account Settings
    Designate the GCP service account to be used with the IoT Board.
  3. Entry Point Settings
    Configure the destination for data sent from devices to the IoT Board.
  4. Processing Settings
    Designate how the IoT Board will process data it receives.
  5. Storage Settings
    Configure how the IoT Board will store the data it has received and processed.
  6. Confirm Settings
    Review/confirm settings and finish Board creation.

Each of these settings is explained in detail below.

Create IoT Board

Clicking “Add Board” from the IoT Board section of the Board list brings up the "Create IoT Board" screen.

From here, the IoT Board's name and Board type are configured.

Easy to remember names make Board management simpler. Be aware that once created, an IoT Board’s name cannot be changed.

Refer to the information below for choosing the Board type.

Type Explanation
Message-receiving type
  • Collects comparatively small data (messages) such as acceleration, temperature, or humidity, and stores it into a BigQuery table.
Message-receiving type outline
File-receiving type
  • Collects data as files, such as images, recordings, etc., from devices and stores them into Google Cloud Storage.
  • Logs (file name, save date, etc.) can also be stored (optional).
  • A Bigdata Board's Flow can be executed upon files being saved (optional).
File-receiving type outline

After assigning a Board name and type, click next to proceed to GCP Service Account Settings.

Clicking “Cancel” will return to the Board list without creating the IoT Board. Any changes made will be lost.

GCP Service Account Settings

After clicking "Next" from the Create IoT Board screen, setup will proceed to GCP Service Account Settings.

From here, upload a GCP service account key (JSON file) in one of the following ways:

  • Drag and drop the file onto the area titled “Uploading a GCP Service Account Key”.
  • Click the “Select file” button and choose a file.

If you have not finished setting up a GCP project and service account, please refer to the Engineer's Blog post Getting started with GCP: Part 1.

Since the blog post focuses on using BigQuery BLOCKS, project and services account names often use the keyword “bigquery”. However, you are free to name these things however you like. Additionally, the “Registering your GCP project into BLOCKS” section does not apply to IoT Boards, and can be ignored.

As a Google account will be necessary, those without an account will need to make one here before continuing.

Once the GCP service account key has finished uploading, click “Next” to continue to the Entry Point Settings screen.

Clicking “Cancel” will return to the Board list without creating the IoT Board. Any changes made will be lost.

Entry Point Settings

The Entry Point Settings screen comes after clicking “Next” on the GCP Service Account Settings screen.

The following settings can be configured for the entry point:

  • Enable App Engine Admin API ("Admin API")
  • Enable Google Cloud Pub/Sub API ("Cloud Pub/Sub API")
  • Connector URL settings

Entry points are where data is sent on its way from devices to IoT Boards. Entry points are denoted by a URL (the connector URL). Devices use the HTTP POST method to send data to this URL.

Entry point image

   Entry point data collection and processing is implemented using the Google App Engine (GAE) and Google Cloud Pub/Sub. While construction of this system is done on the BLOCKS side, the Admin API and Cloud Pub/Sub API must be enabled.

The process for enabling APIs is as follows:

  1. Click the API's link.
    The GCP screen will appear in a new tab.
  2. Click the "Enable" button near the top of the page.
  3. Once the "Enable" button changes to "Disable", close the tab and return to BLOCKS.
  4. Click the "Refresh API status" button.
    The API status will change to "Enabled". If the status does not change automatically, wait a moment and try clicking "Refresh API status" again.

The Connector URL is formatted as shown below. Only one portion can be configured by the user.

https://magellan-iot-<*****>-dot-<project ID>.appspot.com

Item Explanation
<*****>

A string of characters used to specify this IoT Board’s entry point. You can configure this portion. By default, it is set as a random 16-character string.

  • Only lower case letters (a-z), numbers (0-9), and hyphens (-) may be used.
  • The last character must be a letter or number.
  • The number of characters depends on the length of the Project ID, and ranges from 15 to 27.

When creating several IoT Boards for use with one GCP project, take care not to use the same string more than once.

<Project ID> This is set automatically to the pertinent GCP Project ID and cannot be changed.

After configuring the entry point, click "Next" to proceed to Processing Settings.

Clicking “Cancel” will return to the Board list without creating the IoT Board. Any changes made will be lost.

Processing Settings

Processing Settings comes after clicking the “Next” button from the Entry Point Settingsscreen.

These settings relate to how the data sent to the entry point will be processed.

The following processing settings can be configured:

  • Enable Google Container Engine (GKE) API
  • Optional Settings

   Data processing is performed using GKE. While construction of this system is done on the BLOCKS side, the GKE API must be enabled.

The process for enabling the API is as follows:

  1. Click the API's link.
    The GCP screen will appear in a new tab.
  2. Click the "Enable" button near the top of the page.
  3. Once the "Enable" button changes to "Disable", close the tab and return to BLOCKS.
  4. Click the "Refresh API status" button.
    The API status will change to "Enabled". If the status does not change automatically, wait a moment and try clicking "Refresh API status" again.

Under Optional Settings, the following settings related to GKE are configured:

Setting Name Explanation
Machine Type

Designate the type of virtual machine to be used with GKE.

  • n1-standard-1 (1 virtual CPU, 3.75 GB memory)
  • g1-small (shared-core, 1 virtual CPU, 1.70 GB memory)

The default setting is n1-standard-1.

Charges based on GCE machine type pricingwill apply separately from MAGELLAN BLOCKS fees.

VM Nodes

Configure the number of virtual machine instances (nodes) to be used.

Using up to 5 VM nodes is free of charge. From 6 nodes and up, GKE charges apply according to machine type (separate from MAGELLAN BLOCKS fees).

After finishing with Processing Settings, click "Next" to continue to the Storage Settings screen.

Clicking “Cancel” will return to the Board list without creating the IoT Board. Any changes made will be lost.

Storage Settings

Clicking the “Next” button from Processing Settings brings up the Storage Settings screen.

Storage refers to where data sent from devices will be saved. Data can be separated according to data type when stored.

Data types are identified by “message types”. In addition to the type of data sent, these message types are also identifiers for the storage location.

Devices must include message types with the data they send to IoT Boards. The IoT Board decides where to store received data based on its message type.

Message type identifiers

In this section, profiles regarding storage locations for received data are configured. These storage location profiles can be added, edited, and deleted.

  • To add a new profile, click “Add storage location”.
  • To edit a profile, its corresponding pencil icon.
  • To delete a profile, click its corresponding trash icon.

The current version of IoT Boards only support storage to BigQuery. As such, storage location settings are all related to BigQuery information.

For BigQuery storage, information to be added/edited is as follows:

Item Explanation
Message type

Designate the message type of the data to be saved.

Data identified by this message type will be saved to the BigQuery table configured in the setting below.

Dataset Designate the ID of the BigQuery dataset where this data will be saved.
Table Designate the name of the BigQuery table where this data will be saved.
Schema

Designate the table's schema.

Table division

Designate if the data storage table will be divided by day, month, or not.

When using table division, the setting "Times for table division (field)" is also configured.

Regarding table division:

“Times for table division (field)” can be set to either “Use time that data was sent to the IoT Board”, or to a field with the TIMESTAMP data type. When selecting “Use time that data was sent to the IoT Board”, the table will be divided according to the time the IoT Board received the data.

Choosing to divide tables by day or month results in suffixes being added to the table’s name. These suffixes are formatted as follows:

  • Divide by day: _%Y%m%d
  • Divide by month: _%Y%m01

%Y, %m, and %d are replaced with the times stored in the “Times for table division (field)”. The format is year (4 digits), month (2 digits), and day (2 digits). As an example, a table named “sample” with division by day chosen would be divided into tables named “sample_20160905”, and “sample_20160906”.

After finishing, click the “Next” button to continue to the Confirm Settings screen. Clicking “Back” will return to Processing Settings (all changes made so far will be saved).

Clicking “Cancel” will return to the Board list without creating the IoT Board. All setting changes will be lost.

Confirm Settings

Clicking "Next" from Storage Settings brings up the Confirm Settings screen.

In this section, you can review/confirm all settings made so far. If there are any mistakes, click the “Back” button to return to the screen containing the error. Once all problems have been fixed, return to the Confirm Settings screen and click “Finish”.

Clicking “Cancel” will return to the Board list without creating the IoT Board. Any changes made will be lost.

Clicking "Finish" brings up a list of resources related to the IoT Board that will be created in the GCP project. Be careful not to accidentally delete any of these resources within your GCP project after the IoT Board has been created.

If you are ready to create the IoT Board with the settings as they are, clicking "OK" will start Board creation.

Clicking "Cancel" will return to the Confirm Settings screen without creating the IoT Board.

Creation of the IoT Board will take a bit of time. Once complete, the screen will switch to the IoT Board's Details page. A message saying that IoT Board creation has finished will appear at the top of the screen.

Sending data to IoT Boards

This section explains the process of sending data from devices to IoT Boards.

The following three pieces of information are required when sending this data:

  • Connector URL
  • API token
  • Message type

The Connector URL and API token can be checked from the "IoT Board Connection Information" section of an IoT Board's "Details" screen. This is accessed by clicking a Board's "Details" button from the Board list.

IoT Board Connection Information

Message type can be checked from the "Storage" section of "Settings Summary" within the IoT Board's "Details" screen.

Settings Summary

When sending data from devices to an IoT Board, please use the HTTP POST method with this Connector URL.

This data should be in JSON format. Include the API token and Message type in this JSON data.

Send data with API token

JSON format data sent to IoT Boards is formatted like the following:

{
  "api_token": "************************",
  "logs": [
    {
      "type": "message",
      "attributes": {
        "date": 1473642000,
        "name": "device_001",
        "message": "hello"
      }
    }
  ]
}

The meaning of each member (a name/value pair separated by a colon) is explained below:

Name Explanation of value
"api_token" Designate the API token with a string.
"logs"

Designate the data as an array of objects.

  • Objects: Formatted as {...}. The braces contain at least one member (multiple members are separated by commas.
  • Array: Formatted as [{...},{...}]
"type" Designate the message type. The location where data is saved is determined by the message type. In BigQuery's case, this refers to the table where data is to be stored.
"attributes"

Designate the data to be stored as objects.

For BigQuery storage, names refer to the field names of the table where data will be stored , and values refer to the actual data to be stored in those fields.

For fields dealing with date/time, data must be designated using Unix timestamp formatting (in this example, the name is “date” and the value is 1473642000).

The following example puts together two types of data into JSON format to be sent to an IoT Board.

{
  "api_token": "************************",
  "logs": [
    {
      "type": "message",
      "attributes": {
        "date": 1473642000,
        "name": "device_001",
        "message": "hello"
      }
    },
    {
      "type": "temperature",
      "attributes": {
        "date": 1473642000,
        "name": "device_fukuoka_001",
        "temperature": 24.5
      }
    }
  ]
}

GCP Service Charges

IoT Boards create an environment within the user's GCP project that makes use of several GCP services.

As such, GCP service charges will apply separately from MAGELLAN BLOCKS fees. These charges vary depending on the service. For details, refer to the pricing page for each service used by IoT Boards.

How to use IoT Boards

In this section, we’ll demonstrate creating an actual IoT Board, based on the information in How to create IoT Boards, that collects two types of data from IoT devices.

Demonstration IoT Board

Demonstration Board specifications

The Board will collect the following two types of data:

Data type Details (content: format)
Message
  • Date/time the message was sent: UNIX timestamp
  • Name of the device sending the message: String
  • Message: String
Temperature
  • Date/time the temperature was sent: UNIX timestamp
  • Name of the device sending the temperature data: String
  • Temperature: Float

The storage settings for this data will be as follows:

Message data storage:
Item Value
Message type message
Dataset sample
Table messages
Schema
Field name Data format Mode
date TIMESTAMP REQUIRED
name STRING REQUIRED
message STRING NULLABLE
Table division Division by day, using the "date" field
Temperature data storage:
Item Value
Message type temperature
Dataset sample
Table temperatures
Schema
Field name Data format Mode
date TIMESTAMP REQUIRED
name STRING REQUIRED
temperature FLOAT NULLABLE
Table division Division by day, using the "date" field

Make the IoT Board

Using this information as a base, we will create an IoT Board with the following settings:

Screen Item Contents
Create IoT Board Board name Can be assigned freely.
Type Select "Message-receiving type".
GCP Service Account Settings File name Upload a JSON format GCP service account key file.
Entry Point Settings Optional Setting Leave as default.
Processing Settings Optional Settings Leave as default.
Storage Settings Storage location Add the information recorded in "Message data storage" and "Temperature data storage" from Demonstration Board specifications.

Testing it out

Now, we’ll try sending data to our IoT Board following the information outlined in Sending data to IoT Boards. However, instead of using devices, we’ll use a PC to send the data. When sending data from a PC, it’s convenient to use Unix curl commands.

This time, we’ll use the IoT Board Connection Information and data:

IoT Board Connection Information:
Item Value
Connector URL https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com
API token *****
First data type:
Item Name Value
Message type "type" "message"
Date/time the message was sent "date" 1473642000
(2016/09/12 10:00)
Name of the device sending the message "name" "device_001"
Message "message" "hello"
Second data type:
Item Name Value
Message type "type" "temperature"
Date/time the temperature data was sent "date" 1473642600
(2016/09/12 10:10)
Name of the device sending the temperature data "name" "device_fukuoka_001"
Temperature "temperature" 24.5

The curl commands to send the data are as follows:

First data type:

curl --data '{"api_token":"*****","logs": [{"type": "message","attributes": {"date": 1473642000,"name": "device_001","message": "hello"}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

Second data type:

curl --data '{"api_token":"*****","logs": [{"type": "temperature","attributes": {"date": 1473642600,"name": "device_fukuoka_001","temperature": 24.5}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

Or, the two data types can be sent together in one curl command as shown below:

curl --data '{"api_token":"*****","logs": [{"type": "message","attributes": {"date": 1473642000,"name": "device_001","message": "hello"}},{"type": "temperature","attributes": {"date": 1473642600,"name": "device_fukuoka_001","temperature": 24.5}}]}' https://magellan-iot-*****-dot-magellan-iot-sample.appspot.com/

We’ll confirm the results in BigQuery.

Message data:

Message data stored in BigQuery

Temperature data:

Temperature data stored in BigQuery

[1] Organizations are groups of individual MAGELLAN BLOCKS users. Organizations to which your account belongs are added automatically at the time of registering a new account. As of the current release, it is not possible to add new organizations or join existing ones. As such, for the time being, “organization” means the same thing as “user”.