BLOCKS Reference

Block Reference

Output specifications

Image analysis

This page details the types of results returned by the Image analysis BLOCK.

For a detailed example of using the Image analysis BLOCK, refer to Using the Google Cloud Vision API Machine Learning service.

Image analysis results were recorded based on the 9/2016 version of Google Cloud Platform  |  Google Cloud Vision API Documentation  |  Method: images.annotate

Image analysis results

By default, results of successful image analysis are exported as JSON format data in a variable named _ (underscore). The name of this variable can be change freely in the "Results storage variable" setting under the "Advanced settings" portion of the BLOCK property menu.

{
  "faceAnnotations": [{...}],
  "landmarkAnnotations": [{...}],
  "logoAnnotations": [{...}],
  "labelAnnotations": [{...}],
  "textAnnotations": [{...}],
  "safeSearchAnnotation": {...},
  "imagePropertiesAnnotation": {...},
  "gcs_url": string,
  "timestamp": number
}
Name Explanation
"faceAnnotations" The results of facial recognition. Contains various data from the analyzed image.
"landmarkAnnotations" The results of landmark (famous natural or man-made locations) recognition. Contains various data from the analyzed image.
"logoAnnotations" The results of logo (logos of famous products and companies) recognition. Contains various data from the analyzed image.
"labelAnnotations" The results of object recognition (tagging images based on the objects detected within, such as vehicles or animals). Contains various data from the analyzed image.
"textAnnotations" The results of OCR (text recognition). Contains various data from the analyzed image.
"safeSearchAnnotation" The results of adult content recognition (sexual/violent content, etc.).
"imagePropertiesAnnotation" The results of color analysis.
"gcs_url" String which displays the path for the analyzed image saved in GCS. Looks something like the following example: "gs://magellan-blocks-demo/kumamoto2.jpg"
"timestamp" Numerical value showing the date/time (UNIX format) when image analysis occured. For example, analysis performed at 10:20 PM and 37 seconds on Sept. 27, 2016 would display as 1474939237.7195294.

Other than gcs_url and timestamp, these results are only exported when their corresponding features are detected by the analysis. For example, faceAnnotations will only be returned when a face is detected in the image.

Facial recognition ("faceAnnotations")

Facial recognition results are returned as an array ([...] format). The detected facial data that makes up the array are objects ({...} format).

"faceAnnotations": [
  {...},
  {...}
]

The format of the objects showing facial detection results are explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > Facial recognition example.

{
  "boundingPoly": {...},
  "fdBoundingPoly": {...},
  "landmarks": {...},
  "rollAngle": number,
  "panAngle": number,
  "tiltAngle": number,
  "detectionConfidence": number,
  "landmarkingConfidence": number,
  "joyLikelihood": string,
  "sorrowLikelihood": string,
  "angerLikelihood": string,
  "surpriseLikelihood": string,
  "underExposedLikelihood": string,
  "blurredLikelihood": string,
  "headwearLikelihood": string
}
Name Explanation
"boundingPoly" This object is a polygon that outlines the face and head detected in the image. It outlines the face and head in a broader rectangle than the object below, fdBoundingPoly, does.
"fdBoundingPoly" This object is a polygon that outlines only the face detected in the image (a rectangle containing the ears, brow, and just below the mouth). It outlines the face more tightly than boundingPoly does.
"landmarks" An array containing objects for each part of the face (left eye, right eye, etc.).
"rollAngle" A numerical value showing the clockwise/counter-clockwise angle (roll) of the detected face. Degree range is -180 to 180.
"panAngle" A numerical value showing the angle to which the detected face is pointing left/right (yaw). Degree range is -180 to 180.
"tiltAngle" A numerical value showing the up/down angle the detected face is pointing (pitch). Degree range is -180 to 180.
"detectionConfidence" A numerical value showing confidence level of the facial recognition (i.e. how accurately was the face detected). Range is 0 to 1.
"landmarkingConfidence" A numerical value showing confidence of facial feature recognition. Range is 0 to 1.
"joyLikelihood" A string showing the likelihood of a happy expression.
"sorrowLikelihood" A string showing the likelihood of a sad expression.
"angerLikelihood" A string showing the likelihood of an angry expression.
"surpriseLikelihood" A string showing the likelihood of a surprised expression.
"underExposedLikelihood" A string showing the likelihood that the image is underexposed.
"blurredLikelihood" A string showing the likelihood that the image is blurry.
"headwearLikelihood" A string showing the likelihood that the person detected is wearing a hat.
Landmark recognition ("landmarkAnnotations")

Landmark recognition results are returned as an array. The detected landmark data that makes up the array are objects.

"landmarkAnnotations": [
  {...},
  {...}
]

The format of the objects showing landmark detection results are explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > Landmark recognition example.

{
  "mid": string,
  "description": string,
  "score": number,
  "boundingPoly": {...},
  "locations": [{...}]
}
Name Explanation
"mid" The Google Knowledge Graph entity ID of the detected landmark.
"description" The description for the above listed entity ID.
"score" A numerical value showing confidence level of the landmark recognition. Range is 0 to 1.
"boundingPoly" An object that expresses the polygon which outlines the detected landmark.
"locations" Objects that show geographical position information for the detected landmark. This information may exist for both where the landmark is, and where the picture was taken from.
Logo recognition ("logoAnnotations")

Logo recognition results are returned as an array. The detected logo data that makes up the array are objects.

"logoAnnotations": [
  {...},
  {...}
]

The format of objects showing logo detection results are explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK Logo recognition example.

{
  "description": string,
  "score": number,
  "boundingPoly": {...}
}
Name Explanation
"description" A string which expresses the detected logo.
"score" A numerical value showing confidence level of the logo recognition. Range is 0 to 1.
"boundingPoly" An object that expresses the polygon which surrounds the detected logo.
Object recognition ("labelAnnotations")

Object recognition results are returned as an array. The detected object data that makes up the array is an object.

"labelAnnotations": [
  {...},
  {...}
]

The format of the object showing object detection results is explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > Object recognition example.

{
  "mid": string,
  "description": string,
  "score": number
}
Name Explanation
"mid" The Google Knowledge Graph entity ID of the detected object.
"description" The description of the above listed entity ID.
"score" A numerical value showing confidence level of the object recognition. Range is 0 to 1.
OCR ("textAnnotations")

OCR (text recognition) results are returned as an array. The detected text data that makes up the array is an object.

"textAnnotations": [
  {...},
  {...}
]

The format of the object showing OCR results is explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > OCR (text) example.

{
  "locale": string,
  "description": string,
  "boundingPoly": {...}
}
Name Explanation
"locale" A string showing the language code of the detected text.
"description" A string expressing the detected text.
"boundingPoly" An object that expresses the polygon which surrounds the detected text.
Adult content detection ("safeSearchAnnotation")

The format of the data showing adult content detection results is explained below. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > Adult content detection example.

{
  "adult": string,
  "spoof": string,
  "medical": string,
  "violence": string
}
Name Explanation
"adult" A string that shows the likelihood of the image containing sexual content.
"spoof" A string that shows the likelihood that the image is a spoof (an altered version meant to be funny or offensive) of another image.
"medical" A string that shows the likelihood that the image is medical in nature.
"violence" A string that shows the likelihood of the image containing violent content.
Color analysis ("imagePropertiesAnnotation")

The format of color analysis results data is as shown below.

"imagePropertiesAnnotation": {
  "dominantColors": {...}
}
Name Explanation
"dominantColors" An object that shows the dominant colors detected in the image. At the current time, this is the only information contained in imagePropertiesAnnotation.

The dominantColors object contains information about the colors in the image. Usually, it contains information for several colors.

"dominantColors": {
  "colors": [
    {...},
    {...}
  ]
}
Name Explanation
"colors" An array containing information about the dominant colors detected in the image.

The format of the color information is as follows. For an example of outputted results, refer to Examples of using the Image analysis BLOCK > Color analysis example.

{
  "color": {...},
  "score": number,
  "pixelFraction": number
}
Name Explanation
"color" An object that shows the RGB values of the detected color.
"score" A numerical value that shows the level of confidence for color detection. Range is 0 to 1.
"pixelFraction" A numerical value that shows what portion of the image the detected color comprises. Range is 0 to 1.

The objects showing the RGB values of the color are formatted as follows:

{
  "red": number,
  "green": number,
  "blue": number
}
Name Explanation
"red" A numerical value that shows the red level of the detected color. Range is 0 to 255.
"green" A numerical value that shows the green level of the detected color. Range is 0 to 255.
"blue" A numerical value that shows the blue level of the detected color. Range is 0 to 255.
Bounding polygon objects ("boundingPoly"/"fdBoundingPoly")

The format for objects that express polygons is as shown below.

{
  "vertices": [{...}, ..., {...}]
}
Name Explanation
"vertices" An array comprised of objects showing each of the polygon's vertices.
Vertices object

The vertices object shows two-dimensional coordinates within the image. The data is formatted as follows:

{
  "x": number,
  "y": number
}
Name Explanation
"x" The numerical value of the vertex's x coordinate.
"y" The numerical value of the vertex's y coordinate.
Likelihood

Likelihood is shown using one of the six possible strings shown below.

Likelihood Explanation
"UNKNOWN" The likelihood is unknown.
"VERY_UNLIKELY" The likelihood is very low.
"UNLIKELY" The likelihood is low.
"POSSIBLE" There is some likelihood.
"LIKELY" The likelihood is high.
"VERY_LIKELY" The likelihood is very high.
Facial features (landmarks) object

The facial landmarks object shows the type of facial feature (left eye, right eye, etc.) and the feature's position in the image. The data is formatted as shown below.

{
  "type": string,
  "position": {...}
}
Name Explanation
"type" A string that expresses the type of facial feature.
"position" An object showing the position of the facial feature.
Types of features

Facial features are expressed as strings. The various strings and their meanings are as shown below.

Facial feature strings Explanation
"UNKNOWN_LANDMARK" An unrecognizable facial feature.
"LEFT_EYE" The left eye.
"RIGHT_EYE" The right eye.
"LEFT_OF_LEFT_EYEBROW" The leftmost point of the left eyebrow.
"RIGHT_OF_LEFT_EYEBROW" The rightmost point of the left eyebrow.
"LEFT_OF_RIGHT_EYEBROW" The leftmost point of the right eyebrow.
"RIGHT_OF_RIGHT_EYEBROW" The rightmost point of the right eyebrow.
"MIDPOINT_BETWEEN_EYES" The point directly between the eyes.
"NOSE_TIP" The tip of the nose.
"UPPER_LIP" The upper lip.
"LOWER_LIP" The lower lip.
"MOUTH_LEFT" The leftmost point of the mouth.
"MOUTH_RIGHT" The rightmost point of the mouth.
"MOUTH_CENTER" The center point of the mouth.
"NOSE_BOTTOM_RIGHT" The bottom-right point of the nose.
"NOSE_BOTTOM_LEFT" The bottom-left point of the nose.
"NOSE_BOTTOM_CENTER" The bottom-center point of the nose.
"LEFT_EYE_TOP_BOUNDARY" The topmost point of the left eye.
"LEFT_EYE_RIGHT_CORNER" The rightmost point of the left eye.
"LEFT_EYE_BOTTOM_BOUNDARY" The bottommost point of the left eye.
"LEFT_EYE_LEFT_CORNER" The leftmost point of the left eye.
"RIGHT_EYE_TOP_BOUNDARY" The topmost point of the right eye.
"RIGHT_EYE_RIGHT_CORNER" The rightmost point of the right eye.
"RIGHT_EYE_BOTTOM_BOUNDARY" The bottommost point of the right eye.
"RIGHT_EYE_LEFT_CORNER" The leftmost point of the right eye.
"LEFT_EYEBROW_UPPER_MIDPOINT" The central uppermost point of the left eyebrow.
"RIGHT_EYEBROW_UPPER_MIDPOINT" The central uppermost point of the right eyebrow.
"LEFT_EAR_TRAGION" The notch above the tragus (the protrusion that partially covers the entrance to the inner ear) of the left ear.
"RIGHT_EAR_TRAGION" The notch above the tragus (the protrusion that partially covers the entrance to the inner ear) of the right ear.
"LEFT_EYE_PUPIL" The pupil of the left eye.
"RIGHT_EYE_PUPIL" The pupil of the right eye.
"FOREHEAD_GLABELLA" The point between the eyebrows.
"CHIN_GNATHION" The bottommost point of the chin.
"CHIN_LEFT_GONION" The leftmost point of the jaw.
"CHIN_RIGHT_GONION" The rightmost point of the jaw.
Facial feature position object

Three dimensional coordinates showing the position of each facial feature. The data is formatted as follows:

{
  "x": number,
  "y": number,
  "z": number
}
Name Explanation
"x" Numerical value showing the x coordinate.
"y" Numerical value showing the y coordinate.
"z" Numerical value showing the z coordinate (depth).
Geographical position object ("locations")

An array that shows the geographical position. The position is expressed as objects for the latitude and longitude.

"locations": [
  {...},
  {...}
]

The data for the latitude and longitude objects is expressed as shown below.

{
  "latLng": {
    "latitude": number,
    "longitude": number
  }
}
Name Explanation
"latitude" A numerical value showing the latitude.
"longitude" A numerical value showing the longitude.