Getting Started

I understand the idea of IoT Core's Device Shadow, but how do you write an actual client? I thought [^1], so I wrote a Ruby client and tried it out. If you take a look at “Connect to AWS IoT from Ruby's MQTT client”, it's a quick story.

[^1]: There is an implementation using the Device SDK on the official website, but I didn't think it was easy to understand.

Device Shadow

IoT Core has an implementation called Device Shadow. Simply put, Device Shadow is JSON data linked to a physical device and stored on the cloud side.

For example, suppose a device is a vehicle or robot, and it has operation modes fast, normal, slow. IoT Core links this mode to the device and has it as a shadow. When the operator's application wants to switch the device's operation mode, it can rewrite the Shadow mode in the cloud without directly interacting with the device in the field, and IoT Core will sync nicely with the device after that.

In the IoT world where physical devices are involved, the processing of abnormal systems tends to be troublesome because the network is unstable or the device is turned off, and the application and device cannot necessarily synchronize their state in real time. That's why Shadow mediates and synchronizes that even if you don't do your best with the app.

flowchart LR
    ls --update reported--> shadow
    shadow --subscribe /delta--> delta
    shadow--get-->app
    subgraph "Device"
      delta --sync--> ls[Local state]
    end
    subgraph IoT Core
      shadow
    end
    subgraph Cloud
      app--update desired-->shadow
    end

code

It's faster to look at the code, isn't it? I'll paste it for now.

• MQTT is recommended for exchanging shadow data between devices and Device Shadow [^3]. This time I'm going to use ruby-mqtt.

• The device performs actions against shadows, which are retrieval, update, and deletion, by throwing messages to the /get, /update, and /delete topics, respectively.

• The messages exchanged in shadows are determined by the Device Shadow Service Document.

• Execution results come back to MQTT topics at @topics in the code, such as /get/accepted, /get/rejected, etc., so I'll subscribe these [^2]. Here, the loop is threaded so that it doesn't block other operations.

[^3]: Can also be done with HTTP.

[^2]: MQTT isn't basically a Request/Response type protocol, so this area is a bit troublesome. Request/Response is supported in MQTT5, but the operation is similar.

Devices

require 'mqtt'
require 'json'

class Device
  attr_reader :delta
  attr_accessor :local_state

  def initialize
    @mqtt_config = {
      host: "<your-endpoint>-ats.iot.ap-northeast-1.amazonaws.com",
      port: 8883,
      ssl: true,
      cert_file: "device.pem.crt",
      key_file: "private.pem.key",
      ca_file: "AmazonRootCA1.pem"
    }
    @thingName = "ta"
    @shadowName = "sb"

    # classic shadow
    # @shadow_topic = "$aws/things/#{@thingName}/shadow"

    # named shadow
    @shadow_topic = "$aws/things/#{@thingName}/shadow/name/#{@shadowName}" 

    @message = { state: { reported: { mode: nil } } }
    @topics = [
      @shadow_topic + "/delete/accepted",
      @shadow_topic + "/delete/rejected",
      @shadow_topic + "/get/accepted",
      @shadow_topic + "/get/rejected",
      @shadow_topic + "/update/accepted",
      @shadow_topic + "/update/rejected",
      @shadow_topic + "/update/delta",
      @shadow_topic + "/update/documents"
    ]
  end

  def connect
    @client =  MQTT::Client.connect(**@mqtt_config)
  end

  def subscribe
    Thread.new do 
      @client.get(@topics) do |topic, message|
        puts topic
        puts JSON.pretty_generate(JSON.parse(message))
        if topic.end_with?("delta")
          @delta = JSON.parse(message, symbolize_names: true)
        end
      end
    end
  end

  def get
    @client.publish(@shadow_topic + "/get")
  end

  def sync
    @local_state = @delta[:state][:mode]
  end

  def report
    @message[:state][:reported][:mode] =  @local_state 
    @client.publish(@shadow_topic + "/update", @message.to_json)
  end

  def delete
    @client.publish(@shadow_topic + "/delete")
  end

end

Apps

The application side will use the aws command.

THING_NAME=ta
SHADOW_NAME=sb
MODE=$1

aws iot-data update-thing-shadow --thing-name $THING_NAME --shadow-name $SHADOW_NAME \
    --cli-binary-format raw-in-base64-out \
    --payload "{\"state\":{\"desired\":{\"mode\":\"$MODE\"}}}" /dev/stdout \
    | jq .

aws iot-data get-thing-shadow --thing-name da --shadow-name sb /dev/stdout | jq .

Synchronization Scenarios

I think there are various scenarios, but here I'm going to try something like the following.

1. device tells Shadow the current mode reported

2. The application tells Shadow the desired mode desierd

3. Based on that difference delta, the device rewrites the current mode with a desired one and synchronizes

1. device tells Shadow the current mode in reported

The device side will use irb to do it interactively.

irb(main):006:0> require './device.rb'
irb(main):007:0> d = Device.new
irb(main):008:0> d.connect
irb(main):009:0> d.subscribe
irb(main):010:0> d.local_state="normal"
irb(main):011:0> d.report

I wonder if the state of Shadow looks like here

{
  "state": {
    "reported": {
      "mode": "normal"
    }
  }
}

2. The application tells Shadow the desired mode desierd

app sends desired

./app-desire.sh normal

Shadows are merged to look like this

{
  "state": {
    "desired": {
      "mode": "normal"
    },
    "reported": {
      "mode": "normal"
    }
  }
}

Now let's change from normal to fast

./app-desire.sh fast

Shadow calculated the difference and delta was created.

{
  "state": {
    "desired": {
      "mode": "fast"
    },
    "reported": {
      "mode": "normal"
    },
    "delta": {
      "mode": "fast"
    }
  }
}

3. Based on delta, the device rewrites the current mode with desired and synchronizes

If the network is normal, delta should have been transmitted to the device as follows through the $aws/things/ta/shadow/name/sb/update/delta topic, using the previous app message as a trigger. If the device is offline and you go pick it up again after reconnecting, the device can be retrieved with d.get.

{
  "version": 10,
  "timestamp": 1670636133,
  "state": {
    "mode": "fast"
  },
  "metadata": {
    "mode": {
      "timestamp": 1670636133
    }
  }
}

Synchronize delta with the local_state of the device.

irb(main):015:0> d.sync
=> "fast"
irb(main):016:0> d.local_state
=> "fast"

Finally, tell Shadow that they've synced.

irb(main):019:0> d.report

The Shadow side is now fast too.

{
  "state": {
    "desired": {
      "mode": "fast"
    },
    "reported": {
      "mode": "fast"
    }
  }
}

Well done!

Summary

I tried it out by simply implementing Device Shadow, which I had somehow understood, in Ruby. It's easier to understand this kind of case using irb to do it interactively. If I delete Shadow here... It is possible to test various scenarios, including abnormal systems, etc. As for Shadow, “What is a digital twin? Following that concept and implementation with “Device Shadow” of AWS IoT Core” also explains the uses of Shadow well.

I hope this article saves someone time.

Cheers, 🍺