Update the Setting Category for Data Ingestion

Last modified: Apr 3, 2019 @ 14:59

The MQTT Publish Topic Filters Setting Category for the Gateway is used to store the mapping of the Topic format allowed, and the corresponding payload expression. The Settings Category needs to be modified by performing the following procedure. This modification may include the addition, deletion or change of existing settings.

  1. In the DDM Dashboard, navigate to Gateway and click the Edit button of the Setting Category MQTT Publish Topic Filters.

    Example of updating the Setting Category named MQTT Publish Topic Filters.
  2. Click the Add button to enter the new settings. Each line in the list represents a topic. In the Name field, enter the topic name. In the Data Type field, select Text. In the Value field, enter the payload expression. An example is provided below.

    Example of MQTT Publish Topic Filters.
  3. Click Save.

The configuration of topic filters is further discussed in the following sections.

A topic filter consists of two parts:

  • Topic name (Name part of the Settings). The Device sends data to this topic.
  • Data model definition (Value part of the Settings). This specifies in which format the data is sent by the Device.

Topic names

The first part of the topic filter configuration is the topic name. Topic names used by the MQTT client in the Publish message must conform to the MQTT v3.1.1 specification. The DDM MQTT solution supports only exact topic matches and is case sensitive. It does not allow topic wildcards or topics beginning with $. However, the DDM service provides a special filter ${endpoint} that can be used within the topic to indicate the client identifier. Some examples are provided below:

  • /sampleproject/log: This string allows all MQTT clients to publish to the specified topic. The client identifier is not checked.
  • /sampleproject/mydeviceid/log: In this example, it is assumed that MQTT clients publish to a topic of the form /sampleproject/<client-identifier>/log. The client identifier used by device in the CONNECT message (in this example) is mydevice. In this case, the string only allows an MQTT client with client identifier mydeviceid to publish to the topic specified. The same MQTT client cannot publish to topic /sampleproject/mydeviceid/action, since the filter /sampleproject/mydeviceid/action is not added. Similarly, another MQTT client with client identifier anotherdeviceid cannot publish to that topic.
  • /sampleproject/${endpoint}/log: In this example, it is assumed that MQTT clients publish to a topic of the form /sampleproject/<client-identifier>/log. In some cases, it may be necessary to allow clients with different MQTT client identifiers to publish on the same topic. ${endpoint} is a special filter character provided to enable this. In this example, an MQTT client with client ID device1 is allowed to publish to topic /sampleproject/device1/log whereas an MQTT client with client ID device2 is allowed to publish to topic /sampleproject/device2/log.

The above-mentioned topics can be categorized as discussed below, in order to handle the telemetry payload (JSON/String).

Topic for single resource

In this case, a whole payload is sent as a single string to a single resource.

Sample Topic with single resource.

Topic for multiple resources

In this case, the JSON payload is handled and data is sent to multiple resources.

Sample Topic with multiple resources.

Notes:

MQTT topic naming rules are followed, as defined in the specification. For example, sampleproject and /sampleproject are two different topics.

DDM does not allow wildcards in topic filters. Only exact string matches are allowed. The special filter ${endpoint} is provided.

Data model definition

The second part of the topic filter is the data model definition. This definition allows DDM to accept the payload. The payload must be modelled as an IPSO resource. It is possible to model the payload as a single resource, or parse the payload into multiple IPSO resources.

Two examples are provided below. In these examples, an MQTT client sends a Publish message with payload { "value" : 100.0, "units": "Celsius" }.

Data modeled as single resource

In this example, the message is modeled as a Smart Object named LogMessage with ID 10001. This Smart Object contains a Resource log message with ID 10000. When the MQTT Gateway receives the payload, it does not parse the payload and simply transfers the JSON string to DDM.

Sample with single resource.

Latest measurement sample with single resource.

Data modeled as multiple resources

In some cases, it may be required to parse the payload from the MQTT clients. In this example, we assume that the payload { "value" : 100.0, "units": "Celsius" } from the MQTT client maps to two IPSO resources.

  • value maps to the temperature smart object (ID: 3303) sensor value (ID: 5700),
  • units maps to the temperature smart object (ID: 3303) sensor value (ID: 5701).

Hence the topic filter is configured with JSON as follows: {"3303/0/5700" : $.value , "3303/0/5701" : $.units}.

Sample with multiple resources.

Note: The DDM MQTT Service allows MQTT clients to send MQTT Publish messages with QoS 0 and QoS 1. In case the topic filters do not match the publish messages, the messages are silently discarded.

Limitations

The data modeling solution only supports String and JSON format. Only nested JSON objects are allowed but the Array objects are not supported. The system does not put a hard limit on the deep level of nesting.

For examples of nested JSON objects and JSON Arrays, see https://www.w3schools.com/js/js_json_objects.asp and https://www.w3schools.com/js/js_json_arrays.asp, respectively.

The payload data for DDM resources having String, Boolean, Integer, and Float are the only datatypes which are supported for uplink and downlink commands.

Note: Opaque and Timestamp data types are not supported for uplink and downlink commands.

The topic name should contain only one client identifier.

For example:

If there are two client identifiers with names mydevice1 and mydevice2, then the topic name /sampleproject/mydevice1/mydevice2/log is not valid. For mydevice1 use the topic name /sampleproject/mydevice1/log. For mydevice2 use the topic name /sampleproject/mydevice2/log.

This constraint has been added to avoid one device writing on another device’s topic. Having more than one client identifier name in the topic name can lead to the failure of authorization logic.