1

empowering creative people

Encoding and Decoding Payloads on The Things Network

Sending data to the cloud is great! You’ve followed our Getting Started on The Things Network Tutorial and you have data streaming out of your Pycom Lopy4 and into your TTN account. There is just one problem, it’s all bytes! How do we get all that sweet sweet data into a usable format? We decode it! 

This tutorial is all about encoding and decoding your data. To effectively use a LoRaWAN device you need to keep the transmitted packets as small as possible. This means encoding your data into bytes and decoding it on the backend. In this tutorial, we will just scratch the surface and get you sending encoded messages from your LoRaWAN device and decoding them within The Things Network. The good news is there are a lot of available resources to get this done, and we’ve got some code to get you started with some useful examples. We’ve also compiled a great list of resources to help you take that next step farther with encoding and decoding.

Encoding with MicroPython

Since the LoRaWAN device we like to use most is the Pycom Lopy4, we are going to give encoding examples using MicroPython. If you aren’t connected to The Things Network yet, go back and follow our Getting Started on The Things Network Tutorial.

A very simple way to encode into bytes is by using the “struct” python library. It's too much to cover the whole thing here, so if you want all the details read the struct library documentation.

But wait, aren’t we programming in MicroPython? The struct library exists in MicroPython as ustruct, it is largely the same, and only has a few differences that are outlined in the ustruct documentation. ustruct is a little leaner than stuct, so most everything will work. If you run into a weird error verify that ustruct includes the operation you are performing.

What does ustruct do? It encodes your data into bytes. There are a lot of different shapes that this can take, a string takes more bytes than an integer or a float. Choose a format that is appropriate for your data that keeps it to as little bytes as possible. 

Here is an example:

packet = ustruct.pack(format, value1, value2…)

You can unpack the bytes with:

ustruct.unpack(format, packet)

Decoding within The Things Network

The data that comes into The Things Network is all in bytes, so we need to decode it into a usable value again. From the application screen navigate to “Payload Formats”.

screenshot-the-thing-network-payload-formats

From here we can see where we create the decoder. The programming language that we need to use is JavaScript, and with it, we can easily create a JSON value. The encoder can be used to encode values downlinked to your Pycom, but for this tutorial, we will focus on uplink data.

screenshot-the-things-network-payload-decoder

Encoder and Decoder Examples

Short:

2 bytes, 0-65,536. We can the values from two bytes as two individual numbers 0-255 or a single number 0-65,536 (256^2).

To encode a short in MicroPython:

import ustruct # import ustruct library

test = 270 # value to be packed
testpack = ustruct.pack('h',test) # encode value
s.send(testpack) # send it to TTN
print('Sending: ', test) # print the value sent
time.sleep(3) # take a break (you earned it!)

To decode a short in JavaScript:

function Decoder(bytes, port) {
// Decode an uplink message from a buffer
// (array) of bytes to an object of fields.
var decoded = {};
// Decode bytes to int
var testShort = (bytes[1] << 8) | bytes[0];
// Decode int
decoded.short = testShort;
return decoded;
}

Float:

4 bytes, includes a decimal point and can be positive or negative

To encode a float in MicroPython:

# encode the packet, so that it's in BYTES (TTN friendly)
# could be extended like this struct.pack('f',lipo_voltage) + struct.pack('c',"example text")
packet = ustruct.pack('f',lipo_voltage)

# send the prepared packet via LoRa
s.send(packet)

# example of unpacking a payload - unpack returns a sequence of
#immutable objects (a list) and in this case the first object is the only object
print ("Unpacked value is:", ustruct.unpack('f',packet)[0])

To Decode a Float in JavaScript:

function Decoder(bytes, port) {
// Based on https://stackoverflow.com/a/37471538 by Ilya Bursov
function bytesToFloat(bytes) {
// JavaScript bitwise operators yield a 32 bits integer, not a float.
// Assume LSB (least significant byte first).
var bits = bytes[3]<<24 | bytes[2]<<16 | bytes[1]<<8 | bytes[0];
var sign = (bits>>>31 === 0) ? 1.0 : -1.0;
var e = bits>>>23 & 0xff;
var m = (e === 0) ? (bits & 0x7fffff)<<1 : (bits & 0x7fffff) | 0x800000;
var f = sign * m * Math.pow(2, e - 150);
return f;
}
// Test with 0082d241 for 26.3134765625
return {
// Take bytes 0 to 4 (not including), and convert to float:
temp: bytesToFloat(bytes.slice(0, 4))
};
}

String:

To encode a string in Micropython:

#strings are converted to bytes automatically (extremely inefficient)
s.send("Hello World!")

To decode a string in JavaScript (not recommended as it eats up bandwidth!):

function Decoder(bytes, port) {
// Decode plain text; for testing only
return {
receivedString: String.fromCharCode.apply(null, bytes)
};
}

Multiple Bytes:

To encode multiple bytes in a single packet in MicroPython:

s.send(bytes([1, 2]))

To decode multiple bytes in a single packet in JavaScript:

function Decoder(bytes, port) {
// Decode an uplink message from a buffer
// Decode bytes to int
var firstValue = (bytes[1] << 8) | bytes[0];
var secondValue = bytes[3]<

Converter and Validator and Encoder

The data that you have received and decoded might need a little logic to spice it up. That’s where the Converter and Validator come in. These are all programmed in javascript as well. 

  • The Converter is a JavaScript function that takes the object produced by the Decoder and returns an object containing the converted values.
  • The Validator is a JavaScript function that can be set to check the value returned by the Converter and returns a Boolean value indicating the validity of the data.
  • The Encoder is used for downlinks to encode data into bytes using JavaScript before it is sent to your device.

Further Resources

If you want to learn more about encoding and decoding your packets and getting the most out of your bytes. Here are some additional resources that might be helpful:

Struct Python Documentation

ustruct MicroPython Documentation

How to Customise Payload Functions on TTN Video Tutorial - this is very handy

Working with Bytes- TTN Tutorial on using Bytes

The Things Uno Workshop – Workshop on getting started with communicating using TTN and Decoding Payloads

Best Practices when Sending GPS Location Data

How to Send Temperature, Humidity, and Battery Level, using the port number to determine what type of packet is being decoded.

How to use the lora-serialization library

Decrypting Messages for Dummies – More on using bytes, sign extension and port numbers

Help a Poor Guy with this Payload – Decode 7 bytes into two 28 bit numbers

Simple Payload Conversion – Get two integers from a null-terminated string

Decode a Float Sent by Lopy as Node – How to Convert as True IEEE-754 Floating Point

How can I Correctly Cast My Value in Bytes Before Sending – Working with Struct, and decoding it in TTN

Nemeus NIS-UL Ultrasonic Sensor - Working with a variable number and variable type of measurements by using bit masks

Armed with these examples there is nothing standing between you and a working device using The Things Network! If this is one step too far along for you we have a Getting Started on The Things Network Tutorial that walks you through getting connected to TTN using LoRaWAN and the Pycom Lopy4. If you’ve decoded your packets and you are ready to send them to a dashboard, check out our The Things Network Tutorials section for more tutorials about how to utilize your data! If you're ready to send your data out to the great beyond, (starting with Adafruit.io) check out our TTN, IFTTT, and BEYOND Tutorial!

Sending data to the cloud is great! You’ve followed our Getting Started on The Things Network Tutorial and you have dat...

Have a question? Ask the Author of this guide today!