|
Home
| pfodApps/pfodDevices
| WebStringTemplates
| Java/J2EE
| Unix
| Torches
| Superannuation
|
| About
Us
|
|
|
Reliable Remote Control via SMS
|
by Matthew Ford 20th March 2018 (original
19th Sept 2014)
© Forward Computing and Control
Pty. Ltd. NSW Australia
All rights reserved.
This page describes the design of a reliable message delivery system using SMS and pfod (Protocol for Operation Discovery).
“Short Message Service (SMS) is a text messaging service component of phone, Web, or mobile communication systems. It uses standardized communications protocols to allow fixed line or mobile phone devices to exchange short text messages.” ...
“Message delivery is “best effort,” so there are no guarantees that a message will actually be delivered to its recipient, but delay or complete loss of a message is uncommon, typically affecting less than 5 percent of messages” …
“SMS is a stateless communication protocol in which every SMS message is considered entirely independent of other messages. Enterprise applications using SMS as a data bearer require that session management be maintained external to the protocol.” (http://en.wikipedia.org/wiki/Short_Message_Service)
In order to implement a reliable communication protocol of connected messages such as is needed for a remote control system, a session management layer is required on top of the basic SMS. This page describes that layer for use with pfod (Protocol for Operation Discovery) as implemented by the Android pfodApp
SMS delivery is not reliable. Messages may arrive out-of-order or not at all or be repeated.
SMS messages are small, i.e. less then 160 chars
Only GSM encode SMS messages are universally supported by hardware/software
A communication connection based on SMS for pfod needs to have the following:
a) Detect and re-request missing packets or part there of. Every message sent from the pfodApp must be responded to by the pfodDevice or the connection is considered “lost”
b) Pass UTF-8 data
c) Send data packets up to 1024 bytes from pfodDevice to pfodApp and send data packets up to 265 bytes from pfodApp to pfodDevice
d) The pfodDevice can send “raw data” at any time.
e) 128bit Security to prevent unauthorized access.
f) Limit pfodDevice connections to one at a time. i.e. only one pfodApp can communicate with the pfodDevice at a time.
All pfod messages are initiated from the pfodApp. The pfodDevice must respond to each message and should (must) not send unsolicited pfod messages. Although the pfodDevice can send “raw data” at any time.
To allow for re-requests and detection of missing and repeated messages, each message the pfodApp sends will contain a connection number and message sequence number, starting from 0 for a new connection. The connection number is stored in the pfodApp database and incremented each connection, starting from 0. When the pfodDevice replies it includes the pfodApp connection number and the pfodDevice's own seqNo starting from 1 for each new connection.
If the pfodApp does not receive a complete reply from the pfodDevice after the user configured timeout, the pfodApp will resend the command (using the same ConnectionNo and MsgNo) upto five (5) times, before reporting the connection 'lost'.
Connection numbers are 0 to 4095 and message numbers are 0 to 4095. i.e. 12 bits each, 2 6bit bytes per number. If a person started a new connection from the pfodApp every 15mins (after 5 retries of 3mins) it would take about 6 weeks to cycle back to connection no 1.
Once the msgNos either from the pfodApp or pfodDevice approach 4095, then the pfodApp automatically increments the connection number and starts a new connection.
If the pfodApp re-sends a message, due for example to slow delivery or message loss, the pfodDevice can determine from the connection number and message sequence no if it has already received and processed that message. If it has already processed that message then the pfodDevice just resends the previous response (including the previous pfodDevice message sequence number) and does not process the command again,
As the pfodApp receives reply SMS messages, the pfodApp re-orders them via their msgNo so that it the pfodApp needs to resend the command, only the missing messages need to be received to complete the response. Re-requests by the pfodApp can result in the pfodApp receiving rawData messages mixed in with parts of a command response. The parser in the pfodApp is not designed to handle raw data mixed in like this, so the pfodApp needs to accumulates the response SMS messages until it has the complete response before passing the response on to be parsed and displayed to the user.
The pfod specification requires that the transmission system can pass UTF-8 data. The standardized 'text' SMS of 160 bytes only handles 7bit data. There are a number of solutions to this. One is to use the 16-bit UCS-2 SMS message format but that is inefficient for 7 bit ASCII as it limits all SMS messages to maximum of 70 16bit chars each. There is a also an 8-bit format, but that is not available via the standard Android API. Also GRPS hardware always supports GSM format but not necessarily any other format.
So the alternative used here is to take the UTF-8 data and convert it to a Base64 encoding which maps 6 bits at a time into the GMS default character set.
SMS encoding
The actual encoding used is
if (sixBitSlice == 0) {
encodedChar = ' ';
} else if (sixBitSlice == 1) {
encodedChar = '+';
} else if (sixBitSlice < (12)) {
encodedChar = (sixBitSlice-2) + '0'; // 0..9
} else if (sixBitSlice < (12+26)) {
encodedChar = (sixBitSlice-12) + 'A';
} else {
encodedChar = (sixBitSlice-12-26) + 'a';
}
This uses ' ', '+' and 0 to 9 A to Z, a to z,
The reverse decoding is
if ((encodeChar >= 'a')&& (encodedChar <= 'z')) {
sixBixSlice = encodedChar – 'a' + 38;
} else if ((encodedChar >= 'A') && encodedChar <= 'Z')) {
sixBitSlice = encodedChar – 'A' + 12;
} else if ((encodedChar >= '0') && encodedChar <= '9')) {
sixBitSlice = encodedChar – '0' + 2;
} else if (encodedChar == '+') {
sixBitSlice = 1;
} else if (encodedChar == ' ') {
sixBitSlice = 0;
} else {
invalid return 0xff
}
sixBitSlice = 0x3F & sixBitSlice; // clear upper bitsUsing the above 8bit to 6bit encoding limits the maximum number of 8bit bytes per message to 117 UTF-8 bytes per-message allowing for the first 4 6 bit bytes to be used for message number and connection number. (The message number is first) Messages are padded with nulls as necessary to make up the final sixBitSlice. A new connection command from the pfodApp will have message number 0 and so will always start with two spaces.
Since SMS message is limited to 117 8bit Base64 encoded bytes per message, this requirement necessitates multiple SMS messages be sent as a group. The message number is used to keep track of the message order. When the pfodApp sends a command to the pfodDevice it needs at most 3 SMS messages (max 256 bytes). Almost all pfodApp commands will fit in a single SMS message. Only very long multi-selection screens and long text input screens will require more then one SMS message.
Because { is a valid character within a command sent by the pfodApp, the occurrence of a { in the SMS message received by the pfodDevice can not be used to indicate the start of a command. To avoid the pfodDevice having to accumulate and re-order all incoming messages and parse them to find start of commands, the start of each new command sent by the pfodApp, always has a msgNo that satisfies message number mod 3 == 0. This allows the pfodDevice to identify the start of a command just from the msgNo and either ignore it if it is 'old', or resend the previous response if it is the start of the previous already processed command, or pass it to the pfodParser if it is the start of a new command. Also using this approach any 'old' msgNos (msgNo less than next expected msgNo) where the message number mod 3 != 0 are immediately identifiable as the extra parts of an old command and can be completely ignored
The pfodDevice needs at most 9 SMS messages to send back its response (max 1024 bytes + 8 bytes hash). The size of the menus and the their text determines how may SMS messages will be required to send them back to the pfodApp. Short simple menus will easily fit into one SMS message.
The pfodDevice just assigns the next msgNo to each new SMS. If the pfodApp resends a command and the pfodDevice has already processed it, then the pfodDevice just sends back the previous response as multiple SMS messages using the previous message numbers. Note: this implies the pfodDevice library needs to keep a copy of the previous response (up to 1024 bytes), so Uno style micros with only 2048 byte of RAM are not usable with this implementation.
Each command sent by pfodApp and each response sent by the pfodDevice, via SMS, is terminated with one or more nulls, '\0', before it is encoded. The pfodApp uses these nulls to determine the end of response. The pfodDevice just stops processing the SMS message at the first null. If a response from the pfodDevice exactly ends on an SMS (117 byte) boundary, the last 3 bytes are sent, with terminating nulls, in a following SMS message. NOTE: The pfodApp code expects the first character in the first message of a response to be {. That is no leading white space. Any leading white space should have been sent in a raw data message.
The pfodDevice can send 'raw data' at any time. Raw data messages are be indicated by having zero connection and msgNos. Although the pfodDevice will send replies in consecutive order, due to lost msgs and re-requests the SMS messages being received at the pfodApp may intermix raw data messages with parts of command responses. The pfodApp parser is not designed to handle intermixed raw data and part of command responses. For this reason the pfodApp has to accumulate a complete response up to the terminating nulls before passing it on for parsing and processing.
Lost raw data messages are not recoverable. In this design raw data will not be re-requested if the SMS messages is lost. Out of sequence raw data messages will not normally be a problem if the data is being plotted as the plot will sort the data. If you want to reliably transfer some data then send it as text in a menu or other pfod message.
The maximum number of bytes of rawData that can be sent as a single unit is 116 (plus terminating null). Typically you send one line of plot data, in CSV format, as a single unit. While it is possible to send longer lines of data by repeating the x value and then re-assembling them in some external application, pfodApp does not support this in its plotting function.
To simplify the user interface to the pfodDevice pfodSMS library, the library automatically determines if the user is sending a reply to a command response or sending raw data (i.e. any text outside { } ). To do this the library incorporates a parser in the outgoing write path to determine when a command response starts and stops. All other bytes written to the outgoing path are send as raw data. Now as it happens the pfodSecurity parser already has this same outgoing parser for adding the 128bit security hash to command responses. So by using that code, the pfodSMS library, gets the parser it want and also adds 128bit security.
Using a security key is optional and will require two additional SMS messages (each way) to complete the challenge and response, before the main menu can be requested and returned.
If the pfodDevice has 128bit security key defined, then the pfodApp must send {_} with msgNo 0 as the first command of a new connection. If the pfodDevice does not already have an active, not timed-out, connection from some other phone number, this will start the challenge response to open a connection with 128bit security. If there is already an existing, not timed-out, connection from another pfodApp, the pfodDevice just ignores other connections requests.
The security hash code is appended after each command and response. i.e. after the closing }. When the pfodApp sends a command to the pfodDevice, the pfodApp ensures that the terminating } and the hash code are always sent in the same SMS message, so the pfodDevice only needs to check for the } to determine the last SMS is the end of the command and so increment the next expected message number so that next expected message number mod 3 == 0. The pfodApp uses the terminating null(s) to determine the end of a response from the pfodDevice (including any hashcode)
A pfodDevice only communicates with one pfodApp at a time. So only one phone number and one connectionNo define the current connection. When the pfodApp exits it sends the 'close connection' command, {!}. The pfodApp does not expect a response to this command.
If the pfodDevice receives 'close connection', the pfodDevice will mark the current connection as closed and will then accept a new connection from another (or the same) phone number. Since the delivery of this 'close connection' command is not guaranteed, the pfodDevice will also accept a new connection from a different phone number after 10 minutes of in-activity (I.e no new commands from the pfodApp for 10 minutes)
The common case is multiple connections from just one phone number, so the pfodDevice will always accept a new connection (with a larger connection number and message number == 0) from the current phoneNo even if the last connection has not timed out.
In-order to be able to ignore old messages, the pfodDevice keeps track connection numbers of up to 10 phone numbers and will ignore new connection requests that do not have a larger connection no.
The pfodDevice processing matrix for handling phone numbers, connection numbers and message numbers is shown below.
Starting
a new connection closes any current connection. The parser is
re-initialized and its input buffer cleared to remove any partial
commands from the old connection.
The connection numbers issued by pfodApp roll over back to 1 once they are incremented pass 4095, so special processing is used by the pfodDevice to determine if the SMS connection is greater or less than the previous connection number mod 4095
The processing used by
the pfodDevice is:-
if the SMS connection number is not equal to
the previous connection number for the same phone number, the
following algorithm used to compare them:-
If the SMS connection
number less than previous connection number
then the Adjusted SMS
connection number == SMS connection number + 4095
else the
Adjusted SMS connection number == SMS connection number
Then if 0 < (Adjusted SMS connection number - previous connection number) <= 2048, then this SMS connection number is taken as being greater then the previous connection number, otherwise it is taken as being less then the previous connection number.
All messages from connections with 'smaller' connections numbers are ignored as being 'old'. This processing ensures that any messages from the previous 2047 connections will be ignored.
Examples of Connection Number
comparisons:
If
previous connection number is 4094 and SMS message connection number
is 1 (has rolled over) then Adjusted SMS connection number is 4096
and difference is 2 (less than or equal to 2048) so this is taken as
a larger connection number
If previous connection number is 1 (has
rolled over) and SMS connection number is 4094 then Adjusted SMS
connection number is 4094 and difference is negative, ignore this
message.
If previous connection number is 2049 and the SMS
connection number is 1 then Adjusted SMS connection number is 4096
and difference is 2047 (less than or equal to 2048) so this is taken
as a larger connection number.
If previous connection number is
2047 and the SMS connection number is 1 then Adjusted SMS connection
number is 4096 and difference is 2049 (greater then 2048) so this
message is ignored.
If previous connection number is 1 and the SMS
connection number is 3000 then Adjusted SMS connection number is 3000
and difference is 2999 (greater then 2048) so this message is
ignored.
If previous connection number is 3000 and the SMS
connection number is 1 then Adjusted SMS connection number is 4096
and difference is 1096 (less than or equal to 2048) so this is taken
as a larger connection number.
There is one special case. When the SMS connection is first defined in the pfodApp, the connection number is zero. When the pfodDevice receives a message with connection number zero and message number zero, it clears out it memory of that phone number's connection number history. This let the user move their sim card to a new phone, with a clean install of pfodApp and still be able to connect to the pfodDevice.
Connection number and message number processing in the pfodApp is simpler.
RawData messages from the pfodDevice always have message number zero (and connection number zero). Any SMS with message number zero received from a phone number the pfodApp is connected to is always processed as raw data.
When the pfodApp sends a command it assigns the connection number which the pfodDevices uses for its responses. So the pfodApp just ignores any SMS from the pfodDevice that does not match the connection phone number and connection number for the command just sent.
The pfodApp also keeps track of the expected pfodDevice message number. For each new connection, the pfodDevice resets its outgoing message number to 1. So after the pfodApp opens a new connection (by send sending a command with a larger connection number and a message number of zero), the pfodApp expects the first SMS of the response from the pfodDevice to have message number 1 (and the same connection number). Subsequent SMS messages from the pfodDevice have message numbers incrementing by 1 each time. The pfodApp ignores any message number it has already seen and accumulates upto 9 higher messages while it is building up a complete response for parsing. A non raw data SMS terminated with null(s) indicates the end of the response and higher message numbers are ignored. If one or more lower message numbers are missing, the pfodApp (after the timeout) resends the command to force the pfodDevice to resend the response to fill in the missing messages.
The incoming SMS messages are, by default, stored on the sim card. The card has limited capacity for storing messages so as each message is read, it needs to be remove (deleted) from the sim.
There are various formats for representing phone numbers used in SMS. These differ in the representation of the country and area codes (may be missing or normalized). To allow for these differences, the design only checks the last 6 digits when it wants to determine if the two phone numbers are equal.
To prevent unintended commands being sent, the pfodApp will have an option to confirm each menu button press. That is after you press a menu button a pop-up will request confirmation to send the command.
It is important to cleanly close the connection, if possible, by sending the CloseConnection command,{!}, from the pfodApp as this stops the pfodDevice from sending more rawData SMS messages. The pfodApp sends this CloseConnection command to the currently open connection each time the app is exited.
In order to ensure that the SMS pfodDevice will
accept this command, it is always sent with a new connection number
and a message number of 1.
Note: Because of the overlap of send
and received SMS messages, the pfodApp may still receive one or more
rawData SMS messages after it exits.
Also the delivery of this
SMS is not guaranteed, so if rawData SMS messages continue to arrive
you will need to re-connect and exit again.
Finally if pfodApp connects to a pfodDevice, and after the idle time out, another pfodApp connects this new pfodApp will take over the connection. Then after this new pfodApp connection times out, if the previous pfodApp is closed, its CloseConnection command will close the new connection and allow the previous pfodApp to re-connect.
Alerts are a special type of RawData message that consists of six (6) spaces followed by a clear text Ascii message (using GSM 03.38 char set) with a maximum of 154 chars. The pfodApp will decode this message as having a zero message number and zero connection number and zero raw data terminated by nulls. The pfodApp will completely ignore this message but your SMS messaging system will display it. These alerts are designed to let the user know that something has changed at the pfodDevice and that they should refresh their screen. The current library does not implement alerts yet.
The design above is being implemented in pfodApp and in the pfodSMS library for use with GPRS equipped micro-processors. This design allows reliable remote control using the pfod protocol via unreliable SMS messages with the option of 128 security to guard against unauthorized access.
AndroidTM is a trademark of Google Inc. For use of the Arduino name see http://arduino.cc/en/Main/FAQ
The General Purpose Android/Arduino Control App.
pfodDevice™ and pfodApp™ are trade marks of Forward Computing and Control Pty. Ltd.
Contact Forward Computing and Control by
©Copyright 1996-2020 Forward Computing and Control Pty. Ltd.
ACN 003 669 994