|
Home
| pfodApps/pfodDevices
| WebStringTemplates
| Java/J2EE
| Unix
| Torches
| Superannuation
|
| About
Us
|
|
|
Remote High Speed Data Logging
|
by Matthew Ford 10th June 2018 (original
10th December 2016)
© Forward Computing and
Control Pty. Ltd. NSW Australia
All rights reserved.
This page shows you how to remotely log and plot data at up to ~2k (2000) samples/sec using a local network formed from an Adafruit WICED, a GL AR150 mini 5V router and an Android mobile running pfodApp.(V2.0.224+). Since all these devices run from 5V, this set up is suitable for field and lab work.
First let's define what we mean by High Speed Data Logging. For this tutorial, it means a sustained data rate of up to ~40K bytes/sec. That is, being able to sample at ~2000 samples/sec and send the time stamped measurement to a remote Android running pfodApp for logging and plotting with each measurement being approximately 20 bytes (depending on the number of bytes in the time stamp and sample data). While higher burst sampling rates are possible by relying on the internal and network buffers, this page only considers sustained real time sampling rates.
First we will discuss the bottlenecks that restrict the data rate and presents a series of simple test programs to help you quantify them. When using pfodApp to log and plot the data, you can turn on the debug logging to record the rate at which the app is processing the data. That rate varies between mobiles and Android versions and also varies depending of what other apps are running.
The tests here show that using an Adafruit WICED Feather sending data via a GL AR150 mini router to a ASUS zenfone5,running pfodApp (V2.0.224+), you can log an plot at ~40Kb/sec. On a Nexus P6 you can log and plot at ~48K bytes/sec. If you have a laptop handy you can use TeraTerm or CoolTerm to log the data directly to it and so avoiding using pfodApp. However pfodApp/pfodDesignerV2 makes it easy to control other functions on your Arduino as well as collecting and plotting the data.
There are four (4) possible bottlenecks in logging/plotting high speed data :-
Speed of sampling the data in the Arduino
Speed of transferring the data to the communication shield or sub-system
Speed of the communication channel
Speed of the receiving Android device.
Looking at these in turn
The sampling rate in the Arduino needs to be faster then the desired samples/sec because the Arduino needs to do other work, like scaling and handling the communications, as well as doing the sampling. Sometimes the measurement hardware has sever sample/sec limitations or the protocol (SPI, Serial, I2C, etc) used to connect the measurement hardware to the Ardunio limits the maximum samples/sec.
This example sketch, SamplingSpeedTest_Serial.ino, lets you test the maximum samples / sec for your measurement hardware. It uses analogRead() but you should replace the line of code with what ever code you need to get the sample. You should also add any extra code you need to scale/normalize the reading and prepare the measurement time stamp. (If you find the scaling calculations are taking too long, try removing all floating point calculations and send the data as integers.)
Note about Adafruit WICED Com ports:- When you connect Adafruit's WICED you will see two com ports in the Arduino IDE. You need to select the 'Serial' one for programming and the Serial Monitor. On Windows you can check in the Device Manager to see which is which.
The SampleSpeedTest_Serial.ino is initially set to work with Arduino boards and to send the data to a NullPrint. Here are some results for typical boards.
Aduino Mega/Uno
Using NullPrint 1000 loops
in:839mS-> Sample rate:1.19k samples/sec
Example data
record:
15041440,0.14
Arduino 101
Using NullPrint 1000 loops
in:83mS-> Sample rate:12.05k samples/sec
Example data
record:
10534130,0.09
To test the Adafruit WICED you need to uncomment the
#define
USE_WICED
line at the top of the
SampleSpeedTest_Serial.ino sketch. WICED has non-standard pin names
and pinModes (and some non-standard function names also). For the
WICED board the results are higher. WICED also has an ADC higher
speed burst mode using an internal buffer, but that is not used here.
Adafruit WICED
Using
NullPrint 1000 loops in:71mS -> Sample rate:14.08k
samples/sec
Example data record:
10951026,0.31
If your board does not include an integrated communication system you need to either send the data to the Serial port (USB) or transfer the data to a Wifi/Ethernet/Bluetooth shield to send it remotely. The speed of to transfer to the communication shield or sub-system is the next possible bottleneck.
Using Serial, each byte of data is transmitted as 10 bits, 1 start, 8 data and 1 stop bit. So a 115200 baud Serial connection can transfer about 115200/10 = 11520 bytes/sec. If there are 14 bytes in each data record (as there are in these tests), that implies to transmit one data record at 115200baud takes ~1.2mS or ~1215mS for 1000 records. (or 14580mS at 9600baud)
To test the USB serial transfer
rate, in SampleSpeedTest_Serial.ino,
set
boolean
useNullPrint = false;
and
set the baudrate
const
unsigned long baudRate = 115200;
Here is the result for Arduino Mega and Uno at 9600baud
Arduino Mega/Uno
Using Serial at 9600 1000
loops in:15600mS -> Sample rate:0.06kHz
Example data
record:
88322384,0.31
and at 115200baud
Arduino Mega/Uno
Using Serial at 115200
1000 loops in:1275mS -> Sample rate:0.77kHz
Example data
record:
17686572,0.26
Here is the result for the Arduino 101 at 115200baud
Arduino 101
Using Serial at 115200 1000
loops in:1142mS -> Sample rate:0.88kHz
Example data
record:
16888890,0.09
To test the Adafruit WICED you
again need to uncomment the
#define
USE_WICED
line at the top
of the SampleSpeedTest_Serial.ino
sketch.
The Adafruit WICED has a high speed USB serial connection which ignores the baud rate setting and just sends as fast as it can, ~60K bytes/sec
Adafruit WICED
Using Serial at 115200 1000
loops in:235mS -> Sample rate:4.26k samples/sec
Example data
record:
11908689,0.29
So if you can connect directly to your computer via
USB then WICED can log data at about 60K bytes/sec directly.
If
you instead send the WICED data to a standard serial port, e.g.
Serial1, then the results are the same as the Arduino boards.
The sketch, SamplingSpeedTest_Ethernet.ino, tests the Ethernet Shield sample rate, connection to TeraTerm running on computer connected to the same network
The Arduino Ethernet Shield V1 SPI
connection/shield is very slow.
Using Ethernet Shield V1 1000
loops in:2877mS -> Sample rate:0.35kHz
Example data
record:
56096860,0.27
It is not clear if this slowness is due to a) slow SPI speed or b) slow TCP/IP handling on the Ethernet shield. Adding buffering (see below) did not significantly improve the result. This shield has been replaced by a newer Arduino Ethernet Shield V3 but that shield has not been tested.
Some boards provide direct transfer of the data to an on-board communication module. Examples of these boards are Adafruit WICED, ESP8266, ESP32, Arduino 101, etc. This on-board transfer is difficult to measure but is usually not the limiting factor.
The next bottleneck is the remote transfer of the data from the Arduino communication module to the remote Android mobile or computer. For Android the following options are available, slowest to fastest, SMS, Bluetooth Low Energy, Bluetooth (Classic), WiFi/Ethernet (TCP/IP). For computers generally only Bluetooth (Classic) and WiFi/Ethernet (TCP/IP) are available.
Since we are considering high speed data logging, only TCP/IP will be considered. As noted above, the Arduino Ethernet V1 shields are slow and so Ethernet will not be considered further here. (Arduino Ethernet Shield V3 has not been tested). This leaves WiFi as the communication channel for high speed data logging.
Not all WiFi (TCP/IP) implementations have the same speed. In particular the WiFi built into the ESP8266 can only send one packet at a time and waits for that packet to be ACKed before sending the next one. This severely limits its through put. The ESP32 may be better but the Arduino support is currently (as of 4th December 2015) incomplete. The Adafruit WICED board has a better TCP/IP implementation but even with this board, the default code will send lots of small packets. The sketch, WICED_simpleWifiSpeed.ino, finds the speed using the default code with small packets.
Using (unbuffered) Adafruit WICED 1000 loops
in:5031mS -> Sample rate:0.20k samples/sec
Example data
record:
60091255,0.85
Lots a small TCP/IP packets severely limits the through put. Adding buffering to the output, so that the data is held until there is a (almost) full TCP/IP packet and then sending it, gives a much faster through put. The buffered files are in this zip file.
Using buffered Adafruit WICED 1000000 loops
in:131509mS -> Sample rate:7604.04 samples/sec
Time left to
completion: 0.00 sec
Example data record:
1301195611,1.23
The buffering class used here is pfodBufferedClient and is it is zipped up with the revised sketch, WICED_bufferedWifiSpeed.ino, in this zip file. Adafruit WICED has a client.usePacketBuffering(true) method, that also does buffering. Unfortunately it will wait for ever to send a partial buffer if you don't call flush(). The pfodBufferedClient class has a short timeout which sends the packet if nothing has been added in the last 10mS. This timeout is checked on each call to read() available(), peek() or stop() and so is more convenient to use. Adding this buffering to the Arduino Ethernet Shield V1 did not significantly increase its through put.
So to summarize so far, of the various boards I have to hand, the Adafruit WICED is the fastest. Over a 20min test it sent about about ~120K bytes/sec to TeraTerm running on my computer. If you have a computer handy on the WiFi network then you can use TeraTerm or CoolTerm to capture the data to a file. On the other hand if you are using pfodApp to control the board and capture and chart the samples, the data rate also depends on the processing speed of your Android device and its TCP/IP buffers.
The final bottleneck in high speed data logging is the receiving device, be it a computer or a mobile. The previous section tested the speed connecting to a computer. For testing connecting to an Android mobile, pfodApp will be used. For 'remote' data logging a small portable wifi router is used, GL AR150.
To test the Android mobile's speed, the free pfodDesignerV2 app was used to design a simple menu with just one button which opens a single plot of the sampled analog data. Adafruit WICED and pfodDesignerV2 covers designing a menu for WICED and Adding a Chart and Logging Data on How to Display/Plot Arduino Data on Android has details on setting up a chart with one plot and logging data.
The resulting sketch generated by pfodDesignerV2 for the WICED target is WICED_chart.ino and produces the following menu and chart. Data samples are only sent at 1/sec.
The WICED_chart.ino sketch was then modified to add buffering and to send the data at a much faster rate. The files use for the following tests are in pfodWICED_highSpeedChart.zip Unzip those to your Arduino directory and compile and load the WICED board with the pfodWICED_highSpeedChart sketch.
Install pfodApp and set up a connection to your WICED board as described in pfodAppForAndroidGettingStarted.pdf. To see in more detail what is happening, in pfodApp open Add/Edit for the WICED connection and tick the Log Debug data tick box. This will log to the WICED_debug.txt file under /pfodAppDebugData directory on your mobile. By default, Log Plot Data is ticked and pfodApp saves any data received to a file on your Android device under the /pfodAppRawData directory.
After you exit pfodApp (which flushes the last buffer), transfer the /pfodAppDebugData/WICED_debug.txt file to you computer (using say the wifi file transfer app) and open it in a text editor. You will find lines like
< 1481329581822 4.962sec dataRate:42Kbytes/sec {
}
…..
137033.63,1145.00,,
> 1481329581860 0.038sec
dataRate:42Kbytes/sec {}
137034.66,1156.00,,
In this instance pfodApp was running on an ASUS
zenfone5 mobile, running Android 4.4.2.
The dataRate: is the
Kbytes/sec since the last pfod message. The data rate varies between
mobiles and Android versions and also varies depending of what other
apps are running.
The < … { } is a keep alive message sent very 5sec if no other pfod commands are being sent by the user. The > … {} is the response back from the WICED board 0.038sec later. If the data rate is too high for the Android mobile to handle then the 0.038sec will progressively get longer and longer and eventually exceed the connection timeout (default 10sec) and cause the pfodApp to disconnect and reconnect.
The pfodApp chart is updated about once per sec. You can freeze the plot by putting your finger on the chart. You can also use two fingers to zoom the chart and drag the chart.
Reducing the PLOT_DATA_INTERVAL to 500uS
unsigned
long PLOT_DATA_INTERVAL = 500;// uS == 0.5mS == 2kHz samples , edit
this to change the plot data interval
doubles the
sample rate so that the data rate is about 40Kbytes/sec.
The ASUS zenphone5 running Android 4.4.2 has a processing dataRate is about 40Kb/sec, if the WICED's data rate exceeds that, pfodApp cannot keep up and quickly times out. You can set the pfodApp connection time out to 0 so it never times out, but eventually the WiFi TCP buffers will fill up and block the data being sent.
When running pfodApp on a Nexus P6 you can reduce the
PLOT_DATA_INTERVAL to 400uS
unsigned
long PLOT_DATA_INTERVAL = 400;// uS == 0.4mS == 2.5kHz samples , edit
this to change the plot data interval
which give a
data rate of about 48Kb/sec.
There are a number of factors that limit the data rate for an Arduino/Android based system. Of the boards I have here, the Adafruit WICED connected via WiFi to a ASUS zenfone5 or Nexus P6 running pfodApp (V2.0.224+), was the fastest, achieving ~40Kb/sec (~2000 samples/sec.) The pfodApp can also plot the data and you can freeze the plot by putting you finger on it. Using a 5V mini router, like the GL AR150, allows you to take the whole set up out in the field.
AndroidTM is a trademark of Google Inc. For use of the Arduino name see http://arduino.cc/en/Main/FAQ
The General Purpose Android/Arduino Control App.
pfodDevice™ and pfodApp™ are trade marks of Forward Computing and Control Pty. Ltd.
Contact Forward Computing and Control by
©Copyright 1996-2020 Forward Computing and Control Pty. Ltd.
ACN 003 669 994