|
Home
| pfodApps/pfodDevices
| WebStringTemplates
| Java/J2EE
| Unix
| Torches
| Superannuation
|
| About
Us
|
|
|
High Frequency, Long Duration Datalogging
|
by Matthew Ford 16th January 2020
(originally posted 21st June 2019)
© Forward
Computing and Control Pty. Ltd. NSW Australia
All rights reserved.
Update 13th July 2019: Version 0.01 revised cmd handling and add extra buffers, added jar file
This project was developed to support a university medical research team, who needed a wearable that could log 2 x ECG signals at 1000 samples/sec each (2K samples/sec total) continuously for 30 days, in order to detect arrhythmias. The project presented here includes remote control and monitoring of the logging. Remote control is via menus presented on a serial terminal, either on a computer or mobile phone. This project does not cover the ECG measurement or packaging or battery required for the final wearable.
This high speed/long duration version uses Teensy 3.2, Adafruit Micro-SD breakout module, a quality 16G SDHC class 10 SD card to log the data and a Bluetooth communications module for control and monitoring. A less developed and slower UNO/Mega2560 version of this project is also available. While this project use a Bluetooth communications module for control and monitoring of the logging, you can also use WiFi or BLE modules.
This version, based on the Teensy 3.2, is capable of much higher sampling rates than the UNO/Mega2560 version. Using this code the Teensy 3.2 can sample and log two ADC samples at >30Khz with hardware averaging over 4 samples and so easily satisfies the 1000 samples/sec requirement above. The code supports saving 100 .bin files of 128K each. At 30Khz that covers 29hrs 30min. At 1000 samples/sec it covers 37 days. The code can easily be extended to handle more than 100 files, so extending the duration of the run.
At the end of the run you will have >10Gig of data, .bin, files and a .met file of meta data describing the run and results. The supplied SDtoCSV.jar (source code SDtoCSV_src.zip ) can be used to convert the .bin files to .csv files on your computer for further processing. The resulting data is >60Gig. The UNO/Mega2560 version has .bin to .csv conversion included in the Arduino sketch, but given the volume of data logged by the Teensy version, that is not an efficient way to do the conversion.
Teensy
3.2 by PJRC
Adafruit
MicroSD card breakout board+ or similar.
16G
SDHC class 10 MicroSD card of good quality e.g. SanDisk.
5V
USB supply
A led with a 470R resister in series.
2 x 100R
resistors (provides protection from damage due to Tx/Rx wiring
errors)
Bluetooth
Mate Silver OR one of the modules described on Arduino
UNO/Mega Starter, controlled by Android/pfodApp
Download and install Arduino IDE V1.8.9+ from http://arduino.cc/en/Main/Software. That web page has links for various operating systems and a link to GettingStarted (http://arduino.cc/en/Guide/HomePage).
Download and install the Teensyduino (Teensy Support for the Arduino IDE). NOTE carefully the First Usage instructions.
Select Teensy 3.2 as the board and check that the BLINK example program loads and runs.
Download and install the following
libraries :- millisDelay
and SdFat
(A local snapshot of the SdFat library used for these tests is here.)
and the pfodParser.zip
(for the pfodBufferedStream and pfodNonBlockingInput class)
Download
the library zip files and then use the Arduino IDE → Sketch →
Include Library → Add .ZIP library menu to install the libraries
from the zip files.
Unzip the Teensy32AnalogLogger.zip file to your Arduino sketches directory and program the Teensy 3.2 board with Teensy32AnalogLogger.ino (Version 0.01)
Wire up the Teensy 3.2, Bluetooth module and the SD card module as shown below (pdf version)
First format your SD card using
the https://www.sdcard.org/downloads/formatter/
The SD card must be empty in order to start logging.
For initial testing you do not need to connect the Communications module, just connect the Teensy 3.2 + SD module (with an empty card installed) to the Arduino IDE via the USB serial cable. As provided the Teensy32AnalogLogger.ino sketch uses the USB connection for control and monitoring. See Logging Real Data for using a communication device for control and monitoring.
Edit the top of the
Teensy32AnalogLogger.ino sketch to set the COM_SERIAL to Serial, to
output to the Teensy USB connection.
#define COM_SERIAL
Serial
Then upload the
sketch to the Teensy 3.2
Open the Arduino IDE Serial Monitor at 115200 baud (with both NL & CR set). After a few secs the Teensy 3.2 will display a menu of commands
Ver:0.01 enter one of the following commands:
? - current status and metadata
i - initialize files
l - list files
>The ? cmd displays details of the current settings. (See the top of the Teensy32AnalogLogger.ino to change these settings) Cmds must be terminated with a NL or CR or both.
0:00:00.000 of 720:00:00.000
Sample pins: 16 17 Byte order : Little-Endian
ADC bits: 10 ADC sample averages over : 4
Sample Rate: 1000.00 Sample interval: 1000uS
Samples per block: 127 Time to fill block : 127000uS
Time to fill a file : 9:01:52.000 Time to fill ALL files : 894:04:48.000
Max SD latency (includes file close/open) : 0uS Max file close/open latency : 0uS
Number of buffer blocks: 28 Time to fill ALL block buffers : 3556000uS
Max number of buffers saved in call to storeSampleBuffers() : 0
Missed Timers total : 0 Total Missed Samples so far: 0
Total Blocks written : 0 Total Samples written : 0 covering : 0:00:00.000
Current File : In this case current logging runtime is 0 of a requested 720hrs (30days), sampling D16/A2 and D17/A3 (see below for Restrictions on the Choice of ADC Inputs) 1000 times a second. The maximum runtime can be up to 894hrs (37.25days). The main loop() can be occupied for upto 3.5sec (Time to fill ALL block buffers) before all the available buffers are filled and samples start being lost. Buffers saved etc are updated as the run progresses.
Insert an empty SD card, use the 'i' cmd to initialise the 99 files used to store the data. Pre-initializing them here reduces the time delay when switching from one file to the next and allows faster sampling.
Initializing 99 files
Creating new file : log00.bin Elapsed time : 368mS
Creating new file : log01.bin Elapsed time : 520mS
. . .
Creating new file : log98.bin Elapsed time : 15660mS
Creating new file : log99.bin Elapsed time : 15812mSYou can then use the r cmd to start a logging run. The run will for the requested time or until the s cmd is used to stop it. You can also use the ? cmd while logging to get updated timings and counts. Here is a short run stopped early using the s cmd.
LOGGING DATA .....
Ver:0.01 enter one of the following commands:
? - current status and metadata
s - stop data logging
LOGGING DATA... Check with ? command
>
Elapsed Run time : 0:00:10.000 of 720:00:00.000
Elapsed Run time : 0:00:20.000 of 720:00:00.000
. . .
s
Stopping Logging and removing unused files.
. . .
Removing unused file : log98.bin
Removing unused file : log99.bin
0:01:04.976 of 720:00:00.000
Sample pins: 16 17 Byte order : Little-Endian
ADC bits: 10 ADC sample averages over : 4
Sample Rate: 1000.00 Sample interval: 1000uS
Samples per block: 127 Time to fill block : 127000uS
Time to fill a file : 9:01:52.000 Time to fill ALL files : 894:04:48.000
Max SD latency (includes file close/open) : 204uS Max file close/open latency : 0uS
Number of buffer blocks: 28 Time to fill ALL block buffers : 3556000uS
Max number of buffers saved in call to storeSampleBuffers() : 1
Missed Timers total : 0 Total Missed Samples so far: 0
Total Blocks written : 511 Total Samples written : 64832 covering : 0:01:04.832
Current File : log00.bin
ls:
2000-01-01 01:00:00 261632 log00.bin
2000-01-01 01:00:00 240 log.met
DATA LOGGING COMPLETED!
Ver:0.01 enter one of the following commands:
? - current status and metadata
** r - record ADC data ** not available. Data already exists
** i - initialize files ** not available. Data already exists
l - list files
DATA HAS ALREADY BEEN LOGGED, check with ?
>The LED connected to D3 (with D2 providing the GND connection) will turn on solid if any sample is missed and will flash if there is an error. The sketch attempts to continue after errors but may not do so successfully.
When logging real data over long durations, it is more convenient to connect a communications module to the D0/D1 pins and control and monitor the logging remotely. Here a Bluetooth Mate Silver module was used with its default settings, 115200 baud, no hardware handshaking (RTC, CTS), pin code 1234.
Note: When power is applied to the Mate Silver module it goes into a configuration mode, fast red led blinking, for 60sec. During this time you can send $$$ via the serial connection to the module to configure it but you cannot connect the the module. Once the red led is slow blinking, the bluetooth module will accept connections.
Change the COM_SERIAL define in the
Teensy32AnalogLogger.ino to the hardware serial (D0/D1) connection,
Serial1
#define COM_SERIAL Serial1
After pairing with the computer, a new COM port was created on the computer and TeraTerm for PC (or CoolTerm Mac) can be used to connect and control and monitor the logging. Other serially connected communication modules can also be used, such as WiFi or BLE, see Arduino UNO/Mega Starter, controlled by Android/pfodApp for details.
You can also control and monitor the logging from your Android mobile using a bluetooth terminal app such as Bluetooth Terminal app, or using WiFi and TCP terminal app such as TCP Telnet Terminal Pro , or a Uart to BLE mdoule and a BLE terminal app such as nRF UART V2
The Teensy 3.2 has two separate ADC hardware modules, ADC_0 and ADC_1, in its micro-processor so it can sample two inputs at the same time. It also has a built in hardware averaging which takes multiple ADC samples and averages them before turning the result.
There are restrictions on which inputs can be connected to ADC_0, ADC_1. The Teensy3_1_AnalogCard.png image (from https://forum.pjrc.com/threads/25532-ADC-library-update-now-with-support-for-Teensy-3-1), included in the Teensy32AnalogLogger.zip file, details which pins can be connected to which ADC.
In summary:-
For Single Ended Reads i.e. +Volts
referenced to GND
ADC_0 can read A0 to A9, A10, A11, A12,
A14
ADC_1 can read A2, A3, A10, A13, A15 to A20
If you select a
pin that the ADC cannot read it will return 0 (always)
This project uses A2, A3 which can each be accessed by ADC_0 or ADC_1.
The Teensy32AnalogLogger.ino saves the samples as binary in logxx.bin files i.e. log00.bin to log99.bin. Teensy32AnalogLogger.ino also save a log.met file of meta data about the run.
You can use SDtoCSV.jar
(source code SDtoCSV_src.zip
) to convert the .bin files to .csv for further processing. Copy the
files from the SD card to your computer hard disk with at least 70Gig
of free space and copy the SDtoCSV.jar
to the same directory. Open an command window in that directory and
run
java -jar SDtoCSV.jar log
If
your computer does not have Java install then install it from
www.java.com
SDtoCSV will process the log.met file and then each of the available logxx.bin files in the directory and output a .csv file for each .bin. The .csv file has the sample sequence number followed by the two readings.
A sample console output for a 2 adc reading sampled 30303 times/sec is shown here, output.txt. It illustrates how missed samples are reported. (Doubling the number of buffers in the final sketch fixed this these missed samples)
SD_Logging to CSV conversion. Check SDtoCSV.log for progress and errors messages.
Processing log00 Processed 256000 blocks
Processing log01 Processed 256000 blocks
. . .
Processing log25 Processed 256000 blocks
Processing log26 Processed 256000 blocks
Processing log27 Processed 256000 blocks Missed Samples : 2715
Processing log28 Processed 256000 blocks
. . .
Processing log29 Processed 256000 blocks
. . .
Processing log47 Processed 256000 blocks
Processing log48 Processed 35935 blocks
--- Finished Processing
A fuller log file, SDtoCSV.log,
is appended to by each run of SDtoCSV. It includes the meta data
output and any error messages. Here the count:254
is the count of the data stored in that block i.e. 127 samples x 2
adc readings per block. The missedSamples
is the number of pairs of reading missed i.e. lines in the .csv
output.
=== Log File for SD_Logging to CSV conversion Sat Jul 13 13:19:51 AEST 2019
To see progress messages on Console use
java -jar SDtoCSV.jar <basefilename>
Base File Name 'log'
Metadata Version : 0 (Little Endian)
sampleInterval uS:33
adcBits:10
adcAvgs:4
pinCount:2
Pins: 16, 17
samplesPerBlock:127
noBufferBlocks:28
duration mS:51649820
requested runTime mS:106216704
maxBuffersUsed:32 Warning: Exceeds number of buffers available (28). Some samples may be missing.
maxLatency uS:221476 Warning: Exceeds time provided by buffer blocks (117348uS). Some samples will be missing.
maxFileOpenTime uS:20998
missedTimers:0
missedSamplesTotal:2715
totalBlocksWritten:12323935
totalSamplesWritten:1565139665
Processing log00.bin Processed 256000 blocks
Processing log01.bin Processed 256000 blocks
. . .
Processing log26.bin Processed 256000 blocks
Processing log27.bin
!!! Block:57696 count:254 missedSamples:2339
!!! Block:57697 count:254 missedSamples:376 Processed 256000 blocks --- Total Missed Samples : 2715
Processing log28.bin Processed 256000 blocks
. . .
Processing log47.bin Processed 256000 blocks
Processing log48.bin Processed 35935 blocks
--- Finished Processing
A sample of log00.csv output file is
SampleCounter (per 33uS),Pin 16,Pin 17
0,248,205
1,273,195
2,228,337
3,360,302
4,355,369
5,220,281
. . .The sample counter increases from file to file so it can be used as a time stamp. If there are missing samples then the sample counter is incremented by the missed count before outputting the next line so that the counter/time stamp remains accurate for the recorded samples.
The Teensy32AnalogLogger is a heavily modified version of Bill Greiman's AnalogBinLogger example in his SdFat Arduino library. Here the library has been rewritten to run on the Teensy 3.2.
Teensy32AnalogLogger uses timer0 to set the sample interval. The interrupt handler for timer0 starts the two ADC conversions. An interrupt handler for the second ADC module is continually called until they are both finished, usually the first ADC module started ADC_0 will finishes before the second on so the interrupt handler is only called once. The ADC_1 interrupt handler saves the samples to a data buffer.
In the main loop(), the storeSampleBuffer() saves any full buffers to the SD card and recycles the buffers to the empty buffer queue. The large amount of RAM available on the Teensy 3.2 means a large number of buffers can be allocated and so storeSampleBuffer() does not need to be call often. This leaves time for the main loop() to do other work, like process commands and send output.
While this project is functional as it is, there are
some extensions that could be considered.
i) Add real time
control and monitoring of the sampled wave form via pfodApp using
pfodApp's plotting function to show snapshots of the wave form
ii)
Extend the file numbers past 99 for longer sampling runs
iii)
Sample more than 2 inputs. Since the Teensy 3.2 has dual ADC modules,
you could modify the code to add extra inputs in pairs to maximize
the sample rate.
iv) Add battery monitoring to track battery
charge. The Teensy 3.2 uses about 1100mAhrs over 24hrs, including the
bluetooth and SD module, but excluding the sensor module
v) Add a
dual battery supply circuit to allow for battery changes with out
interrupting the logging.
For use of the Arduino name see http://arduino.cc/en/Main/FAQ
The General Purpose Android/Arduino Control App.
pfodDevice™ and pfodApp™ are trade marks of Forward Computing and Control Pty. Ltd.
Contact Forward Computing and Control by
©Copyright 1996-2020 Forward Computing and Control Pty. Ltd.
ACN 003 669 994