das2C

Das streams are a self-describing, streamable format that allows for the transmission of large and complex data sets between different systems and programming environments. Unlike common file formats, streaming data formats do not provide total lengths embedded in array headers, and they do not provide arrays as a single continuous block of memory, instead everything is record oriented, including the number of records! Stream processors are able to perform calculations on data as it passes from input to output. Handling data as streams allows for processing effectively infinite datasets on modest hardware. Stream data processing is the heart of Das.

Das systems have been around for a while. This a short video provides a historical overview of das processing.

The initial version, now called das1 was designed by Larry Granroth at U. Iowa in 1996 and provided web forms for driving server-side processing that creating static plots which were then displayed in a browser. Processing was broken in to readers which generated data streams and plotters which provided graphical output. Readers were space-mission specific, but plotters were not. This basic division continues through das3.

Work on the now widely deployed das2 system began in 2002 with the goal of bringing mission agnostic interactive display and analysis tools to the desktop. A precise description of das v2.2 streams can be found in the das v2.2.2 Iterface Control Document on github. The most successful outgrowth of das2 is the feature-rich Autoplot Java application.

This Version

This edition of das2C adds support for das3 streams via the new dataset.c and codec.c modules. It also provides support for the fault-tolerant das federated catalog via node.c, and many more new features. Supporting more complex datasets and reduction algorithms required a new data model which is depicted in the diagram below. Keeping the following container hierarchy in mind will help while buried deep in code.

External projects providing other new das3 tools include:

• dasFlex for streaming data in multiple formats via self-advertised APIs
• dasTelem for auto-parsing raw CCSDS instrument packets from PostgreSQL

In addition to the C library, a small collection of stream processing programs are included in the utilities folder. These are:

• das1_ascii - Convert das1 binary streams to text
• das1_bin_avg - Reduce the size of das1 streams by averaging data in the X direction
• das2_ascii - Convert das2 binary streams to text
• das1_bin_avg - Reduce the size of das2 streams by averaging data in the X direction
• das2_bin_avgsec - Reduce the size of das2 streams by averaging data in time bins
• das2_bin_peakavgsec - Another reducer, this one produces both averages and max values in time
• das2_bin_ratesec - Provide a das2 stream data rate per event second summary
• das2_cache_rdr - Read from "database" of pre-reduced das2 streams
• das2_from_das1 - Upconvert das 1.0 streams to das 2.2 format
• das2_from_tagged_das1 - Upconvert das v1.1 streams to das 2.2 format.
• das2_hapi - Convert a das2 stream to a Heliophysics API stream.
• das2_histo - Covert a das2 stream to a stream of histograms
• das2_psd - Convert a das2 amplitude stream to a das2 Power Spectral Density stream
• das3_cdf - Write das v2 & v3 streams as CDF files.
• das3_csv - Write das v2 & v3 streams as CSV files
• das3_node - Get the location and REST API of a das3 data source

Reading Streams

A das stream is typically read by:

1. Making an HTTP request via das_http_getBody() (optional)
2. Creating a DasIO object
3. Registering your stream handling callbacks via DasIO_addProcessor()
4. Calling DasIO_readAll() read the stream an invoke your callbacks

or, if you want to break with the stream processing mentality and just spool all data in RAM:

1. Making an HTTP request via das_http_getBody() (optional)
2. Creating a DasIO object
3. Creating a ::DasDsBldr object and passing it into DasIO_addProcessor()
4. Calling DasIO_readAll() to process your input.
5. Calling DasDsBldr_getStream() to get all data objects.

The following example is a minimal stream reader that prints information about all datasets found in a das stream file.

#include <das/core.h>
 
#define PROG_ERR 63  // be nice to the shell, don't use 0 here 
 
int main(int argc, char** argv)
{
  das_init(argv[0], DASERR_DIS_EXIT, 0, DASLOG_INFO, NULL);
  if(argc < 2)
    return das_error(ERR_RET, "Input file name missing");   
  
  DasIO* pIn = new_DasIO_file("das_example", argv[1], "r");
  DasDsBldr* pBldr = new_DasDsBldr();
  DasIO_addProcessor(pIn, (StreamHandler*)pBldr);
  
  DasIO_readAll(pIn);
 
  size_t uDataSets = 0;
  DasDs** pDataSets = DasDsBldr_getDataSets(pBldr, &uDataSets);
  
  if(uDataSets == 0)
    daslog_info_v("No datasets found in %s", argv[1]);
  
  char sBuf[2048] = {'\0'};
  for(size_t u = 0; u < uDataSets; ++u){
    DasDs_toStr(pDataSets[u], sBuf, 2047); // one less insures null termination
    daslog_info(sBuf);
  }
  return 0;
}

Note

This example doesn't bother to free unneeded memory since it's built as a standalone program and the OS will free it on exit. When working in long running environments del_DasDs() and related functions should be called to free memory when it's no longer needed.

Here's an example of building the program via gcc:

gcc -o test_prog test_prog.c libdas3.a -lfftw -lssl -lcrypto -lexpat -lpthread -lz -lm

cl.exe /nologo /Fe:test_prog.exe test_prog.c das3.lib fftw3.lib zlib.lib libssl.lib \
                                 libcrypto.lib expatMD.lib Advapi32.lib User32.lib \
                                 Crypt32.lib ws2_32.lib pthreadVC3.lib

In all likelihood you'll need to use "-L" or "/LIBPATH" to provide the location of the libraries above unless you've copied them to the current directory.

Here's an example for reading one of the das2 streams out of the included test directory:

./test_prog test/das2_ascii_output1.d2t

Relationship to other das Libraries

Das2C is used by the following external projects:

• das2py for fast reading of binary data directly into NumPy arrays.
• Das2DLM to provide a native extension module for the Interactive Data Language (IDL)
• das2D for D language support. SDDAS for reading data provided by das servers

Hopefully das2C is useful for your projects as well.