DasIO Struct Reference

Tracks input and output operations for das2 stream headers and data. More...

#include <das3/io.h>

Collaboration diagram for DasIO:

Public Member Functions
DAS_API DasIO *	new_DasIO_cfile (const char *sProg, FILE *file, const char *mode)
	Create a new DasIO object from a standard C FILE.
DAS_API DasErrCode	DasIO_model (DasIO *pThis, int nModel)
	Set the parsed stream data model.
DAS_API void	DasIO_embedAsBytes (DasIO *pThis, bool bEnable)
	Recover undecodable embedded formats as opaque bytes instead of failing.
DAS_API DasIO *	new_DasIO_cmd (const char *sProg, const char *sCmd)
	Create a new DasIO object from a shell command.
DAS_API DasIO *	new_DasIO_file (const char *sProg, const char *sFile, const char *mode)
	Create a new DasIO object from a disk file.
DAS_API DasIO *	new_DasIO_socket (const char *sProg, int nSockFd, const char *mode)
	Create a new DasIO object from a socket.
DAS_API DasIO *	new_DasIO_str (const char *sProg, char *sbuf, size_t len, const char *mode)
	Create a new DasIO object for reading/writing to memory buffer.
DAS_API DasIO *	new_DasIO_ssl (const char *sProg, void *pSsl, const char *mode)
	Create a new DasIO object using an encrypted connection.
DAS_API int	DasIO_addProcessor (DasIO *pThis, StreamHandler *pProc)
	Add a packet processor to be invoked during I/O operations.
DAS_API int	DasIO_readAll (DasIO *pThis)
	Starts the processing of the stream read from FILE* infile.
DAS_API DasErrCode	DasIO_writeStreamDesc (DasIO *pThis, DasStream *pSd)
	Writes the data describing the stream to the output channel (e.g.
DAS_API DasErrCode	DasIO_writePktDesc (DasIO *pThis, PktDesc *pd)
	Writes the data describing a packet type to the output channel (e.g.
DAS_API DasErrCode	DasIO_writeDesc (DasIO *pThis, DasDesc *pDesc, int iPktId)
	Serialize a top-level descriptor to an destination.
DAS_API DasErrCode	DasIO_writePktData (DasIO *pThis, PktDesc *pPd)
	Sends the data packet on to the stream after checking validity.
DasErrCode	DasIO_writeData (DasIO *pThis, DasDesc *pDesc, int iPktId)
	Send all packet data associated with a descriptor.
DAS_API DasErrCode	DasIO_writeException (DasIO *pThis, OobExcept *pSe)
	Output an exception structure.
DAS_API DasErrCode	DasIO_writeComment (DasIO *pThis, OobComment *pSc)
	Output a StreamComment Stream comments are generally messages interpreted only by humans and may change without affecting processes.
DAS_API DasErrCode	DasIO_sendLog (DasIO *pThis, int level, char *msg,...)
	Send a log message onto the stream at the given log level.
DAS_API void	DasIO_setLogLvl (DasIO *pThis, int minLevel)
	Set the minimum log level that will be transmitted on the stream.
DAS_API int	DasIO_getLogLvl (const DasIO *pThis)
	Get logging verbosity level.
DAS_API DasErrCode	DasIO_setTaskProgress (DasIO *pThis, int progress)
	Place rate-limited progress comments on an output stream.
DAS_API void	DasIO_throwException (DasIO *pThis, DasStream *pSd, das_except_t type, char *msg)
	Set the logging level.
DAS_API void	DasIO_close (DasIO *pThis)
	Normal stream close with no unusual conditions Closes the output file descriptor, flushes a gzip buffer, etc.
DAS_API int	DasIO_printf (DasIO *pThis, const char *format,...)
	Print a string with a format specifier (Low-level API) This works similar to the C printf function.
DAS_API size_t	DasIO_write (DasIO *pThis, const char *data, int length)
	Analog of fwrite (Low-level API)
DAS_API int	DasIO_read (DasIO *pThis, DasBuf *pBuf, size_t nBytes)
	Analog of fread (Low-level API)
DAS_API int	DasIO_readUntil (DasIO *pThis, DasBuf *pBuf, size_t nBytes, char cStop)
	Read until encountering a given byte (Low-level API)
DAS_API int	DasIO_getc (DasIO *pThis)
	Analog of getc (Low-level API)

Detailed Description

Tracks input and output operations for das2 stream headers and data.

Members of this class handle overall stream operations reading writing packets, checking packet lengths, passing XML string data off to descriptor object constructors, triggering processing callbacks and most other general Das2 Stream IO tasks.

Member Function Documentation

new_DasIO_cfile()

DAS_API DasIO * new_DasIO_cfile	(	const char *	sProg,
		FILE *	file,
		const char *	mode
	)

Create a new DasIO object from a standard C FILE.

Create a DasIO object that reads or writes to C standard IO files. The C file object's read or write state should be compatible with the DasIO state. For example:

DasIO* dio = new_DasIO_cfile(stdin, "myprogram", "rc");

is invalid, if standard input is not compressed, but the library has no way to tell until read operations start.

Parameters

sProg	A spot to store the name of the program creating the file this is useful for automatically generated error and log messages
file	a C standard IO file object.
mode	A string containing the packet tag mode, one of: 'r' read either tag type 'r2' read only das v2 packet tags (error on anything else) 'r3' read only das v3 packet tags (error on anything else) 'w','w2' write das v2 stream uncompressed 'w3' write das v3 stream uncompressed 'wc','wc2' write das v2 stream compressed 'wc3' write das v3 stream compressed

DasIO_model()

DAS_API DasErrCode DasIO_model	(	DasIO *	pThis,
		int	nModel
	)

Set the parsed stream data model.

When set to false either das2 or das3 data structs are generated depending on the stream content. When set to true, das2 data structures encountered in the stream are up-converted to das3.

Parameters

pThis	The DasIO object to configure, must be in read mode
nModel	The internal data structure version to use. If set to 2 any das3 structures encountered will trigger a failure. If set to 3 then any das2 structures will be upgraded to das3. Use -1 to indicate mixed model streams.

Returns

DAS_OKAY if successful or an error code if not.

DasIO_embedAsBytes()

DAS_API void DasIO_embedAsBytes	(	DasIO *	pThis,
		bool	bEnable
	)

Recover undecodable embedded formats as opaque bytes instead of failing.

By default a stream carrying an embedded format (an encoding="blob" value with a mime, e.g. "image/png") whose codec extension is not registered is rejected with a named error. When this option is enabled the reader instead ingests the raw bytes as a degenerate per-record blob and records the lost declared morphology (mime, index, semantic, storage) as "embedded*" properties on the variable. This is lossy: the decoded index structure is NOT preserved.

Parameters

pThis	The DasIO object to configure, must be in read mode
bEnable	true to recover embedded formats as bytes, false (default) to fail loud on an undecodable embedded format.

new_DasIO_cmd()

DAS_API DasIO * new_DasIO_cmd	(	const char *	sProg,
		const char *	sCmd
	)

Create a new DasIO object from a shell command.

Create a DasIO object that reads from a sub command. The sub command is run in the shell '/bin/sh' on POSIX systems

DasIO* dio = new_DasIO_cmd("/my/das/reader 2017-01-01 2017-01-02", "wc");

is invalid, but the library has no way to tell until operations start.

Parameters

sProg	A spot to store the name of the program running the command this is useful for automatically generated error and log messages
sCmd	the command line to run, will be started via popen

new_DasIO_file()

DAS_API DasIO * new_DasIO_file	(	const char *	sProg,
		const char *	sFile,
		const char *	mode
	)

Create a new DasIO object from a disk file.

Parameters

sProg	A spot to store the name of the program creating the file this is useful for automatically generated error and log messages
sFile	the name of a file on disk. If the file doesn't exist it is created.
mode	A string containing the mode, one of: 'r' read (reads compressed and uncompressed files) 'w' write uncompressed 'wc' write compressed

new_DasIO_socket()

DAS_API DasIO * new_DasIO_socket	(	const char *	sProg,
		int	nSockFd,
		const char *	mode
	)

Create a new DasIO object from a socket.

Parameters

sProg	A spot to store the name of the program creating the file this is useful for automatically generated error and log messages
nSockFd	The socket file descriptor used for recv/write calls
mode	A string containing the mode, one of: 'r' read (reads compressed and uncompressed files) 'w' write uncompressed 'wc' write compressed

Returns

A new DasIO object allocated on the heap

new_DasIO_str()

DAS_API DasIO * new_DasIO_str	(	const char *	sProg,
		char *	sbuf,
		size_t	len,
		const char *	mode
	)

Create a new DasIO object for reading/writing to memory buffer.

Parameters

sProg	A spot to store the name of the program creating the file this is useful for automatically generated error and log messages
sbuf	The string to read.
len	The length of the string to read
mode	A string containing the mode, one of: 'r' read (reads compressed and uncompressed files) 'w' write uncompressed 'wc' write compressed

Returns

A new DasIO object allocated on the heap

new_DasIO_ssl()

DAS_API DasIO * new_DasIO_ssl	(	const char *	sProg,
		void *	pSsl,
		const char *	mode
	)

Create a new DasIO object using an encrypted connection.

This class does not handle communications setup but can take over after a connection has been established and all needed headers have been sent and or read.

It is also assumed that the SSL connection has been established on a socket using BLOCKING I/O and is not suitable for use as an action for a select() or poll() statement. To handle multiple connections at once in a single program create more than one DasIO object.

You should also setup the SSL connection with the SSL_MODE_AUTO_RETRY option to SSL_set_mode or SSL_CTX_set_mode to let the openssl library handle session renegotiation transparently. The http_getBodySocket() call does initialize any SSL structures it generates in auto-retry mode.

Parameters

sProg	A spot to store the name of the program creating the file this is useful for automatically generated error and log messages
pSsl	A pointer to OpenSSL SSL structure referencing an open connection that has been initialized in BLOCKING mode.
mode	A string containing the mode, one of: 'r' read (reads compressed and uncompressed files) 'w' write uncompressed 'wc' write compressed

Returns

DasIO_addProcessor()

DAS_API int DasIO_addProcessor	(	DasIO *	pThis,
		StreamHandler *	pProc
	)

Add a packet processor to be invoked during I/O operations.

A DasIO object may have 0 - DAS2_MAX_PROCESSORS packet processors attached to it. Each processor contains one or more callback functions that will be invoked and provided with DasDesc objects.

During stream read operations, processors are invoked after the header or data have been serialized from disk. During stream write operations processors are invoked before the write operation is completed.

Parameters

pThis	The DasIO object to receive the processor.
pProc	The StreamProc object to add to the list of processors.

Returns

The number of packet processors attached to this DasIO object or a negative error value if there is a problem.

DasIO_readAll()

DAS_API int DasIO_readAll

(

DasIO *

pThis

)

Starts the processing of the stream read from FILE* infile.

This function does not return until all input has been read or an exception condition has occurred. If a StreamHandler has been set with DasIO_setProcessor() then as each header packet and each data packet is read callbacks specified in the stream handler will be invoked. Otherwise all data is merely parsed and then discarded.

Returns

a status code. 0 indicates that all processing ended with no errors, otherwise a non-zero value is returned.

DasIO_writeStreamDesc()

DAS_API DasErrCode DasIO_writeStreamDesc	(	DasIO *	pThis,
		DasStream *	pSd
	)

Writes the data describing the stream to the output channel (e.g.

File* ).

This serializes the descriptor structure into XML and writes it out.

The DasStream object type and version will be set before serialization

Parameters

pThis	The IO object, must be set up in a write mode
pSd	The stream descriptor to serialize

Returns

0 on success a positive integer error code other wise.

DasIO_writePktDesc()

DAS_API DasErrCode DasIO_writePktDesc	(	DasIO *	pThis,
		PktDesc *	pd
	)

Writes the data describing a packet type to the output channel (e.g.

File* ).

This serializes the descriptor structure into XML and writes it out. The packet type will be assigned a number on the das2Stream, and data packets will be tagged with this number.

DasIO_writeDesc()

DAS_API DasErrCode DasIO_writeDesc	(	DasIO *	pThis,
		DasDesc *	pDesc,
		int	iPktId
	)

Serialize a top-level descriptor to an destination.

This is a replacement for DasIO_writePktDesc() that handles both das2 DasPkt and newer das3 DasDs items. Since DasDs does not hold a copy of it's packet ID, it must be supplied externally.

Parameters

pThis	the IO object to use for sending the data
pDesc	A top level in-band data descriptor such as DasPkt or DasDs.

Returns

DAS_OKAY if the operation was successful or an positive value on an error.

DasIO_writePktData()

DAS_API DasErrCode DasIO_writePktData	(	DasIO *	pThis,
		PktDesc *	pPd
	)

Sends the data packet on to the stream after checking validity.

This check insures that all planes have been set via setDataPacket.

Parameters

pThis	the IO object to use for sending the data
pPd	a Packet Descriptor loaded with values for output.

Returns

0 if the operation was successful or an positive value on an error.

DasIO_writeData()

DasErrCode DasIO_writeData	(	DasIO *	pThis,
		DasDesc *	pDesc,
		int	iPktId
	)

Send all packet data associated with a descriptor.

Note

For descriptors of type PACKET, this is the same as calling DasIO_writePktData()

Parameters

pThis	the IO object to use for sending the data
pDesc	A descriptor that can write packetized data, in this case one of PACKET, or DATASET

Returns

DAS_OKAY if the operation was successful or a positive value on error.

DasIO_writeComment()

DAS_API DasErrCode DasIO_writeComment	(	DasIO *	pThis,
		OobComment *	pSc
	)

Output a StreamComment Stream comments are generally messages interpreted only by humans and may change without affecting processes.

Progress messages are sent via stream comments.

DasIO_sendLog()

DAS_API DasErrCode DasIO_sendLog	(	DasIO *	pThis,
		int	level,
		char *	msg,
			...
	)

Send a log message onto the stream at the given log level.

Note that messages sent at a level greater than INFO may be thrown out if performance would be degraded.

DasIO_setLogLvl()

DAS_API void DasIO_setLogLvl	(	DasIO *	pThis,
		int	minLevel
	)

Set the minimum log level that will be transmitted on the stream.

The point of the DasIO logging functions is not to handle debugging, but to inform a client program what is going on by embedding messages in an output stream. Since the point of a stream is to transmit data, placing too many comments in the stream will degrade performance.

Unless this function is called, the default logging level is LOGLEVEL_WARNING

Parameters

pThis	the DasIO object who's log level will be altered.
minLevel	A logging level, one of: LOGLVL_ERROR - Only output messages that indicate improper operation. LOGLVL_WARNING - Messages that indicate a problem but processing can continue LOGLVL_INFO - Extra stuff the operator may want to know LOGLVL_CONFIG - Include basic setup information as well

DasIO_getLogLvl()

DAS_API int DasIO_getLogLvl

(

const DasIO *

pThis

)

Get logging verbosity level.

Parameters

pThis

a DasIO object to query

Returns

A logging level as defined in DasIO_setLogLevel()

DasIO_setTaskProgress()

DAS_API DasErrCode DasIO_setTaskProgress	(	DasIO *	pThis,
		int	progress
	)

Place rate-limited progress comments on an output stream.

Messages are decimated dynamically to try to limit the stream comment rate to about 10 per second (see targetUpdateRateMilli). If Note the tacit assumption that is the stream is reread, it will be reread at a faster rate than it was written. Processes reducing a stream should take care to call setTaskProgress themselves instead of simply forwarding the progress comments, so that the new stream is not burdened by excessive comments. See setTaskSize().

DasIO_throwException()

DAS_API void DasIO_throwException	(	DasIO *	pThis,
		DasStream *	pSd,
		das_except_t	type,
		char *	msg
	)

Set the logging level.

Close and free an output stream with an exception.

Call this to notify stream consumers that there was an exceptional condition that prevents completion of the stream. For example, this might be used to indicate that no input files were available. This is a convenience function for the calls:

• DasIO_writeStreamDesc()
• DasIO_writeException()
• DasIO_close()
• del_DasIO()

Parameters

pThis	the output object to receive the exception packet
pSd	The stream descriptor. All Das2 Streams have to start with a stream header. If this header hasn't been output then this object will be serialized to create a stream header.
type	the type of exception may be one of the enumerated values DAS_EX_NO_DATA, DAS_EX_SERVER_ERR, DAS_EX_QUERY_ERR or some other short type string of your choice.
msg	a longer message explaining what went wrong.

DasIO_close()

DAS_API void DasIO_close

(

DasIO *

pThis

)

Normal stream close with no unusual conditions Closes the output file descriptor, flushes a gzip buffer, etc.

May possibly send a stream termination comment as well.

Parameters

pThis

DasIO_printf()

DAS_API int DasIO_printf	(	DasIO *	pThis,
		const char *	format,
			...
	)

Print a string with a format specifier (Low-level API) This works similar to the C printf function.

Parameters

pThis
format
...	The variables to print

Returns

The number of characters printed.

DasIO_readUntil()

DAS_API int DasIO_readUntil	(	DasIO *	pThis,
		DasBuf *	pBuf,
		size_t	nBytes,
		char	cStop
	)

Read until encountering a given byte (Low-level API)

Read until hitting the stop byte. The stop byte is copied to the buffer.

---

The documentation for this struct was generated from the following file:

• das3/io.h