Volumes¶

A volume is the top-level TIDAS object. It defines the “backend” data format and holds a metadata database for fast(er) data selection and query operations. It also contains the root of a hierarchy (a tree) of “blocks”.

Planning the Layout of a Volume¶

An important feature of TIDAS volumes is the ability to open / copy / link only a subset of the data. This selection operation is performed with a regular expression match on the tree of block names. This means that it is critical to plan out the organization of blocks to make it possible to do any desired data selections.

Most data collected from monitoring systems, experimental apparatuses, etc, have natural ways that the data might be split up into pieces. For example, if the data collection is started fresh each day, you might have a block for each day. If there are different sorts of data within one day which you might want to frequently split up, then you could have sub-blocks in each day for the different data types. Going the other direction up the hierarchy, you might organize the days into blocks for each month, each quarter, each year, etc.

Interface¶

Here is a basic reference for the volume interface.

Python¶

class tidas.Volume(*args, **kwargs)¶

Overloaded function.

__init__(self: tidas._pytidas.Volume, arg0: str, arg1: tidas._pytidas.AccessMode) -> None
__init__(self: tidas._pytidas.Volume, arg0: str, arg1: tidas._pytidas.BackendType, arg2: tidas._pytidas.CompressionType, arg3: Dict[str, str]) -> None

duplicate(self: tidas._pytidas.Volume, arg0: str, arg1: tidas._pytidas.BackendType, arg2: tidas._pytidas.CompressionType, arg3: str, arg4: Dict[str, str]) → None¶

info(self: tidas._pytidas.Volume) → None¶

link(self: tidas._pytidas.Volume, arg0: str, arg1: tidas._pytidas.LinkType, arg2: str) → None¶

root(self: tidas._pytidas.Volume) → tidas._pytidas.Block¶

C++¶

class tidas::volume

A volume is the top-level TIDAS object.

A volume stores the backend format and properties and the metadata index. It contains the root block which is the parent of all blocks in the tree. Some public methods are only used internally and are not needed for normal use of the object. These are labelled “internal”.

Public Functions

volume(): Default constructor.

~volume(): Destructor.

volume &operator=(volume const &other): Assignment operator.

volume(volume const &other): Copy constructor.

volume(std::string const &path, backend_type type, compression_type comp, std::map<std::string, std::string> extra = std::map< std::string, std::string >()): Create a new volume.

volume(std::string const &path, access_mode mode): Open an existing volume.

volume(volume const &other, std::string const &filter, backend_path const &loc): (Internal) Copy constructor, with optional selection and new location.

void copy(volume const &other, std::string const &filter, backend_path const &loc): (Internal) Create a copy of a volume, with optional selection and new location.

void duplicate(std::string const &path, backend_type type, compression_type comp, std::string const &filter = "", std::map<std::string, std::string> extra = std::map< std::string, std::string >()) const: Export a filtered subset of the volume to a new location.

void link(std::string const &path, link_type const &type, std::string const &filter) const: Create a (hard or soft) linked filtered subset of the volume to a new location.

void wipe(std::string const &filter) const

Delete the on-disk data and metadata associated with this object.

In-memory metadata is not modified.

backend_path location() const: (Internal) The current location.

block &root(): Get the root block of the volume.

block const &root() const: Get the (const) root block of the volume.

void info(std::ostream &out) const: Print info to a stream.

template <class P>

void exec(P &op, exec_order order, std::string const &filter = "")

Pass over the root block and all descendents, calling a functor on each one.

The specified class should provide the operator() method.

Blocks¶

A “block” is simply a logical collection of data which has a “name” (a string) associated with it. A block can contain data itself, other blocks, or both. A volume contains a special “root” block, which is the top of the tree / hierarchy of blocks in the volume.

Python¶

class tidas.Block(self: tidas._pytidas.Block) → None¶

aux_dir(self: tidas._pytidas.Block) → str¶

block_add(self: tidas._pytidas.Block, arg0: str, arg1: tidas._pytidas.Block) → tidas._pytidas.Block¶

block_del(self: tidas._pytidas.Block, arg0: str) → None¶

block_get(self: tidas._pytidas.Block, arg0: str) → tidas._pytidas.Block¶

block_names(self: tidas._pytidas.Block) → List[str]¶

clear(self: tidas._pytidas.Block) → None¶

clear_blocks(self: tidas._pytidas.Block) → None¶

clear_groups(self: tidas._pytidas.Block) → None¶

clear_intervals(self: tidas._pytidas.Block) → None¶

group_add(self: tidas._pytidas.Block, arg0: str, arg1: tidas._pytidas.Group) → tidas._pytidas.Group¶

group_del(self: tidas._pytidas.Block, arg0: str) → None¶

group_get(self: tidas._pytidas.Block, arg0: str) → tidas._pytidas.Group¶

group_names(self: tidas._pytidas.Block) → List[str]¶

info(self: tidas._pytidas.Block, arg0: std::ostream, arg1: int, arg2: bool) → None¶

intervals_add(self: tidas._pytidas.Block, arg0: str, arg1: tidas._pytidas.Intervals) → tidas._pytidas.Intervals¶

intervals_del(self: tidas._pytidas.Block, arg0: str) → None¶

intervals_get(self: tidas._pytidas.Block, arg0: str) → tidas._pytidas.Intervals¶

intervals_names(self: tidas._pytidas.Block) → List[str]¶

range(self: tidas._pytidas.Block) → Tuple[float, float]¶

select(self: tidas._pytidas.Block, arg0: str) → tidas._pytidas.Block¶

C++¶

class tidas::block¶

A block represents a logical grouping of data.

A block can have zero or more groups, intervals and sub-blocks. Child objects are added with a “name” associated to them. These names are used when performing selection / query operations. Some public methods are only used internally and are not needed for normal use of the object. These are labelled “internal”.

Public Functions

block()¶: Default constructor.

~block()¶: Destructor.

block &operator=(block const &other)¶: Assignment operator.

block(block const &other)¶: Copy constructor.

block(backend_path const &loc, std::string const &filter = "")¶: (Internal) Load a block from disk with the specified selection filter.

block(block const &other, std::string const &filter, backend_path const &loc)¶: (Internal) Create a copy of a block, with optional selection and new location.

void relocate(backend_path const &loc)¶: (Internal) Change the location of the block.

void sync(std::string const &filter = "")¶

(Internal) Recursively read the metadata for the block and its children from disk.

Optionally apply a selection filter.

void flush() const¶: (Internal) Recursively write metadata for the block and its children.

void copy(block const &other, std::string const &filter, backend_path const &loc)¶: (Internal) Copy with optional selection and relocation.

void link(link_type const &type, std::string const &path) const¶: (Internal) Create a link at the specified location.

void wipe() const¶

(Internal) Delete the on-disk data and metadata associated with this object.

In-memory metadata is not modified.

backend_path location() const¶: (Internal) The current location.

string aux_dir() const¶: Return the filesystem path of the auxilliary directory for this block.

void range(time_type &start, time_type &stop) const¶

Returns the extrema of timestamps of all groups in this block.

This is not recursive (does not traverse sub-blocks).

void clear()¶: Removes all groups, intervals, and sub-blocks from this block.

group &group_add(std::string const &name, group const &grp)¶

Add a group to this block using the specified name.

Returns a refererence to the newly added group.

group &group_get(std::string const &name)¶: Get a (non-const) reference to the group with the specified name.

group const &group_get(std::string const &name) const¶: Get a (non-const) reference to the group with the specified name.

void group_del(std::string const &name)¶: Delete the specified group.

vector<string> group_names() const¶: Return a list of the names of all groups.

void clear_groups()¶: Remove all groups from this block.

intervals &intervals_add(std::string const &name, intervals const &intr)¶

Add an intervals object to this block using the specified name.

Returns a refererence to the newly added intervals.

intervals &intervals_get(std::string const &name)¶: Get a (non-const) reference to the intervals object with the specified name.

intervals const &intervals_get(std::string const &name) const¶: Get a (const) reference to the intervals object with the specified name.

void intervals_del(std::string const &name)¶: Delete the specified intervals object.

vector<string> intervals_names() const¶: Return a list of the names of all intervals.

void clear_intervals()¶: Remove all intervals objects from this block.

block &block_add(std::string const &name, block const &blk)¶

Add a sub-block to this block using the specified name.

Returns a reference to the newly added block.

block &block_get(std::string const &name)¶: Get a (non-const) reference to the sub-block with the specified name.

block const &block_get(std::string const &name) const¶: Get a (const) reference to the sub-block with the specified name.

void block_del(std::string const &name)¶: Remove the specified sub-block.

vector<string> block_names() const¶: Return a list of the names of all sub-blocks.

void clear_blocks()¶: Remove all sub-blocks from this block.

block select(std::string const &filter = "") const¶: Recursively copy the current block and all descendents, applying the specified matching pattern.

void info(std::ostream &out, size_t indent, bool recurse) const¶

(Internal) Print metadata info for this block.

Mainly used for debugging.

template <class P>

void exec(P &op, exec_order order)¶

Pass over this block and all descendents, calling a functor on each one.

The specified class should provide the operator() method. Blocks may be modified by this version.

template <class P>

void exec(P &op, exec_order order) const¶

Pass over this block and all descendents, calling a functor on each one.

The specified class should provide the operator() method. Blocks are treated as const in this version.

MPI Volumes¶

On supported systems, an MPI interface to TIDAS volumes will be installed. These are very similar to serial volumes. When a volume is opened through the MPI interface, all processes get a copy of the metadata index. If any objects are added or resized on one process, the other processes will not see those changes until the synchronization method is called.

Warning

Multiple processes writing to the same group or intervals object is not supported, and may corrupt the data files for those objects!

Python¶

class tidas.mpi.MPIVolume(*args, **kwargs)¶

Overloaded function.

__init__(self: tidas._pytidas_mpi.MPIVolume, arg0: object, arg1: str, arg2: tidas._pytidas.AccessMode) -> None
__init__(self: tidas._pytidas_mpi.MPIVolume, arg0: object, arg1: str, arg2: tidas._pytidas.BackendType, arg3: tidas._pytidas.CompressionType, arg4: Dict[str, str]) -> None

comm(self: tidas._pytidas_mpi.MPIVolume) → object¶

comm_rank(self: tidas._pytidas_mpi.MPIVolume) → int¶

comm_size(self: tidas._pytidas_mpi.MPIVolume) → int¶

duplicate(self: tidas._pytidas_mpi.MPIVolume, arg0: str, arg1: tidas._pytidas.BackendType, arg2: tidas._pytidas.CompressionType, arg3: str, arg4: Dict[str, str]) → None¶

info(self: tidas._pytidas_mpi.MPIVolume) → None¶

link(self: tidas._pytidas_mpi.MPIVolume, arg0: str, arg1: tidas._pytidas.LinkType, arg2: str) → None¶

meta_sync(self: tidas._pytidas_mpi.MPIVolume) → None¶

root(self: tidas._pytidas_mpi.MPIVolume) → tidas._pytidas.Block¶

C++¶

class tidas::mpi_volume

Public Functions

mpi_volume(MPI_Comm comm, std::string const &path, backend_type type, compression_type comp, std::map<std::string, std::string> extra = std::map< std::string, std::string >()): Create a new volume.

mpi_volume(MPI_Comm comm, std::string const &path, access_mode mode, std::string const &dist = ""): Open an existing volume.

void duplicate(std::string const &path, backend_type type, compression_type comp, std::string const &filter = "", std::map<std::string, std::string> extra = std::map< std::string, std::string >()) const: Export a filtered subset of the volume to a new location.

void link(std::string const &path, link_type const &type, std::string const &filter) const: Create a (hard or soft) linked filtered subset of the volume to a new location.

void wipe(std::string const &filter) const

Delete the on-disk data and metadata associated with this object.

In-memory metadata is not modified.

block &root(): Get the root block of the volume.

block const &root() const: Get the (const) root block of the volume.

void info(std::ostream &out) const: Print info to a stream.