Publications

Results 2126–2150 of 9,998

Search results

Jump to search filters

A vision for managing extreme-scale data hoards

Proceedings - International Conference on Distributed Computing Systems

Logan, Jeremy; Mehta, Kshitij; Heber, Gerd; Klasky, Scott; Kurc, Tahsin; Podhorszki, Norbert; Widener, Patrick W.; Wolf, Matthew

Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.

More Details
TYPE Conference Poster YEAR 2019
DOIOSTIScopus
Results 2126–2150 of 9,998
Results 2126–2150 of 9,998