Abstract

The modern world is beset with massively large data sets. Unfortunately, statistical methodology is struggling to keep up with the development of analytical procedures to handle them. Usually, some form of aggregation (driven by underlying scientific questions) takes place producing sets of aggregated data which may be assembled into new types of data formats, such as intervals, or histograms, and so on. The dilemma occurs when trying to analyze these new data formats. More than a century of endeavor has given us numerous methodologies when the data are classical point observations in d-dimensional space; but now the observations become versions of hypercubes in d-dimensional space.
To date, often, analyses on these new data formats proceed by using classical surrogates; these surrogates can take various forms, but typically they are some function of the mean (of the values within the aggregated set), or a range or variance measure, or extreme values (such as interval end-points). While these approaches can certainly be useful at early stages, ultimately they are inadequate since almost-by-definition a lot of information in the data is ignored. The imperative for analysts is to develop methodologies, along with the appropriate theoretical underpinning, that take into account all the information in the data, rather than the quick-and-easy classical surrogate approach.
This presentation aims to review some available methodologies which use all the data information contrasting the results with those from classical surrogates, such as regression, principal components, clustering, and time series. Hence, opportunities for new directions will be highlighted and discussed.