In-Memory Based Spatial Queries for Large-Scale 3D Data

3D spatial data has beenused in numerous industrial applications, such as CAD, urban planning, 3D mapping such as Google Earth, terrain modeling, and mineral exploration and environmental assessments. Managing and analyzing large amount of 3D spatial data to derive values and guide decision making have become essential to business success and scientific discoveries. The rapid growth of 3D spatial data is driven by not only industrial applications but also scientific applications such as 3D digital pathology imaging. Large amount of 3D micro-objects such as 3D cells and 3D blood vessels with complex structures are generated from 3D whole slide image volume. There are four main challenges for managing and querying massive 3D spatial data in spatial queries: the explosion of spatial data, the complex tree-like 3D structures, the multiple levels of details representation, and the high 3D geometrical computation complexity of spatial queries.

iSPEED is an effective and scalable in-memory based spatial querying system for large-scale 3D data. iSPEEED achieves low latency by storing data in memory in a highly compressed form using an effective progressive compression approach that compresses each 3D object individually with successive levels of detail. To minimize search space and computation cost, iSPEED provides global spatial indexing in memory through partitioning at subspace level and partitioned cuboid level. iSPEED provides an in-memory 3D spatial query engine INTENSE, which can be invoked on-demand for running many instances in parallel. The parallelization of queries is implemented in, but not limited to, MapReduce. At run time, iSPEED dynamically decompresses only needed 3D objects at the specified level of detail, and creates necessary spatial indexes in-memory to accelerate query processing, such as on-demand object-level indexing and structural indexing on complex structured objects. iSPEED supports multiple spatial queries, including spatial joins, nearest neighbor search and spatial proximity estimation, and can be easily extended to others.

iSPEED provides features such as:

  • Support for high performance spatial queries and analytics for 3D spatial big data on MapReduce
  • 3D mesh compression
  • In-memory data storage
  • Multi-level 3D spatial indexing
  • On-demand in-memory spatial query engine
  • 3D spatial data partitioning
  • Dynamic boundary objects handling

More details can be found in following links: