Cloud Data Management at Exabyte Scale

Uncategorized

Earlier this year I hosted a webinar that discussed the key data management challenges enterprises face when running or migrating big data workloads to the cloud. These workloads are moving in greater numbers to the cloud for well-known reasons: business agility, flexibility, and minimizing capital expenditures, among others. Yet, often left unanswered is the question of how companies optimize data storage and data management for these new workloads.

 

This post highlights how the Imanis Data architecture optimizes cloud data management. Our philosophy from the very beginning is to be compatible with any infrastructure deployment architecture: whether exclusively on-prem, exclusively in the cloud, or hybrid deployments. This flexible approach enables us to support a wide variety of different data backup, mirroring and recovery use cases, and is enabled by how the Imanis Data file system handles these different storage requirements.

 

The Imanis Data File System

The Imanis Data file system is built on a storage tiering model. It can federate data over multiple tiers of storage transparently based on user defined policies. The first storage tier is typically the block storage tier – examples include Elastic Block Store (EBS), managed disks, direct attached drives, storage-attached network/network-attached storage devices. The second storage tier is an object storage platform such as Amazon S3 or Azure Blob Storage. Finally, the last tier of storage is a cold storage platform such as Amazon Glacier. A user could define a policy to keep the data in the block tier for 5 days, in the object tier for 25 days and in the cold tier for 6 months. The Imanis Data file system transparently migrates data between different tiers. Similarly the data access also is transparent to the user since our file system has built-in capabilities to natively retrieve the data from different tiers using supported access protocols.

 

Data Backup and Mirroring in the Cloud

The storage in Imanis Data is unbounded. What this means is that we built the Imanis Data file system in such a way that it can automatically span the different types of storage highlighted above. For example, Imanis Data can back up a directory where two files are stored on local storage while the remaining six files are moved to S3. More importantly, this process is completely transparent to the user as to where the files are stored because our file system presents a unified namespace across different storage tiers. If a user migrates data from local storage to cloud storage using a policy that they have created, the data movement happens asynchronously without the user needing to be aware of the underlying migration process. As soon as the migration is done, the space occupied by the original files is freed up and immediately available for re-use.

Let’s take this one step further. If your company deployed workloads in a multi-cloud environment, say AWS and Azure, Imanis can designate specific backup workflows to go to S3 and others to go to Azure Blob Storage. Because we separated the compute and storage layers, Imanis Data can handle huge amounts of storage relative to the compute requirements. Furthermore, the Imanis Data storage optimization engine can store de-duplicated data on S3/Azure Blob Storage, reducing your storage footprint even further.

 

Data Recovery from the Cloud

During recovery, the Imanis Data file system will immediately figure out if the data is, for example, on the local file system or on the object storage tier and read the data from the appropriate storage location. All of the data restore operations are completely transparent to the users and the data is seamless fetched from the storage tier where it resides. This is in stark contrast to what would happen if you were to use scripts: you would need specific scripts for each storage location.

 

Conclusion

The Imanis Data architecture is flexible enough to address the key issues of scale, bandwidth, and cost of cloud data management. As a result, we’re used by some of the largest enterprises running the most demanding big data workflows. We encourage you to check out this video of how Imanis Data works and contact us to learn more about the ideal big data cloud management solution.

Sign Up To Receive Imanis Data Updates