3. Do you have challenges with your Data Lake? 

• Hard to append data – Adding newly arrived data leads to incorrect reads 
• Modification of existing data is difficult– GDPR/CCPA requires making fine-grained changes to the existing data lake 
• Jobs failing mid-way – Half of the data appears in the data lake, the rest is missing 
• Real-time operations – Mixing streaming and batch leads to inconsistency 
• Costly to keep historical versions of the data – Regulated environments require reproducibility, auditing, governance 
• Difficult to handle large metadata– For large data lakes the metadata itself becomes difficult to manage 
• “Too many files” problems– Data lakes are not great at handling millions of small files 
• Hard to get great performance– Partitioning the data for performance is error-prone and difficult to change 

Mold your own, personalized Data World 

• Openness: The storage formats they use are open and standardized, such as Parquet, and they provide an API so a variety of tools and engines, including machine learning and Python/R libraries, can efficiently access the data directly 
• Schema enforcement and governance: The Lakehouse should have a way to support schema enforcement and evolution, supporting DW schema architectures such as star/snowflake-schemas. The system should be able to reason about data integrity, and it should have robust governance and auditing mechanisms. 
• Storage is decoupled from compute: In practice, this means storage and compute use separate clusters, thus these systems are able to scale to many more concurrent users and larger data sizes. Some modern data warehouses also have this property. 

Don’t hesitate to schedule a meeting if you are interested in:

• Demo
• Consultations
• Proof of Concept