Blog Archives - Elitmind

Banking Transformation: A Central Platform for Enterprise-Class Automated Analytics

Client: A prominent European banking institution providing comprehensive financial services across 40+ countries, specializing in retail banking, corporate financing, and innovative digital banking solutions. Industry: Finance Department: Data Mining, CRM Support & Marketing Automation Problem: Customer-centric analytics requires C360 data model (thousands of customer-centric parameters) Propensity to buy (ML) models too slow […]

Use All Spark Workers’ CPUs to read from a REST API

Spark driver and workers: Recently, we explored how to create our data sources in Spark by using syntax like: spark.read.format(“myrestdatasource”) to read from a REST API. Now, we’ll examine how to use all worker CPUs to read simultaneously from a REST API. Key idea Spark will read in parallel if we define multiple […]

Apache Spark Big Data ETL Python REST API

SQL Pipe gives headaches but comes with benefits

Databricks Runtime 16.2 introduced a new SQL pipeline syntax that reverse the traditional SQL query structure. This feature lets chain query operations using a pipe operator (|>), constructing queries step by step. While powerful, it initially feels awkward for those who have written SQL the same way for years. In this article, let […]

data engineering Databricks Pipeline Syntax SQL

Feature Store at Scale with Azure Databricks Unity Catalog

Addressing customer pain points Many organizations face the challenge of hyper-personalization at scale in their data and AI initiatives. Customers often ask: How can we optimize preparing, calculating, and storing data for hyper-personalization at scale? How can we optimize data preparation, computation, and storage for large-scale hyper-personalization? Let’s explore a real-world scenario where we […]

Big Data Cloud Computing Databricks Feature Store Unity Catalog