Blog Archives - Elitmind

Use All Spark Workers’ CPUs to read from a REST API

Spark driver and workers: Recently, we explored how to create our data sources in Spark by using syntax like: spark.read.format(“myrestdatasource”) to read from a REST API. Now, we’ll examine how to use all worker CPUs to read simultaneously from a REST API. Key idea Spark will read in parallel if we define multiple […]

SQL Pipe gives headaches but comes with benefits

Databricks Runtime 16.2 introduced a new SQL pipeline syntax that reverse the traditional SQL query structure. This feature lets chain query operations using a pipe operator (|>), constructing queries step by step. While powerful, it initially feels awkward for those who have written SQL the same way for years. In this article, let […]

data engineering Databricks Pipeline Syntax SQL

Feature Store at Scale with Azure Databricks Unity Catalog

Addressing customer pain points Many organizations face the challenge of hyper-personalization at scale in their data and AI initiatives. Customers often ask: How can we optimize preparing, calculating, and storing data for hyper-personalization at scale? How can we optimize data preparation, computation, and storage for large-scale hyper-personalization? Let’s explore a real-world scenario where we […]

Big Data Cloud Computing Databricks Feature Store Unity Catalog

Foundations of digital success: Expert insights from Elitmind’s leaders

As a leading partner of tech giants like Microsoft, Databricks, and Snowflake, Elitmind has a unique perspective on the evolving IT landscape. In an exclusive interview with Piotr Waszczuk from IT WIZ, Elitmind’s leadership – Robert Woźniak, CEO and Co-founder, and Maciej Chojnacki, Head of Delivery & Technology – share their insights on […]

AI Strategy Cybersecurity Data platforms Databricks Microsoft