What’s new in Databricks – September 2024
27 September 2024
Over the past month, we’ve received many exciting news from Databricks. Here’s a breakdown of the latest developments across various domains:
Data Engineering
Workflow Enhancements: There are further improvements to workflows, which will soon be called Lakeflows. You can now loop tasks and pass parameters to them from an array, enhancing the flexibility and efficiency of your data pipelines. For further insights, check out my latest video For each loop feature in Databricks Workflows !
Delta Live Tables Updates: Updates related to Delta Live Tables (DLT) include the introduction of liquid partitioning for DLT tables. A significant improvement is the ability to create Slowly Changing Dimensions (SCD) from a full data snapshot. For example, if your customer sends you a CSV file with a list of available products, you can now effortlessly convert it to an SCD Type 2 table.
AI
AI Gateway: Perhaps the most exciting update is the introduction of the AI Gateway. Databricks can now serve as a catalog for all AI models in the market, controlling how you access them. Features include guardrails to block Personally Identifiable Information (PII) or certain keywords, auditing capabilities, evaluation and testing of model quality, and load balancing (if a model is down, you can switch to a different provider), among others.
Administration
Policy Compliance Monitoring: Enhanced monitoring of policy compliance makes it easier for administrators to catch and fix policy violations, simplifying administrative tasks and improving governance.
Notebook Experience
AI Assistant Improvements: The AI assistant for notebooks in now more robust. You can use more slash command and display the entire command palette by clicking Ctrl + Shift + P, streamlining your workflow.
General Availability (GA)
A host of features that were previously in preview are now generally available. If GA is a requirement for you, you can start using:
- Lakehouse Monitoring
- System Tables
- Row Filtering and Column Masking
- New Runtime 15.4 LTS
What’s Coming Next
We have some exciting features becoming available in the coming weeks. One that we’re eagerly anticipating is improvements in SQL language support, which will make migrations to Databricks much easier.
Conclusion
It’s exciting to see how Databricks continues to evolve. At Elitmind, we’re dedicated to harnessing these new features to drive innovation for our clients, and we believe they hold great potential for making your work easier and more efficient. I hope the insights shared in this article have given you a clearer picture of what’s new.
If you’re interested in a deeper dive, I invite you to check out my last video, “What’s New in Databricks in September,” where I share practical examples and tips.
I encourage you to watch it and be part of this learning journey. Happy exploring!