Databricks Solution Architect (Data Engineering, AI & MLOps)

Job Summary

We are seeking a highly skilled and hands-on Databricks Solution Architect with deep expertise in designing, building, and optimizing Lakehouse-based Data Engineering, Machine Learning, and AI solutions on the Databricks platform. This contract role is based in Singapore, offering flexibility to work on-site with enterprise clients or remotely within the region.

The ideal candidate will have strong experience in architecting modern data platforms, implementing scalable data pipelines, enabling MLOps, and delivering AI/LLM use cases using Databricks technologies. You will lead technical solution design, mentor engineering teams, and collaborate closely with business stakeholders to deliver secure, scalable, and high-performance data solutions.

Key Responsibilities

• Design, architect, and implement enterprise-scale Databricks Lakehouse solutions for analytics, data engineering, machine learning, and AI workloads.
• Develop scalable batch and streaming data pipelines using Delta Live Tables (DLT), Apache Spark, Spark SQL, Python, and Scala.
• Build, optimize, and maintain Delta Lake architectures for high-performance data processing.
• Configure and administer Databricks Workspaces, Unity Catalog, SQL Warehouses, clusters, and job orchestration.
• Design secure, governed, and compliant data platforms using Unity Catalog, access controls, and data governance best practices.
• Implement Infrastructure as Code (IaC) using Terraform and automate deployments through Git-based CI/CD pipelines.
• Optimize ETL, ELT, streaming, and machine learning workloads for performance, scalability, cost efficiency, and reliability.
• Build and operationalize MLOps pipelines using MLflow, model registry, experiment tracking, and automated model deployment.
• Develop AI and Generative AI solutions leveraging DBRX, Mosaic AI, Retrieval-Augmented Generation (RAG), and LLM frameworks.
• Integrate enterprise AI services such as OpenAI, Amazon Bedrock, Azure OpenAI, or Hugging Face where applicable.
• Implement monitoring, observability, and cost optimization using Splunk, Prometheus, CloudWatch, or similar monitoring platforms.
• Collaborate with cross-functional teams, business stakeholders, architects, and customers to define technical roadmaps and deliver enterprise data solutions.
• Provide technical leadership, conduct architecture reviews, and mentor engineering teams on Databricks and cloud best practices.

Required Skills & Experience

• 8+ years of experience in Data Engineering, Big Data, Cloud Data Platforms, or Analytics.
• 4+ years of hands-on experience with the Databricks platform.
• Strong expertise in:
• Databricks Workspaces
• Delta Lake
• Delta Live Tables (DLT)
• Unity Catalog
• SQL Warehouses
• Notebooks
• Job orchestration
• Strong programming skills in Python and/or Scala.
• Extensive experience with Apache Spark and Spark SQL.
• Strong knowledge of Lakehouse architecture, data modeling, ETL/ELT design, and streaming data processing.
• Experience implementing MLOps practices using MLflow, model lifecycle management, and automated deployment pipelines.
• Hands-on experience with Terraform, Git, Azure DevOps, GitHub Actions, or Jenkins.
• Strong understanding of cloud platforms:
• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)
• Experience implementing security, governance, and data access controls using Unity Catalog.
• Experience with monitoring, logging, and observability tools.
• Strong problem-solving, communication, and stakeholder management skills.