.png)
Job Description
Capstone IT is helping our client to find a Senior Data Architect – Databricks who will lead the design, implementation, and modernization of enterprise data platforms using Databricks and cloud technologies.
This individual will be responsible for building scalable, secure, and high-performing data environments from initial architecture and configuration through production deployment. They will design Databricks Lakehouse solutions, develop data pipelines, implement data governance, and help migrate legacy data systems into modern cloud platforms.
The ideal candidate combines strategic architecture experience with strong hands-on engineering skills. They must be comfortable making technical decisions, writing code, configuring environments, troubleshooting complex issues, and leading engineering teams.
The biggest differentiator: Our client needs someone who has personally led a complete Databricks implementation—not just someone who has worked within an established Databricks environment.
Top 4 Required Skills
1. Databricks Architecture & End-to-End Implementation
3+ years of hands-on Databricks experience, including at least one enterprise implementation from initial architecture and environment setup through production.
2. Python, PySpark, Apache Spark & SQL
Advanced data engineering skills, including building and optimizing distributed data pipelines and ETL/ELT solutions.
3. Cloud Data Architecture & Lakehouse Design
Strong Azure, AWS, or GCP experience with Delta Lake, medallion architecture, cloud storage, and enterprise platform modernization.
4. Unity Catalog, Governance & Technical Leadership
Experience implementing governance, security, access controls, data quality, architectural standards, and guiding engineering teams.
Main Responsibilities
-
Architect and deploy enterprise Databricks Lakehouse platforms from the ground up.
-
Lead environment configuration, data integration, migration, testing, and production deployment.
-
Design data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake.
-
Implement Bronze, Silver, and Gold data layers.
-
Establish governance and security using Unity Catalog.
-
Modernize legacy data warehouses and ETL platforms.
-
Optimize compute performance, scalability, reliability, and cloud costs.
-
Implement CI/CD pipelines and infrastructure automation.
-
Develop architecture diagrams, standards, and technical roadmaps.
-
Provide hands-on technical leadership to engineering, DevOps, analytics, and BI teams.
Required Qualifications
|
Qualification |
Requirement |
|---|---|
|
Overall experience |
8+ years in data architecture, engineering, or warehousing |
|
Databricks |
3+ years hands-on |
|
Implementation |
At least one complete Databricks implementation |
|
Programming |
Advanced Python, PySpark, and SQL |
|
Cloud |
Azure, AWS, or GCP |
|
Lakehouse |
Delta Lake, Unity Catalog, Spark |
|
Data engineering |
ETL/ELT, integration, modeling, optimization |
|
Modernization |
Legacy-to-cloud migration experience |
|
Leadership |
Architecture ownership, technical mentoring, stakeholder communication |
Preferred: Databricks certifications, Terraform, CI/CD, Azure Data Factory, AWS Glue, dbt, Airflow, Power BI, Tableau, streaming, and AI/ML workloads.
What Success Looks Like
The successful candidate delivers a production-ready Databricks platform, establishes secure and reusable architecture patterns, improves data processing performance, and enables engineering teams to build reliable analytics solutions independently.