Databricks SME

Role Summary

Apply Databricks best practices across all pillars, including

• Security & Governance (Unity Catalog, IAM, data access controls)
• Reliability & Resilience (job retries, checkpointing, failure isolation)
• Performance Efficiency (cluster sizing, autoscaling, caching)
• Cost Optimization (compute policies, workload separation, monitoring)
• Operational Excellence (monitoring, automation, CI/CD, runbooks)
• Implementation & Engineering
• Lead Databricks workspace, cluster, and Unity Catalog implementations
• Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
• Build and optimize ETL/ELT pipelines using Spark and SQL
• Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
• Establish CI/CD pipelines for notebooks, jobs, and infrastructure
• Security, Governance & Compliance
• Implement Unity Catalog for centralized governance
• Define data classification, lineage, and audit strategies
• Enforce least-privilege access and secure networking patterns
• Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)
• Operations & Optimization
• Monitor platform health, performance, and cost
• Troubleshoot production issues across jobs, clusters, and data pipelines
• Perform workload tuning and cost-performance optimization
• Define SLOs, alerts, and operational metrics
• Collaboration & Leadership
• Partner with Data Engineering, Analytics, ML, Platform, and Security teams
• Translate business requirements into technical architectures
• Provide architectural guidance and technical mentorship
• Communicate risks, tradeoffs, and recommendations to leadership

Responsibilities

• Architecture & Platform Design
• Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
• Define reference architectures for batch, streaming, analytics, and ML workloads
• Select and standardize cluster, compute, and workspace architectures
• Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
• Ensure architectures meet scalability, availability, and performance requirements
• Well-Architected Framework Alignment

Requirements

• 7+ years in data engineering, analytics, or platform architecture
• 3–5+ years hands-on Databricks experience in production environments
• Proven experience applying the Databricks Well-Architected Framework
• Experience designing cloud-native lakehouse architectures
• Experience supporting mission-critical data platforms

Technical Skills

• Databricks Lakehouse Platform
• Apache Spark (PySpark / Scala / Spark SQL)
• Delta Lake, Delta Live Tables, Structured Streaming
• Unity Catalog (governance, lineage, access controls)
• Cloud platforms: AWS, Azure, or GCP
• Infrastructure as Code (Terraform strongly preferred)
• CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
• Data formats and protocols (Parquet, JSON, Avro)

Certifications Required

• Databricks Certified Data Engineer Professional
• Databricks Certified Professional Architect (or equivalent advanced certification)
• Preferred / Additional Certifications
• AWS Certified Solutions Architect (Associate or Professional)
• Azure Solutions Architect Expert
• Google Professional Data Engineer
• Databricks Machine Learning Professional
• Snowflake or other cloud data platform certifications
• Soft Skills
• Strong architectural decision-making and documentation skills
• Excellent communication with technical and non-technical stakeholders
• Ability to lead design reviews and architecture governance forums
• Strong troubleshooting and performance-tuning mindset
• Nice-to-Have Experience
• MLflow and MLOps architectures
• Real-time analytics and streaming pipelines
• Multi-region or cross-account data architectures
• Consulting or MSP delivery experience