We are seeking an experienced Databricks Architect to design, develop, and implement scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in data architecture, cloud-based data engineering, big data technologies, and modern data platforms, with hands-on experience building and optimizing data pipelines and analytics solutions.
The Databricks Architect will work closely with business stakeholders, data engineers, data scientists, and cloud teams to translate business requirements into robust data architectures that support enterprise analytics, reporting, and data-driven decision-making.
Responsibilities:
Design and implement scalable data architectures using the Databricks Lakehouse Platform.
Develop and optimize data pipelines using Apache Spark, PySpark, SQL, and Delta Lake.
Architect and implement ETL/ELT workflows for batch and real-time data processing.
Design and manage data lakehouse solutions, including data ingestion, transformation, storage, and consumption layers.
Implement data governance, security, access controls, data quality, and metadata management using Unity Catalog and related tools.
Optimize Spark jobs, cluster configurations, query performance, and compute costs.
Integrate Databricks with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
Design and implement workflow orchestration and scheduling using Databricks Workflows and tools such as Apache Airflow.
Establish CI/CD pipelines, infrastructure automation, and deployment best practices using tools such as Git, Terraform, and cloud-native DevOps services.
Implement monitoring, logging, error handling, and performance optimization for production data pipelines.
Collaborate with data engineers, analysts, data scientists, and business teams to define technical requirements and deliver data solutions.
Provide technical leadership, conduct code reviews, establish architecture standards, and mentor engineering teams.
Ensure compliance with enterprise data security, privacy, governance, and regulatory requirements.
Evaluate emerging data technologies and recommend improvements to the existing data platform.
Databricks Architect