Job Description

Greymatter Innovationz helps you stay digitally relevant across domains, technologies, and skillsets, every day.

We are looking for:

Lead Azure Databricks Platform Administrator

Location: NJ/ Hybrid/ Remote

Duration: Contract Or FTE

The Azure Databricks Platform Administrator / Databricks Platform and Service Engineering Specialist is a critical role responsible for architecting, administering, and optimizing Azure Databricks environments and the services that run on them. This role bridges platform engineering and data engineering, enabling scalable, secure, and high-performing analytics solutions on Azure Databricks. The specialist collaborates closely with data engineers, cloud platform teams, and business stakeholders to ensure reliable operations, efficient data workflows, and robust analytics capabilities.

Candidates with 10+ years of overall experience and 3–4+ years of hands-on Databricks experience are preferred.

Platform Engineering

• Design, build, and administer Azure Databricks platform infrastructure, ensuring scalability, resilience, and cost optimization.
• Automate cluster provisioning, workspace configuration, and resource management using Infrastructure as Code (IaC) tools such as Terraform or ARM/Bicep.
• Implement and manage security controls, including RBAC, service principals, secrets management, and data encryption within Azure Databricks and Azure AD.
• Monitor, troubleshoot, and fine-tune platform performance using Databricks CLI, REST APIs, and Azure-native observability tools.

Service Engineering

• Collaborate with data engineering teams to deploy, monitor, and optimize ETL pipelines, streaming workloads, and ML models on Azure Databricks.
• Architect and implement Delta Lake integrations with enterprise data warehouses, data lakes, and real-time data platforms.
• Support and troubleshoot Spark workloads, optimizing resource utilization and resolving performance bottlenecks.
• Establish CI/CD workflows for notebooks, jobs, and ML models using Azure DevOps, GitHub Actions, or Jenkins.

Governance and Compliance

• Enforce robust data governance using Unity Catalog, ensuring data lineage, access control, and compliance with regulatory standards.
• Implement comprehensive logging, monitoring, and alerting systems for platform health and operational oversight.
• Develop governance frameworks for workspace provisioning, cluster policies, and secure data access patterns.

Collaboration and Leadership

• Act as the subject matter expert for Azure Databricks, guiding engineers, analysts, and business users on best practices and optimization strategies.
• Work cross-functionally with Cloud Engineering, DevOps, Security, and Data Science teams to align platform capabilities with organizational objectives.
• Create and maintain documentation for platform architecture, operational runbooks, and troubleshooting processes to streamline onboarding and operations.

Required Skills & Qualifications

Technical Expertise

• Strong experience administering and optimizing Azure Databricks and Spark-based workloads.
• Hands-on cloud expertise with Azure (preferred), plus experience with AWS or GCP Databricks environments.
• Proficiency with IaC tools such as Terraform, ARM, Bicep, CloudFormation, or Pulumi.
• Strong programming experience in Python, Scala, and SQL.
• Deep understanding of Delta Lake, schema evolution, and data partitioning strategies.
• Familiarity with MLflow for experiment tracking and model management.

Platform & Service Management

• Expertise in CI/CD practices and tools (Azure DevOps, GitHub, Jenkins) and version control (Git).
• Strong understanding of identity, access control, and data security frameworks including GDPR, HIPAA, or equivalent.
• Experience with observability tools like Azure Monitor, Prometheus, Datadog, or CloudWatch.

Soft Skills

• Excellent troubleshooting and analytical skills for distributed systems and data pipelines.
• Strong communication skills with the ability to collaborate effectively across technical and non-technical teams.
• Ability to produce high-quality documentation and knowledge-sharing artifacts.

Preferred Qualifications

• Databricks Certified Associate Developer, Data Engineer, or Architect certifications.
• Experience with enterprise data governance tools such as Alation or Collibra.
• Knowledge of containerization and orchestration (Docker, Kubernetes).

At Greymatter Innovationz,

We offer:

Motivating Work Environment.

Excellent Work Culture.

Help you to upgrade yourself to the next level.

And More!!!

Job Tags

Similar Jobs