Azure Databricks Platform Administration

Administer and manage Azure Databricks workspaces, clusters, and jobs

Configure and maintain Databricks environments (Dev, QA, Prod)

Monitor cluster performance, job execution, and system health

Manage notebooks, libraries, and dependencies

Perform upgrades, patches, and platform maintenance

Cluster Management & Optimization

Design and manage cluster configurations (auto-scaling, instance types, pools)

Optimize job performance, resource utilization, and execution times

Troubleshoot cluster failures, memory issues, and job bottlenecks

Implement best practices for Spark performance tuning

Security & Governance

Implement and enforce data security and access controls (RBAC, ACLs, IAM)

Manage workspace security, secrets (Key Vault integration), and credential passthrough

Ensure compliance with enterprise security and governance policies

Implement data governance, auditing, and lineage tracking

Azure Cloud Integration

Integrate Databricks with Azure services such as:

Azure Data Lake Storage (ADLS Gen2)

Azure Key Vault

Azure Data Factory (ADF)

Azure Synapse Analytics

Manage connectivity and networking (VNet, Private Endpoints, NSGs)

✅ Core Skills

10–15 years of IT experience with strong Azure Databricks administration

Hands-on expertise in Apache Spark and Databricks platform management

Strong experience with Azure cloud services (ADLS, ADF, Key Vault, Networking)

Proven experience in production support and platform operations

✅ Technical Skills

Deep knowledge of cluster configuration, job scheduling, and performance tuning

Strong understanding of Spark architecture and optimization techniques

Experience with Terraform / ARM / Bicep (IaC tools)

Knowledge of security models, IAM, RBAC, and data governance

Familiarity with CI/CD tools and DevOps practices

Preferred Skills

Experience with Delta Lake and Lakehouse architecture

Exposure to Python / PySpark / Scala

Familiarity with monitoring tools (Azure Monitor, Splunk)

Knowledge of multi-cloud environments (AWS/GCP)

Experience in data engineering or ETL pipelines