Senior Data Engineer (Azure/Databricks)
Acuity Analytics
- Bulgaria Malta Island
- Permanent
- Full-time
- Build and manage Databricks Workflows to orchestrate large-scale data pipelines
- Design and deliver ETL/ELT pipelines using best practices
- Work with Delta Lake and Lakehouse architecture to structure data
- Create and manage data assets using Unity Catalog
- Develop data transformations using Python (PySpark) and SQL
- Build and deploy PyFunc models into production pipelines
- Use Azure services like Key Vault, Logic Apps, Service Bus, and APIM to support secure and scalable solutions
- Set up and improve CI/CD pipelines with Azure DevOps
- Work with stakeholders to turn business needs into practical, scalable data solutions
- Strong experience with Azure Databricks (PySpark / Spark SQL)
- Solid Python and SQL skills
- Experience building data pipelines and data models
- Good understanding of Delta Lake / Lakehouse architecture
- Experience working with Azure DevOps and CI/CD
- Comfortable working with large, complex datasets
- Able to work with different teams and stakeholders
- Experience in insurance, reinsurance, or claims reporting
- Knowledge of NoSQL (e.g. Cosmos DB)
- Familiarity with event-driven or microservices architecture
- Experience with monitoring tools like Datadog or Application Insights
- Exposure to Microsoft Cognitive Services
- Agile, collaborative environment
- Focus on ownership and quality
- Supportive team that shares knowledge and ideas
- Work that has real impact across the business