Senior Data Engineer (Azure, Databricks)

This initiative centers on a mature, high-availability data hub architected within Azure Databricks, utilizing Unity Catalog, Spark Structured Streaming, and Asset Bundles to ensure robust performance and governance. The solution is built upon the Medallion Architecture framework, providing a scalable foundation that is now being expanded to ingest, transform, and seamlessly integrate new data sources into the existing ecosystem.

You will be responsible for

• Developing Databricks pipelines in PySpark / SQL
• Ingesting data from APIs
• Handling Spark Structure Streaming pipelines
• Designing and managing objects in Unity Catalog according to Medallion Architecture

You will need at least one of these

• At least 6 years of commercial experience in Data Engineering with focus on large scale data pipelines.
• Proven track record using advanced solutions in Azure cloud and Databricks
• Knowledge of Spark Structured Streaming
• Proficiency in Python and SQL
• Ability to build pipelines using Github or similar tools

You will get

• 100% remote work with the possibility to use an office in Warsaw
• Flexible working hours and engaging company events
• Access to Azure and GCP training packages available to official partners
• Possibility to use an in-house incubator to develop your own ventures
• A supportive community of 50+ senior data consultants with extensive experience in Data Warehouse and Big Data
• and much more!