Data Engineer - DataBricks

About Us

BySix Engineers the future and drives innovation. We are more than just a standard software engineering company, we believe that true impact comes from combining technical expertise with human values. That's why we reflect that in our approach - technology is not the end goal, but the enabler.

Why Us?

🫸 Multidisciplinary team 📒 Training on demand 💴 Attractive salary and benefits package 📈 Career progression plans 👌 And amazing get together events

What are we looking for?

We are looking for a Data Engineer with:

• Proficiency with PySpark and Spark SQL for data processing;
• Experience with Databricks using Unit Catalog;
• Knowledge of Delta Live Tables (DLT) for automated ETL and workflow orchestration in Databricks;
• Familiarity with Azure Data Lake Storage;
• Experience with orchestration tools (e.g., Apache Airflow or similar) for building and scheduling ETL/ELT pipelines;
• Knowledge of data partitioning and data lifecycle management on cloudbased storage;
• Familiarity with implementing data security and data privacy practices in a cloud environment;
• At least one year of experience with Terraform and knowledge of good practices of GitOps; Additional Knowledge and Experience that is a Plus:
• Databricks Asset Bundles;
• Knowledge of Kubernetes;
• Knowledge of Apache Kafka;
• Knowledge of Vault.

What do we expect from you?

• Projects understanding and Communication;
• Understand problems from a user perspective and communicate to clearly understand the issue;
• Ensure the architecture provided by the Data Architect is clearly understood by yourself;
• Strong understanding of data warehousing concepts, data lakes, ETL/ELT processes, and data modeling;
• Develop data pipelines based on the defined architecture;
• Ensure the regular good practices are applied;
• Deploy requested infrastructure, particularly using Terraform;
• Make peer reviews and ask to your peers to review your code when merging a new version of the codebase;
• Define tests with your project manager, based on the functional and technical requirements of the pipeline you’re developing;
• Perform those tests and communicate regularly on the results;
• Regularly summarize the results of your tests in a dedicated document;
• Present to the Data Architect in charge of the architecture, and the Lead DataOps, the development that was performed through our Deployment Reviews;
• Track and communicate on any potential errors in the entire period of active monitoring following a deployment;
• Ensure diligent application of deployment process, logging, and monitoring strategy.

Are you ready?

If you're ready to make a real impact by leveraging cutting-edge technology and fostering human-centric solutions, BySix is the place for you. Together, we'll drive innovation and create lasting business value.

Note: BySix is an equal opportunity employer. All applicants will be considered and analysed regardless of ethnicity, religion, gender identity, sexual orientation, national origin, age, or disability status.