Data Engineer
BySix develops the best solutions to face companies' challenges and needs.
We work closely with our partners, and our projects grow towards business value creation.
We believe that true impact comes from combining technical expertise with human values. That's why we reflect that in our approach - technology is not the end goal, but the enabler.
As a Full-Time Big Data Engineer, you will play a crucial role in our organization’s data-driven initiatives, responsible for designing, developing and maintaining large-scale data processing systems. You will work collaboratively with cross-functional and multi-cultural teams to implement efficient data solutions that facilitate data ingestion, storage, processing and analysis, enabling data-driver decision-making across the organization. Your expertise in big data technologies and engineering will be vital in driving the success of our data-driven projects and contributing to the grown and innovation.
Because you matter, you will...
- Be integrated into a multidisciplinary team;
- Have continuous training in soft skills and hard skills;
- Have an attractive flexible benefits package.
Your daily challenges will be:
- Design and maintain scalable and robust big data processing infrastructure, including data pipelines and ETL workflows;
- Design data ingestion pipelines to gather data from various sources such as databases, APIs and files;
- Design data processing pipelines to clean, filter and pre-process data for analysis;
- Setup and manage distributed computing clusters and frameworks to ensure efficient data processing and analytics;
- Implement and optimize secure data storage solutions to accommodate large volumes of structured and unstructured data;
- Optimize data pipelines, framework code, cluster utilization and configuration to ensure the best performance from pipelines;
- Design, maintain and enforce data governance and data catalogue;
- Maintain a coherent documentation of the framework, cluster and projects;
- Collaborate with data scientists, data analysts and stakeholders to elicit data requirements and deliver effective solutions;
- Communicate complex technical concepts to non-technical stakeholders effectively;
- Evaluate and recommend new tools, frameworks and techniques to improve our data engineering projects and platform;
- Work in a continuous development framework with iterative deliveries, respecting deadlines and actively following tasks.
We are needing you because:
- You have a Bachelor’s or Master’s degree in Computer Science, Software Engineering or a related field;
- You have at least 1 year of experience as a Software Engineer, Back-End Software Engineer or Data Engineer;
- You have strong proficiency in Python (proven experience developing data-related Python frameworks);
- You have solid Software Development fundamentals;
You have proficiency in SQL queries and a good understanding of database architecture;
- You have strong analytical skills to ensure data quality and to know how to develop automated quality checks;
- You have good understanding of data modelling, data structures, data management and best practices;
- You have solid understanding of distributed computing principles and parallel processing;
- You have excellent communication skills and the ability to work effectively in a collaborative and multi-cultural team environment;
- You have knowledge on Apache Spark, Apache Hadoop, Airflow, Presto and Apache Drill is a plus;
- You are familiarity with in-house big data platforms is a plus;
- You have experience creating data sources on Data Visualization tools like Tableau and PowerBI is a plus (as well as being able to perform exploratory analysis).
Come join the perfect experience!
person_pinLocation
Lisboa
historyExperience
Min. 1 years
architectureMain skills
