Senior Data Engineer

Our client is a US-based company that provides In-flight Internet and entertainment services to aircraft across the globe, serving commercial aircrafts. At this moment we cooperate on Business Intelligence, Data Analysis, Data Science  and BigData directions and look for talents who can contribute to the complex data management and analysis projects.

About the Role:

We are looking for a Data Engineer with strong expertise in DBT, PostgreSQL, and Python, who will be responsible for working with a live database that requires near real-time data processing. The key requirement of the role will be ensuring that data is processed, stored, and made available with minimal delay, as latency is not acceptable. The engineer must understand the importance of optimizing performance to make data available to consumers instantaneously while doing the minimal necessary processing. 

Experience with AWS, Airflow, and Snowflake is a plus.

Key Responsibilities:

• Work with a live database that requires near real-time data processing.
• Ensure that data is processed, stored, and made available with minimal delay, as latency is not acceptable.
• Optimize system performance to enable instantaneous data availability for consumers.
• Design and implement efficient data processing solutions while minimizing unnecessary overhead.
• Monitor and troubleshoot data pipelines to maintain high availability and reliability.
• Design, develop, and maintain efficient ETL pipelines.
• Work with PostgreSQL to ensure optimal database performance and scalability.
• Collaborate with cross-functional teams to understand data needs and implement solutions.
• Ensure data quality, integrity, and compliance with best practices.
• Work with cloud-based data solutions, primarily AWS (or learn on the job).

Requirements:

• Proficiency in Python for data engineering tasks.
• At least 3 years of experience with dbt.
• Expertise in PostgreSQL as a target technology (this role involves leveraging specific PostgreSQL features, such as hypertables for automatic partitioning and pattern-based data parsing, to ensure high performance and efficiency)  
• Deep technical expertise and experience in handling real-time data, ensuring that system architecture is designed with zero delays in data availability and maximum efficiency.
• Experience with ETL/ELT processes, data ingestion, and transformation best practices.
• Familiarity with cloud-based solutions like AWS is a plus.
• Knowledge of Apache Airflow and Snowflake is advantageous, but willingness to learn is sufficient.
• Strong problem-solving skills and ability to work independently and in a team.
• Excellent communication and documentation skills.