Senior Data Engineer | AI automations | Azure and AWS

Role Summary

USE YOUR PASSION FOR TECHNOLOGY TO MAKE A POSITIVE IMPACT AND #VemSerLuminiIT

Here at Lumini IT, innovation and resilience are part of our routine.

We are an IT solutions company that values quality and agility in the delivery of its services.

Our big secret to always being one step ahead of the market is the audacity and transformation that each LUMINER promotes every day.

Our integrity drives us and, best of all, everyone collaborates with everyone, no matter who came up with the idea, we are a company with a horizontal strategy, everyone is willing to help and legitimately gets involved to make it work and deliver results.

Did you like it? Apply for the position!

We are looking for a Data Engineer to work on building and integrating data pipelines in Azure and AWS environments, ensuring data ingestion, processing, availability and support for automation with AI.

The professional will be responsible for integrating existing APIs, ETLs and databases, in addition to implementing scalable architectures aimed at analysis and decision making.

The project is part of a long-term program focused on modernization, innovation, Data, AI, Productivity, Optimization and Process Automation.

Responsibilities

• Design and develop data ingestion pipelines from different sources (ETLs, APIs, relational and non-relational databases, files, telemetry sensors, etc.);
• Integrate and consolidate data from different sources in the Azure and AWS ecosystems, ensuring quality, consistency and reliability;
• Implement reference architectures for data processing and storage;
• Create and manage Data Lakes, Data Warehouses and Data Marts;
• Develop and optimize ETL/ELT processes using tools such as Azure Data Factory, Databricks, Synapse Analytics, AWS Glue, AWS Lambda and Redshift;
• Ensure the governance, security and scalability of data pipelines;
• Monitor and optimize the performance of data loads, identifying bottlenecks and proposing continuous improvements;

Work in direct collaboration with data scientists, functional analysts and business areas.

Requirements

Desired Profile

• Solid experience as a Data Engineer in building scalable pipelines;
• Mastery of cloud solutions in AWS (S3, Redshift, Glue, Athena) and Azure (Data Factory, Synapse, Databricks, Azure SQL, Blob Storage, Monitoring);
• Experience in data integration via REST/SOAP APIs and real-time processing;
• Mastery of relational (SQL Server, PostgreSQL) and non-relational (CosmosDB, DynamoDB, MongoDB) databases;
• Proficiency in Python, Spark, SQL and PowerShell/Bash for data manipulation and engineering;
• Practice with DataOps culture, code versioning (Git) and CI/CD mats for data;
• Knowledge in data modeling, query optimization and database performance;
• Knowledge of good security, governance and compliance practices (LGPD);

Good verbal and written communication, with technical documentation skills and interlocution with business teams.

Differentiators

• Experience with event-driven architectures using AWS Kinesis, Azure Event Hub or Apache Kafka;
• Experience in implementing Machine Learning and MLOps pipelines in cloud environments;
• Microsoft (e.g.: DP-203 – Azure Data Engineer Associate) or AWS (e.g. AWS Certified Data Engineer / Data Analytics) certifications.

Benefits

• Work Mode: Home Office;
• Hiring: PJ;
• Team performance schedule: commercial.
• Important: We value diversity and inclusion, we consider all candidates for our vacancies, regardless of color, race, religion, gender and gender identity, nationality, disability, sexual orientation, ancestry, age, etc.
• Originally posted in Himalayas