Data Engineer

Job Description:

Design, develop, deploy and maintain reliable ETL/ELT batch & streaming data pipelines, ingest structured/semi-structured/unstructured data from multiple source systems (DB, API, log, Kafka etc.)

Perform dimensional data modeling, build and iterate data warehouse / data lake, define table schema, partition strategy, data layers (ODS/DWD/DWS/ADS)51job

Write & optimize complex SQL, tune query performance, reduce resource cost and improve pipeline stability and SLA

Implement data quality rules, anomaly detection, monitoring & alerting, troubleshoot data lineage and data consistency issues

Use workflow orchestration tools to schedule, manage and monitor data jobs (Airflow/Dagster)

Collaborate with analysts, DS and business stakeholders to translate business requirements into data solutions, build reusable datasets and data APIs

Participate in data governance: metadata management, data security, access control, documentation for schema and pipelines51job

Evaluate and adopt cloud data technologies (Snowflake/BigQuery/Redshift), continuously optimize storage and compute cost

Job Qualification:

Experience on AWS/Azure/GCP cloud data stack

Real-time streaming data pipeline development (Kafka, Flink, Pulsar)

Data governance, data lineage, metadata platform construction experience

Experience supporting ML feature platform / feature engineering

English working proficiency

Domain knowledge: automotive, manufacturing, finance, retail etc.