Senior Data Engineer W2 - Full Time

Be a member of an innovative team working in a collaborative DataOps environment assembled from data engineers, data scientists, visualization and analytics experts drawn from IT and R&D teams. Relationship building, collaboration and influencing will be key in an environment of data complexity and difficult data integration challenges.

Responsibilities

• Support an analytical data infrastructure providing application, tool and ad-hoc access to large datasets consisting of complex data types including genomic, phenotypic, environmental, image and geospatial data.
• Lead, coach and develop more junior engineers and IT professionals.
• Engage with business stakeholders to understand strategic roadmaps and build a technical strategy to support them.
• Interface with DevOps teams to extract, load and transform data from a wide variety of data sources using big data technologies in AWS cloud including Glue, EMR, and Lambda.
• Deploy high value, high performance datasets in Snowflake.
• Creation and support of real-time data pipelines built on AWS technologies including Glue, S3, Lambda, EMR, EventBridge, Athena, Kinesis, and IoT Core.
• Embed quality and intelligent reporting capabilities into data pipelines including detection of anomalies and changes in trends with meaningful alerts and statistics.
• Continual research of the latest big data and visualization technologies to provide new capabilities and increase efficiency.
• Collaborate with data scientists and other tech teams to implement advanced analytics algorithms into our data pipelines that exploit our rich datasets for statistical analysis, prediction, clustering and machine learning.
• Help continually improve automation and simplifying data as a service.

Qualifications

• Degree in computer science, information science, engineering, mathematics or related technical discipline.
• 8+ years of industry experience in software development, data engineering, business intelligence, data science, or related field with a track record of manipulating, processing, and extracting value from large datasets.
• Strong experience with data integration (ETL/ELT) concepts.
• Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets
• Knowledge of, and experience with data transformation technologies and tools
• Experience with SQL and NoSQL technologies
• Desire to work with Agile/DataOps practices and methodologies
• Experience with continuous integration practices
• Able to write, debug, unit test, and performance test data integration processes
• Able to clearly define data quality issues
• Strong problem solving/critical thinking skills
• Ability to lead, mentor and coach other engineers
• Ability to communicate with, and influence, senior stakeholders

Additional Knowledge, Skills, Traits & Abilities:

• Relational databases including Amazon RD.
• AWS technologies including Redshift/Spectrum, S3/parquet, Glue, EMR, Lambda
• Data streaming technologies (Kinesis, Kafka) desirable
• IoT pipelines (AWS IoT Core, Kinesis) desirable
• Geospatial data manipulation and storage desirable
• Python and Java programming
• R and statistical methods
• Business rules engines such as Drools a plus
• Data modeling and virtualization
• Machine learning

Tagged as: data engineer, senior data engineer