Lead Data Engineer

About the role

HackerRank's data platform is at an inflection point. We've completed a multi-year modernisation - migrating from Redshift to StarRocks + Apache Hudi - and cut export latencies from 25 seconds to under 5 seconds. The infrastructure groundwork is done. Now we're building the AI-native data layer that will power revenue-generating features like natural language querying for HackerRank for Work customers.

As Lead Data Engineer, you'll be a senior individual contributor at the heart of the data organisation - owning complex platform decisions, collaborating cross-functionally with AI, product, and go-to-market teams, and shipping data-driven features that directly drive revenue. This is a greenfield opportunity to shape the next phase of data at HackerRank.

What you will do

• Own and evolve the data platform - StarRocks (OLAP), Apache Hudi (Data Lake), Trino, Spark, and Apache Ranger - ensuring performance, reliability, and security at scale.
• Build the next-gen AI-optimised data layer: clean, structured datasets that power natural language querying and AI add-on features for HackerRank for Work customers.
• Own in-product data features - exports, insights dashboards, interview analytics, and the self-serve Custom Reports interface.
• Enable self-service pipelines for internal teams (AI platform, analytics, go-to-market), reducing ad-hoc data requests and scaling data access across the org.
• Enforce robust data security - access controls, Apache Ranger policies, and confidence-scoring guardrails for AI-generated outputs.
• Lead technical design reviews and define engineering standards for the data team.
• Partner with PMs and business stakeholders to proactively identify and scope AI-enabled data use cases.

Who you are

• 6+ years of data engineering experience, with at least 2 years in a senior or lead capacity.
• Deep hands-on expertise with OLAP databases - StarRocks, ClickHouse, Druid, or similar.
• Strong experience with data lake technologies - Apache Hudi, Iceberg, or Delta Lake.
• Proficient with distributed query engines (Trino / Presto) and batch/streaming compute with Apache Spark.
• Solid understanding of data security, RBAC, and access control tools like Apache Ranger.
• Comfortable working in a hybrid AWS + open-source self-managed environment.
• Strong communicator who can translate technical decisions for non-technical stakeholders and drive cross-functional projects independently.

Even better if you have

• Hands-on experience with AI/LLM-adjacent data work - confidence scoring, agentic pipelines, RAG architectures, or vector stores.
• Prior exposure to agentic workflows and understanding how to operationalise emerging AI concepts at production scale.
• Experience scaling data infrastructure at a SaaS or B2B product company.
• Familiarity with natural language querying interfaces or building data products for end-customer consumption.

You will thrive in this role if

• You're energised by working on a platform that's both technically mature and still has enormous greenfield ahead of it.
• You don't wait for a PM to hand you a roadmap - you proactively connect data capabilities to business outcomes.
• You care as much about how other teams use data as you do about the pipelines that produce it.
• You're genuinely curious about AI and want to be close to where data and intelligence intersect.
• You thrive in lean, cross-functional environments where your decisions have visible, company-wide impact.