Data Platform Engineer - II

Experience: 3 to 7 years

Are you passionate about solving roadblocks and challenges faced by MSMEs in India?

MSMEs contribute significantly to India’s total GDP. Around 90% of India’s ~$1 trillion retail market is controlled by Micro, Small & Medium Enterprises (MSMEs). This means approximately $900 billion worth of commerce flows through nearly 60 million MSMEs across the country.

At Khatabook, we envision empowering MSMEs and helping them increase their incomes. We have built a product that brings efficiency to MSME operations by providing easy-to-use tools to manage receivables, inventory, and billing, creating transparency in cash flow. We have also enabled access to short-term working capital loans for select users demonstrating good credit behavior.

Our app has been downloaded over 100 million times, with 8 million+ monthly active users generating 220 million+ transactions with a transaction value of $18 billion.

Why work with us

People are our biggest asset! At Khatabook, every one of us is a dynamic superstar. We have carefully built an ecosystem that brings together incredibly talented people who can dream, collaborate, experiment, and break new ground. We are a strong team that looks out for each other.

Your role

We, at Khatabook, are looking to hire a Data Engineer II for our Data Engineering team.

As a Data Engineer II, you will build, maintain, and scale reliable, secure, and cost-efficient data infrastructure that enables data-driven decision-making across the organisation. You will work on production-grade data pipelines, cloud data warehouses, CDC systems, streaming platforms, and large-scale data processing.

You will play a key role in ensuring data reliability, quality, security, and performance while contributing to AI-ready data infrastructure that supports AI/ML and Generative AI use cases.

What would you do at Khatabook?

-

Design, develop, and maintain scalable batch and streaming data pipelines using Python and Apache Airflow.

-

Build reliable data ingestion workflows with appropriate error handling, retries, monitoring, and recovery mechanisms.

-

Develop incremental data processing workflows and Change Data Capture (CDC) pipelines to ensure timely and consistent data availability.

-

Build, maintain, and optimize cloud data warehouses, preferably Snowflake, with a focus on scalability, performance, and cost efficiency.

-

Develop and maintain SQL-based data models and transformation workflows using tools such as DBT.

-

Build event-driven data workflows using Kafka or other messaging and queuing technologies.

-

Develop and optimize Apache Spark jobs for large-scale data processing.

-

Build and operate data engineering workloads on AWS and work with Kubernetes and containerised environments where appropriate.

-

Optimize SQL queries, data models, Spark jobs, warehouse utilisation, and pipeline execution to improve performance and reduce infrastructure costs.

-

Implement data quality checks, validation frameworks, monitoring, alerting, and failure-recovery mechanisms to ensure data accuracy, freshness, and reliability.

-

Implement data access controls, role-based access control (RBAC), secure data sharing, and appropriate security practices for sensitive data.

-

Support data governance, metadata management, and data lineage to improve data discoverability and trust.

-

Build and maintain data pipelines that prepare structured and unstructured data for AI/ML and Generative AI applications.

-

Support AI-powered search and analytics use cases involving Large Language Models (LLMs), embeddings, vector databases, and Retrieval-Augmented Generation (RAG).

-

Collaborate with engineering, product, and analytics teams to understand data requirements and deliver trusted, actionable data.

-

Take ownership of production workloads, troubleshoot pipeline failures, investigate incidents, and continuously improve data platform reliability and operational efficiency.

-

Explore AI-assisted development tools to improve Python and SQL development, debugging, and code quality while validating generated code for correctness and security.

-

Participate in a shared production on-call rotation approximately once every three weeks to ensure the reliability of critical data workflows.

What are we looking for?

-

You have 3–7 years of experience in Data Engineering or a related technical role.

-

You have strong Python programming skills and advanced SQL knowledge, including complex joins, window functions, query optimisation, and performance tuning.

-

You have hands-on experience building and maintaining production-grade data pipelines using Apache Airflow.

-

You have experience working with modern data warehouses such as Snowflake or Amazon Redshift.

-

You have experience with CDC pipelines, incremental data ingestion, or database replication.

-

You have working knowledge of AWS cloud services and experience operating data engineering workloads.

-

You have experience with Apache Spark or PySpark for large-scale data processing.

-

You have experience with Kafka or other messaging and queuing technologies.

-

You have a strong understanding of relational databases such as PostgreSQL or MySQL.

-

You understand data modeling, data quality, pipeline reliability, monitoring, troubleshooting, and production support.

-

You have strong analytical, problem-solving, and debugging skills.

-

You understand distributed systems, data security, and performance optimisation.

-

You have a strong sense of ownership and can independently troubleshoot and improve production systems.

-

You can collaborate effectively with engineering, product, and analytics teams to deliver scalable data solutions.

Good to have:

-

Strong knowledge of Snowflake security, RBAC, secure data sharing, and compute and storage cost optimisation.

-

Familiarity with DBT for data transformation, testing, documentation, and data model management.

-

Understanding of dimensional modeling and modern data warehouse architecture.

-

Familiarity with Docker, Kubernetes, CI/CD, and automated testing.

-

Experience with data observability, metadata management, and data lineage.

-

Understanding of Generative AI concepts, including LLMs, embeddings, vector databases, and RAG.

-

Exposure to AI-ready data pipelines for structured and unstructured data.

-

Familiarity with AI-assisted development tools for Python and SQL.

-

Exposure to AI application platforms such as AWS Bedrock or Snowflake Cortex.

-

Familiarity with BI and analytics platforms such as Tableau, Redash, or similar tools.