Suriya Rajan
CV (PDF)

வணக்கம் · Hello, I'm Suriya

I build the data platforms behind media measurement and AdTech.

Nine years on AWS, mostly making big data jobs faster, cheaper and easier to trust. Right now I'm building serverless integrations, a RAG chatbot and a BI migration for Warner Bros. Discovery.

Email me Download CV (PDF)
  • AWS Solutions Architect – Associate
  • AWS AI Practitioner
  • Astronomer Airflow

Experience

9+ years in data engineering

  • Based in Bangalore, India
  • Languages English (fluent), Tamil (native)

Results

What changed because of my work

Six numbers from the past nine years, with where each one happened. Before-and-after bars are indexed to 100 unless a unit is shown.

Compute cost

Nielsen

−40%

Moved hourly Spark jobs from 5-node r4.4xlarge EMR clusters, which bill while they spin up and down, to EMR Serverless, which bills only for job runtime.

Spark job speed

Nielsen

30%+ faster

Upgraded 15+ Spark 2.x ETL jobs to 3.x and switched on adaptive query execution and dynamic partition pruning. Runs got faster and used less memory.

Retailer onboarding

PDI

2 weeks → 2–3 days

Automated onboarding with PySpark ETL and GitHub Actions CI/CD. Deployment time halved too.

Manual intervention

PDI, TCS

−60%

Rebuilt Airflow orchestration at PDI. Earlier, at TCS, automating Airflow workflows cut manual effort by 40%.

ETL throughput

TCS (Nielsen)

+25%

Tuned SQL queries and ETL jobs so downstream reports landed sooner.

Data volume

Nielsen

1–1.5 TB per daily pipeline

Owned the Connected TV ad-ratings platform, from ingestion through to reporting.

View these numbers as a table
Before and after results
MeasureBeforeAfterWhere
Compute cost (index)10060Nielsen
Spark job runtime (index)10070 or lessNielsen
Retailer onboardingAbout 2 weeks2–3 daysPDI Technologies
Manual intervention (index)10040 at PDI, 60 at TCSPDI, TCS
ETL throughput (index)100125TCS (Nielsen)
Daily pipeline size1–1.5 TB each, two pipelines; 3–5 ad hoc pipelines of 50–300 GBNielsen

How I work

How a data platform fits together

A simplified picture of the platforms I've built and run. Pick a stage to see the tools I used there and where. It's a general view, not any one employer's system.

Ingest

Getting data in without people in the loop.

  • API Gateway
  • Lambda
  • SQS
  • S3
  • WBDDesigned a serverless REST middleware on API Gateway for the Nordic AdTech platform, replacing manual work by business staff. In pilot with 3 external vendors.
  • PDIRe-platformed legacy ingestion for 50+ retailer pipelines onto serverless AWS services, for better fault tolerance and availability.
  • TCSBuilt data flows from Nielsen's on-premises systems to AWS for Watch Analytics.

Process

Turning raw events into something people can use, fast and without overspending.

  • Apache Spark 2.x/3.x
  • PySpark
  • Scala
  • EMR
  • EMR Serverless
  • Glue
  • NielsenMoved hourly Spark jobs to EMR Serverless (40% lower cost) and upgraded 15+ jobs to Spark 3.x (at least 30% faster).
  • PDIAutomated retailer onboarding with PySpark ETL, from about 2 weeks to 2–3 days.
  • TCSWrote Scala utilities and Git-based CI/CD for JAR build, test and deployment.

Store

Keeping data safe, cheap to hold and easy to query.

  • S3
  • Apache Iceberg
  • Oracle
  • PostgreSQL
  • Hive
  • HDFS
  • PDIMigrated terabyte-scale retail datasets from on-premises to AWS (S3, EMR, Glue) with no downtime and no loss of data integrity.
  • WBDMoving the datasets that about 400 MicroStrategy reports need from Oracle to Apache Iceberg.

Serve

Getting answers in front of the people who need them.

  • Amazon Quick Suite
  • Athena
  • MicroStrategy
  • FAISS
  • Amazon Bedrock
  • WBDMigrating about 400 MicroStrategy reports to Amazon Quick Suite.
  • WBDBuilt a RAG chatbot over 500+ market-research documents. Prototyped on EC2, then moved to Quick Suite with chat personas and dashboards. Approved in UAT.
  • ProjectSathyaMool.ai uses Textract, Translate and Bedrock to verify Indian property documents.

Orchestrate

Making sure every job runs in the right order, and that it's repeatable.

  • Apache Airflow
  • Terraform
  • GitHub Actions
  • GitLab CI
  • Git
  • NielsenRedesigned the CTV ingestion DAGs with retries and structured alerting. SLA adherence improved and recovery after failures got quicker.
  • PDIRebuilt Airflow orchestration, cutting manual pipeline intervention by 60%.
  • TCSDesigned the Airflow framework for the CTV rebuild and put the CTV infrastructure on Terraform in 2019. Earlier, moved a 10-person team from SVN to Git.

Observe

Catching problems early, before the people reading the reports do.

  • CloudWatch
  • Structured logging
  • Data-quality checks
  • Schema validation
  • SLAs
  • NielsenAdded data-quality checks and schema validation so bad upstream data stops before it reaches reporting.
  • PDIAdded CloudWatch monitoring and structured logging, so the team caught problems earlier and resolved incidents faster.
  • WBDDefined SLA, observability and cost-control standards with the Nordic engineering teams.

Experience

Nine years, five roles

Current marks where I work now, Completed marks earlier roles.

  1. Current

    – Present

    Remote

    Senior Data Engineer

    Lancesoft, client Warner Bros. Discovery

    I'm the only data engineer on the Market Research GenAI and MicroStrategy migration work, and the technical lead for the Nordic middleware. Client title: Data Engineer IV.

    • Designed a serverless REST middleware on Amazon API Gateway for WBD's Nordic AdTech platform. It replaces manual work by business staff and is in pilot with 3 external vendors. I mentored a junior developer on architecture and system design throughout the build.
    • Proposed and built a RAG chatbot for the Market Research team over 500+ documents (FAISS, embeddings and a 3B-parameter LLM). I prototyped it on EC2, then moved it to Amazon Quick Suite with chat personas and dashboards. It has passed UAT, and go-live is planned for November 2026.
    • Leading the migration of about 400 MicroStrategy reports to Amazon Quick Suite, including moving the datasets they need from Oracle to Apache Iceberg. The first 50 priority reports come first.
    • Defined SLA, observability and cost-control standards with the Nordic engineering teams.

  2. Completed

    –

    Bangalore

    Data Engineer (Member of Technical Staff II)

    Nielsen, DAR CTV

    I owned the AWS data platform (EMR Serverless, S3, Airflow) behind Nielsen's Connected TV Digital Ad Ratings, from ingestion through to reporting.

    • Ran two major daily pipelines of 1–1.5 TB each, plus 3–5 ad hoc pipelines of 50–300 GB.
    • Cut compute cost by 40% by moving hourly Spark jobs from 5-node r4.4xlarge EMR clusters (billed for spin-up and spin-down) to EMR Serverless, which charges only for job runtime. I also right-sized executors and tuned dynamic allocation.
    • Upgraded 15+ Spark 2.x ETL jobs to Spark 3.x and enabled adaptive query execution and dynamic partition pruning. Runs got at least 30% faster and used less memory.
    • Redesigned the Airflow DAGs for CTV ingestion with retries and structured alerting. SLA adherence improved and recovery after failures got quicker.
    • Added data-quality checks and schema validation so bad upstream data is stopped before it reaches reporting.
    • Mentored 4 junior engineers and an intern on distributed design, Spark internals and Airflow, and helped experienced new joiners get up to speed and start delivering quickly.

  3. Completed

    –

    Remote

    Data Engineer II, Retailer Onboarding

    PDI Technologies

    I made retailer onboarding faster and more automatic, and moved the data estate to AWS.

    • Automated retailer onboarding with PySpark ETL pipelines and GitHub Actions CI/CD. Onboarding a new retailer dropped from about 2 weeks to 2–3 days, and deployment time halved.
    • Migrated terabyte-scale retail datasets from on-premises to AWS (S3, EMR, Glue) with no downtime and no loss of data integrity.
    • Re-platformed legacy ingestion for 50+ retailer pipelines onto serverless AWS services, improving fault tolerance and availability.
    • Rebuilt Airflow orchestration, cutting manual pipeline intervention by 60%.
    • Added CloudWatch monitoring and structured logging, so the team caught problems earlier and resolved incidents faster.

  4. Completed

    –

    Chennai

    System Engineer, DAR CTV

    Tata Consultancy Services, for Nielsen

    I ran production CTV analytics pipelines and designed the Airflow framework for the CTV rebuild.

    • Ran production CTV analytics pipelines for Nielsen Digital Ad Ratings across 10+ streaming channels, keeping to SLAs and data-accuracy targets.
    • Designed the Airflow framework for the CTV rebuild, and wrote Scala utilities and Git-based CI/CD for JAR build, test and deployment.
    • Automated workflows in Airflow, cutting manual intervention by 40% and reducing processing errors.
    • Put the CTV infrastructure on Infrastructure as Code with Terraform in 2019, and have used it as the standard way to provision ever since.

  5. Completed

    –

    Chennai

    Assistant System Engineer, CORE DAR

    Tata Consultancy Services, for Nielsen

    My first role: connecting Nielsen's on-premises systems to AWS.

    • Built data flows between Nielsen's on-premises systems and AWS for Watch Analytics, so measurement data kept moving during the transition.
    • Tuned SQL queries and ETL jobs, raising throughput by 25% and getting downstream reports out sooner.
    • Migrated a 10-person team from SVN to Git and set branching rules that reduced merge conflicts.

Stack

Tools I reach for

What I use day to day, and what I know well enough to work with.

Data engineering

  • Apache Spark 2.x/3.x
  • PySpark
  • Scala
  • Apache Airflow
  • ETL/ELT
  • Data modelling
  • Data quality
  • Governance
  • Observability
  • SLA management

AWS

  • EMR
  • EMR Serverless
  • Glue
  • Lambda
  • API Gateway
  • SQS
  • S3
  • Athena
  • CloudWatch
  • IAM
  • VPC
  • EC2
  • Amazon Bedrock
  • Amazon Quick Suite

Languages

  • Python
  • Scala
  • SQL
  • Bash

DevOps

  • Terraform (IaC since 2019)
  • AWS CloudFormation (working knowledge)
  • GitHub Actions
  • GitLab CI
  • Git
  • Docker (home lab)
  • Serverless design

Lakehouse and BI

  • Apache Iceberg
  • Oracle
  • PostgreSQL
  • MicroStrategy

GenAI

  • RAG
  • LLMs
  • FAISS
  • Embeddings

Hadoop ecosystem

  • Cloudera
  • Hive
  • Oozie
  • HDFS
  • Unix

Foundational knowledge

  • Amazon Kinesis
  • Databricks
  • Snowflake
  • Amazon Redshift
  • dbt
  • Kubernetes

Domains

  • AdTech
  • Media measurement
  • CTV and OTT ad ratings
  • Retail data onboarding

Credentials

Projects, certifications and education

Project ·

SathyaMool.ai

AI for Bharat Hackathon, Hack2skill. Selected for the prototype stage of the national programme.

A serverless AWS platform that uses generative AI to verify Indian property documents. It's built with Amazon Textract, Translate and Bedrock, plus Lambda, API Gateway, SQS, DynamoDB, S3 and AWS CDK.

  • Textract
  • Translate
  • Bedrock
  • Lambda
  • API Gateway
  • SQS
  • DynamoDB
  • S3
  • AWS CDK
View repository on GitHub (opens in a new tab)

Certifications

  • AWS Certified AI Practitioner Amazon Web Services · , valid to
  • AWS Certified Solutions Architect – Associate Amazon Web Services · , valid to
  • Apache Airflow 3 Fundamentals Astronomer ·
  • Airflow DAG Authoring and Fundamentals Astronomer ·
Verify badges on Credly (opens in a new tab)

Education

B.Tech, Electrical and Electronics Engineering
SASTRA University, Tamil Nadu, India · 2017

Languages

English (C1–C2, fluent) · Tamil (native)

Contact

Get in touch

The easiest way to reach me is email. LinkedIn works too.

suriyarajan12@gmail.com