PG Master Diploma in Data Engineering with AI Full Stack

Topics:

• Introduction to programming using Python
• Variables, data types
• Control structures (if, loops)
• Functions & modules
• Error handling
• Writing clean, modular code

Topics:

• Core data structures
• Lists, dictionaries, sets
• Stacks & queues (conceptual)
• Searching & sorting basics
• Time complexity (Big-O basics)

Topics:

• Handling real-world data
• File formats: CSV, JSON, Parquet (concept)
• Reading/writing files
• Data parsing & transformation

Topics:

• Data processing
• Using Pandas
• Using NumPy
• Filtering, grouping, aggregation
• Handling missing data

Topics:

• Relational databases
• SQL basics: SELECT, WHERE, JOIN
• GROUP BY, HAVING
• Data modeling basics
• Normalization concepts

Topics:

• Data exchange between systems
• REST APIs
• JSON data handling
• API requests using Python

Topics:

• Data flow concepts
• ETL vs ELT
• Batch vs streaming (conceptual)
• Pipeline design basics

Topics:

• Working in Linux environment
• File system navigation
• Shell commands
• Basic bash scripting

Topics:

• Code management
• Git basics
• Branching & merging
• Collaboration workflows
• Tools: Git

Topics:

• Environment consistency
• Basics of Docker
• Containers vs VMs
• Running Python apps in containers

Topics:

• Industry fundamentals
• Data lakes vs data warehouses
• Data quality & governance
• Data lifecycle

Project Focus:

• Read data from API + files

Project Focus:

• Extract → Transform → Load

Project Focus:

• Complex queries + reporting

Project Focus:

• API → Processing → Database

Scenarios:

• Build data pipelines
• Clean and transform raw data
• Integrate multiple data sources
• Automate workflows
• Manage data storage

Skills Gained:

• Programming for data
• Data handling & transformation
• SQL & database skills
• Pipeline fundamentals

Topics:

• What is data storage
• Types of storage systems
• Structured vs semi-structured vs unstructured data
• OLTP vs OLAP systems
• Row-based vs column-based storage
• Storage hierarchy (memory, disk, cloud)

Topics:

• Traditional databases
• Tables, rows, columns
• Primary keys & foreign keys
• Indexing & constraints
• Normalization vs denormalization
• Tools: MySQL, PostgreSQL

Topics:

• Modern storage systems
• Types of NoSQL: Key-value, Document, Column-family, Graph
• CAP theorem basics
• Use cases
• Tools: MongoDB, Cassandra

Topics:

• Analytical storage
• Data warehouse architecture
• Star schema & snowflake schema
• Fact & dimension tables
• ETL vs ELT

Topics:

• Big data storage
• Data lakes vs data warehouses
• Schema-on-read
• Lakehouse architecture
• Data zones (raw, curated, gold)
• Tools: Apache Hadoop

Topics:

• Efficient data storage
• File formats: CSV, JSON, Parquet, ORC
• Compression techniques
• Partitioning strategies

Topics:

• Scalable storage
• Distributed file systems
• Replication & fault tolerance
• Data partitioning
• Tools: HDFS

Topics:

• Cloud-based storage
• Object storage
• Block vs file storage
• Data durability & availability
• Tools: Amazon S3, Azure Data Lake Storage, Google Cloud Storage

Topics:

• Real-world architecture
• Data partitioning
• Sharding
• Indexing strategies
• Caching basics

Topics:

• Protecting data
• Encryption
• Access control
• Data masking
• Compliance basics

Topics:

• Improving storage performance
• Query optimization
• Index tuning
• Data pruning
• Caching

Project Focus:

• Schema + ETL flow

Project Focus:

• Multi-layer storage design

Project Focus:

• Store & manage large datasets

Project Focus:

• Database + Lake + Analytics

Scenarios:

• Choosing correct storage system
• Designing scalable architecture
• Managing big data storage
• Optimizing query performance
• Handling structured & unstructured data

Skills Gained:

• Database design
• Data storage architecture
• Cloud storage handling
• Performance tuning

Topics:

• What is data transformation in Data Engineering
• ETL vs ELT concepts
• Data cleansing, enrichment, normalization
• Batch vs streaming transformations
• Real-world transformation workflows

Topics:

• Data processing with Python
• File handling (CSV, JSON, Excel)
• Data manipulation using Pandas
• Filtering, grouping, aggregation
• Handling missing & duplicate data
• Data type conversion

Topics:

• Complex transformations
• Feature engineering
• Data merging & joins
• Window operations
• Data validation rules

Topics:

• Big data concepts
• Limitations of traditional processing
• Distributed computing basics
• Introduction to Apache Spark

Topics:

• Working with PySpark
• Spark architecture (Driver, Executor)
• RDDs vs DataFrames
• SparkSession
• Lazy evaluation

Topics:

• Scalable transformations
• Filtering & selecting columns
• Aggregations & groupBy
• Joins (inner, outer)
• Handling null values
• Column transformations

Topics:

• Complex big data processing
• Window functions
• User Defined Functions (UDFs)
• Partitioning & bucketing
• Performance optimization

Topics:

• Working with different formats
• CSV, JSON, Parquet
• Reading/writing data in Spark
• Partitioned data handling

Topics:

• End-to-end data workflows
• Extract → Transform → Load
• Scheduling jobs
• Error handling

Topics:

• Cloud data processing
• Integration with Amazon S3, Azure Data Lake Storage
• Data lake transformations

Topics:

• Improving efficiency
• Caching & persistence
• Partition tuning
• Debugging Spark jobs

Project Focus:

• Data cleaning + transformation

Project Focus:

• Large dataset transformation

Project Focus:

• Python + PySpark + Cloud

Project Focus:

• Streaming pipeline design

Scenarios:

• Cleaning large enterprise datasets
• Transforming raw logs into analytics-ready data
• Building scalable ETL pipelines
• Processing millions of records efficiently

Skills Gained:

• Python data transformation
• PySpark big data processing
• ETL pipeline development
• Data optimization

Topics:

• What is Big Data
• 3Vs & 5Vs of Big Data
• Traditional vs Big Data systems
• Batch vs real-time processing
• Introduction to distributed systems

Topics:

• Distributed data processing
• Spark architecture (Driver, Executors)
• RDD vs DataFrames vs Datasets
• Lazy evaluation
• Tools: Apache Spark

Topics:

• Working with Databricks
• Workspace overview
• Clusters & notebooks
• Databricks Runtime
• Job scheduling

Topics:

• Data transformation
• DataFrames API
• Filtering, grouping, joins
• Aggregations
• Tools: PySpark

Topics:

• Complex data processing
• Window functions
• UDFs
• Handling nested data (JSON)

Topics:

• Modern data storage
• What is Delta Lake
• ACID transactions
• Time travel
• Data versioning

Topics:

• Loading data into Databricks
• Batch ingestion
• Streaming ingestion basics
• File formats (CSV, JSON, Parquet)

Topics:

• Building pipelines
• Extract → Transform → Load
• Data validation
• Error handling

Topics:

• Real-time processing
• Structured Streaming
• Streaming sources & sinks

Topics:

• Cloud-based data engineering
• Integration with Amazon S3, Azure Data Lake Storage
• Mounting storage in Databricks

Topics:

• Improving Spark jobs
• Partitioning & caching
• Broadcast joins
• Query optimization

Topics:

• Data security
• Access control
• Data governance
• Role-based access

Topics:

• Production workflows
• Databricks jobs
• Workflow orchestration basics

Project Focus:

• Process large datasets using PySpark

Project Focus:

• Build ACID-compliant storage

Project Focus:

• Real-time data processing

Project Focus:

• Ingestion → Transformation → Storage → Analytics

Scenarios:

• Build scalable ETL pipelines
• Handle TB-level data
• Optimize Spark jobs
• Manage data lakes
• Process real-time data streams

Skills Gained:

• Spark & PySpark
• Databricks platform
• Delta Lake
• ETL pipelines

Topics:

• Introduction to cloud computing
• Role of Data Engineer
• IaaS, PaaS, SaaS
• Batch vs streaming processing
• Data lifecycle
• Architecture basics
• Platforms: Amazon Web Services, Microsoft Azure, Google Cloud Platform

Topics:

• Data storage in cloud
• Object storage vs block storage
• Data lakes
• Storage tiers & lifecycle policies
• Tools: Amazon S3, Azure Data Lake Storage, Google Cloud Storage

Topics:

• Managed data services
• Relational vs NoSQL
• Data warehousing concepts
• Partitioning & indexing
• Tools: Amazon RDS, Azure SQL Database, BigQuery

Topics:

• Data collection pipelines
• Batch ingestion
• Streaming ingestion
• API ingestion
• File-based ingestion
• Tools: AWS Glue, Azure Data Factory

Topics:

• ETL/ELT pipelines
• Data cleaning & transformation
• Schema evolution
• Distributed processing
• Tools: Apache Spark, PySpark

Topics:

• Modern data storage
• Data lake architecture
• Lakehouse concept
• Data partitioning

Topics:

• Distributed systems
• Spark architecture
• Performance tuning
• Handling large datasets

Topics:

• Pipeline automation
• DAGs & scheduling
• Dependency handling
• Tools: Apache Airflow

Topics:

• Streaming pipelines
• Event-driven architecture
• Stream processing basics
• Tools: Apache Kafka

Topics:

• Deployment strategies
• Container basics
• Deploy pipelines using containers
• Tools: Docker

Topics:

• Data protection
• IAM roles & policies
• Encryption
• Data governance

Topics:

• Performance & cost
• Logging & monitoring
• Cost optimization
• Debugging pipelines

Project Focus:

• Multi-source ingestion → transformation → storage

Project Focus:

• Kafka + Spark integration

Project Focus:

• Design and implement scalable storage

Project Focus:

• Full pipeline deployment on cloud

Scenarios:

• Build scalable ETL pipelines
• Handle real-time data streams
• Optimize big data processing
• Manage cloud storage systems

Skills Gained:

• Cloud platforms (AWS/Azure/GCP)
• ETL & pipeline development
• Big data processing
• Workflow orchestration

Topics:

• Evolution of data engineering → AI-driven pipelines
• Traditional ETL vs AI-augmented pipelines
• GenAI use cases in data workflows
• Structured vs unstructured data challenges

Topics:

• Core LLM concepts
• Tokens, embeddings, transformers
• Prompt engineering basics
• API usage with GPT models

Topics:

• Designing effective prompts
• Zero-shot, few-shot, chain-of-thought
• Prompt templates
• Context handling

Topics:

• Knowledge-based AI systems
• RAG architecture
• Data ingestion & chunking
• Embeddings & retrieval
• Tools: FAISS, Pinecone

Topics:

• Preparing data for AI
• Cleaning & normalization
• Handling structured/unstructured data
• Metadata extraction

Topics:

• Intelligent ETL
• AI-based data cleaning
• Schema inference
• Data enrichment

Topics:

• Autonomous AI systems
• What is Agentic AI
• Task planning & execution
• Tool usage by agents

Topics:

• Multi-step automation
• Multi-agent systems
• Workflow orchestration
• Decision-making agents
• Tools: LangChain

Topics:

• Context-aware data retrieval
• Embedding models
• Similarity search
• Indexing strategies

Topics:

• Deploying AI pipelines
• Integration with Amazon Web Services, Microsoft Azure
• Scalable architecture

Topics:

• Responsible AI
• Data privacy
• Hallucination handling
• Access control

Topics:

• Performance tuning
• Latency optimization
• Cost control
• Model evaluation

Project Focus:

• Query enterprise datasets using LLM

Project Focus:

• Automated data cleaning & transformation

Project Focus:

• Self-operating ETL workflow

Project Focus:

• Ingestion → AI Processing → Insights

Scenarios:

• Automating data pipelines using AI agents
• Building chatbot interfaces for data querying
• Creating intelligent data validation systems
• AI-driven anomaly detection

Skills Gained:

• LLM integration
• RAG systems
• Agentic AI workflows
• AI-powered data pipelines

Topics:

• Traditional vs AI-driven pipelines
• ETL/ELT concepts
• Manual vs automated workflows
• Challenges in pipeline automation

Topics:

• AI in data workflows
• LLM capabilities
• Prompt-based automation
• AI decision-making
• Tools: OpenAI

Topics:

• Designing AI instructions
• Few-shot prompting
• Context injection
• Prompt chaining

Topics:

• Context-aware automation
• RAG architecture
• Data retrieval
• Knowledge integration
• Tools: FAISS, Pinecone

Topics:

• Smart ingestion pipelines
• Schema inference
• Data classification
• Metadata extraction

Topics:

• Intelligent ETL
• Data cleaning automation
• Schema mapping
• Data enrichment

Topics:

• Autonomous pipeline systems
• AI agents
• Task planning & execution
• Tool integration
• Tools: LangChain

Topics:

• Smart orchestration
• DAG automation
• Event-driven pipelines
• Decision-based workflows
• Tools: Apache Airflow

Topics:

• Autonomous systems
• Anomaly detection
• Error handling
• Self-recovery

Topics:

• Scalable deployment
• Integration with Amazon Web Services, Microsoft Azure
• Pipeline deployment

Topics:

• Responsible automation
• Data privacy
• Access control
• Bias & hallucination

Topics:

• Efficiency
• Latency optimization
• Cost management
• Resource allocation

Project Focus:

• Automated ingestion + transformation

Project Focus:

• Detect & fix errors automatically

Project Focus:

• Multi-agent pipeline orchestration

Project Focus:

• Fully automated pipeline

Scenarios:

• Automating data validation using AI
• Building pipelines that fix failures automatically
• AI-driven anomaly detection in data
• Intelligent workflow orchestration

Skills Gained:

• AI-driven automation
• Agentic workflows
• Pipeline orchestration
• Self-healing systems

Topics:

• What is MLOps
• ML lifecycle
• Differences between DevOps & MLOps
• Role of Data Engineers in MLOps

Topics:

• Data pipelines for ML
• Data collection & ingestion
• Feature engineering basics
• Data validation
• Tools: Pandas

Topics:

• ML fundamentals
• Supervised vs unsupervised learning
• Model training workflow
• Evaluation metrics
• Tools: Scikit-learn

Topics:

• Managing ML experiments
• Version control for models
• Tracking experiments
• Reproducibility
• Tools: MLflow

Topics:

• Automation of ML workflows
• CI/CD concepts
• Pipeline automation
• Model retraining pipelines
• Tools: Jenkins, Git

Topics:

• Serving ML models
• REST APIs for ML
• Batch vs real-time deployment
• Containerization
• Tools: Docker

Topics:

• Scalable deployment
• Deploying on Amazon Web Services, Microsoft Azure
• Serverless vs container-based

Topics:

• Pipeline scheduling
• DAGs
• Job scheduling
• Tools: Apache Airflow

Topics:

• Production monitoring
• Model performance tracking
• Data drift detection
• Logging & alerts

Topics:

• Safe deployment
• Access control
• Model governance
• Compliance

Topics:

• Performance tuning
• Scaling ML services
• Cost optimization
• Resource management

Project Focus:

• Data → Training → Deployment

Project Focus:

• Automated training & deployment

Project Focus:

• API-based inference

Project Focus:

• Deploy full ML system

Scenarios:

• Automating model deployment
• Monitoring model performance
• Handling model drift
• Scaling ML systems

Skills Gained:

• MLOps pipelines
• Model deployment
• CI/CD automation
• Monitoring & scaling