Data Ingestion into Big Data Systems and ETL #BigData #3

Data ingestion, the first layer or step for creating a data pipeline, is also one of the most difficult tasks in the system of Big data. In this layer, data gathered from a large number of sources and formats are moved from the point of origination into a system where the data can be used for further analyzation.

Ingestion of Big data involves the extraction and detection of data from disparate sources. Data ingestion moves data, structured and unstructured, from the point of origination into a system where it is stored and analyzed for further operations. It is the rim of the data pipeline where the data is obtained or imported for immediate use.

Data can be either ingested in real-time or in batches. Real-time data ingestion occurs immediately, however, data is ingested in batches at a periodic interval of time.

Effective data ingestion process starts with prioritizing data sources, validating information, and routing data to the correct destination.



-- TIMESTAMPS --

00:00 Data Ingestion into Big Data Systems and ETL
00:43 Data Ingestion Overview
02:34 Data Ingestion
04:17 Apache Sqoop
06:20 Sqoop and Its Uses
09:24 Sqoop Processing
11:35 Sqoop Import Process
14:01 Sqoop Connectors
18:23 Apache Flume
21:05 Flume Model
23:02 Scalability in Flume
24:36 Components in Flume
27:15 Configuring Flume Components
29:15 Apache Kafka
31:10 Demo Ingesting Twitter Data from Apache Flume into HDFS
35:53 Aggregating User Activity Using Kafka
37:28 kafka data model
40:24 Partitions
42:29 Apache Kafka Architecture
45:30 Producer Side API Example
48:01 Consumer Side API
48:44 Consumer Side API Example
51:20 Kafka connect
52:37 Demo Creating Sample Kafka Data Pipeline using Producer and Consumer
54:50 Key Takeaways


#Data_Ingestion_into_Big_Data_Systems_and_ETL #kafka #flume #sqoop #dataintegration #simplitraining #youtubeshorts #tutorial #bigdatatraining #scala #programming #2023 #artificialintelligence #newtechnology #tunisia #hadoop #spark
#teaching #learning #facts #support #goals #like #nonprofit #career
#educationmatters #technology