AI Fundamentals Part 1: Introduction to AI

What is AI?

Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and learn like humans.

History of AI

The concept of artificial intelligence dates back to ancient civilizations, but the term "AI" was formally coined in 1956 at the Dartmouth Conference.
AI can throw around some intimidating terms, so let’s break down a few key jargon :

Types of AI

AI can be categorized into Narrow AI (or Weak AI), which is designed for specific tasks, and General AI (or Strong AI), with human-like cognitive abilities.

Note: Chat-GPT is not classified as ASI or Strong AI. A sophisticated illustration of Narrow or Weak AI is GPT-4. Please feel free to get this confirmed from ChatGPT directly.

Now, What is Machine Learning?

Machine Learning is a subset of AI that involves the use of data and algorithms to imitate the way humans learn, gradually improving accuracy.

Types of ML

There are three main types of machine learning: supervised, unsupervised, and reinforcement learning.

Supervised Learning

Supervised learning involves learning a function that maps an input to an output based on example input-output pairs. It's like “Teach me what to learn”.

Unsupervised Learning

Unsupervised learning involves learning patterns from untagged data, used for clustering, association, and dimensionality reduction tasks. It's like “I will find what to learn”.

Reinforcement Learning

Reinforcement Learning is a technique that teaches computer programs to make decisions in order to produce the best possible outcomes. It's like “I’ll learn from my mistakes at every step (Hit & Trial!)”.

Now, What is Deep Learning?

Deep Learning is a subset of machine learning that uses neural networks with many layers (deep nets) to learn from data.

Neural Networks

Neural networks are computing systems vaguely inspired by the biological neural networks that constitute animal brains.

Convolutional Neural Networks (CNNs)

CNNs are deep neural networks used primarily to classify images, cluster them by similarity, and perform object recognition within scenes.

Recurrent Neural Networks (RNNs)

RNNs are networks with loops in them, allowing information to persist, making them ideal for sequence prediction tasks.

AI vs. ML vs. DL

AWS Kinesis Overview

Amazon Kinesis is a family of services provided by Amazon Web Services (AWS) for processing and analyzing real-time streaming data at a large scale. Amazon Kinesis is composed of four main services: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.

- Kinesis is a managed “data streaming” service
- Great for application logs, metrics, IoT, clickstreams
- Great for “real-time” big data
- Great for streaming processing frameworks (Spark, NiFi, etc...)
- Data is automatically replicated synchronously to 3 AZ.
- Kinesis Data Streams: low latency streaming ingest at scale.
- Kinesis Data Analytics: perform real-time analytics on streams using SQL.
- Kinesis Data Firehose: load streams into S3, Redshift, ElasticSearch & Splunk.

This is how it Amazon Kinesis looks like, Source can be Click Streams, IoT devices and Metrics & logs; Then do some Amazon Kinesis for processing and analyzing real-time streaming of data at a big scale and then data can be pushed to Amazon S3 and Redshift.

Kinesis Streams Overview

- Streams are divided in ordered Shards / Partitions

Producers --> Shard 1/Shard 2/Shard 3 --> Consumers

- Data retention is 24 hours by default, can go up to 365 days
- Ability to reprocess / replay data
- Multiple applications can consume the same stream
- Real-time processing with scale of throughput
- Once data is inserted in Kinesis, it can’t be deleted (immutability)

Kinesis Producers & Consumers

KINESIS PRODUCERS

- AWS SDK: simple producer.
- Kinesis Producer Library (KPL): batch, compression, retries, C++, Java.
- Kinesis Agent:
- Monitor log files and sends them to Kinesis directly.
- can write to Kinesis Data Streams
AND Kinesis Data Firehose.

KINESIS CONSUMERS

- AWSSDK:simpleconsumer
- Lambda:(throughEventsourcemapping)
- KCL:checkpointing,coordinatedreads