Showing posts with label NoSQL. Show all posts
Showing posts with label NoSQL. Show all posts

Monday, 30 January 2023

Amazon Athena Overview

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL.
Use cases : Buisness intelligence / analytics / reporting, analyze, & query VPC Flow logs, ELB logs, CloudTrails, etc.
- Serverless query service to analyse data stored in Amazon S3.
- Use standard SQL language to query files(built on pesto).
- Support CSV, JSON, ORC, Avro and Parquet.
- Pricing $5.00 per TB of data scanned.
- Commonly used with Amazon Quicksight for reporting/dashboards.


Amazon Athena : Performance Improvement

- Use Columnar Data for cost savings (by doing less scan).
- Apache Parquet or ORC is recommended.
- Huge performance improvement.
- Use Glue to convert your data to Parquet or ORC format.
- Compress data for smaller retrievals (bzip2, gzip, Iz4, snappy, zlip...)
- Partition datasets in S3 for easy querying on virtual columns
- s://yourbucketname/pathtotable
/[PARTITION_COLUMN_NAME]=[VALUE]
/[PARTITION_COLUMN_NAME]=[VALUE]
etc..

- Example s3://athenabucket/flight/parquet/year=1999/month=1/day=1/
- Use larger files (>128MB) to minimize overhead.

Amazon Athena : Federated Query


- Allows you to run SQL queries across data stored in relation, non-relational, objects or custom data sources (AWS or on-premises).

- Uses Data Sources Connectors that run on AWS Lambda to run federated queries (like on Cloudwatch logs, DynamoDB, RDS DB, Elasticache etc..)

- Store results back in Amazon S3.


Monday, 23 January 2023

Amazon DocumentDB Overview

We will talk about DocumentDB so we all know about Aurora RDS and Aurora RDS is AWS implementation of Postgresql or MySQL because it has the same API but it's built on AWS and it has some nice benefits to be cloud native.
DocumentDB is the same as Aurora RDS but for MongoDB (Which is NoSQL database).
- Mongodb is used to store, query and index JSON data so thanks to DocumentDB.
- Similar "deplyement concept" as Aurora RDS.
- Fully Managed, highly available with replication across 3 AZ.
- DocumentDB storage automatically grows in increments of 10GB.
- Automatically scales to workloads with millions of requests per seconds.
If using MongoDB on AWS so you pay for what you use in DocumentDB and there is no upfront cost but you need to understand the Architecture of DocumentDB, which is very similar to Aurora in order to understand its pricing.


DocumentDB - Pricing

- Pay as you Go, so pay for what you use. No up front cost.
- On-Demand Instances (pay per second with the minimum of 10 minutes)
- Database I/O: amount of IO used when Read/Write (per million IOs)
- Database Storage (per GB/month)
- Backup Storage (per GB/month)

We have database storage and then we have On Demand instances. We have primary instance and then replica instances that's up to you're going to pay for these on-demand instances, you can pay per second with the minimum of 10 minutes then these instances are going to do a read and write operations against your database storage so you're going to base for database IOs and this going to be counted per million IOs then of course the data is being stored on your database storage and therefore you're going to pay for the derby storage itself in gigabytes per month.

Friday, 28 October 2016

NoSQL Vs RDBMS

Relational Database Management System (RDBMS)

RDBMS Database is a relational database. It is the standard language for relational database management systems.Data is stored in the form of rows and columns in RDBMS. Some common relational database management systems that use SQL are: Oracle, Sybase, Microsoft SQL Server, Access, etc. All these RDBMS follows ACID property.

NoSQL

NoSQL commonly referred to as “Not Only SQL”. With NoSQL, unstructured ,schema less data can be stored in multiple collections and nodes and it does not require fixed table sachems, it supports limited join queries , and we scale it horizontally. Some common NoSQL databases are: MongoDB, Cassandra DB, Neo4j etc and all these NoSQL databases follows CAP theorem

How NoSQL Differs from Relational Databases

NoSQL and RDBMS’s are designed to support different application requirements and typically they co-exist in most enterprises.

Where to use RDBMS and NoSQL:

Now main decision points on when to use RDBMS/NoSQL which include the following:



NoSQL Database Overview

What is NoSQL Database?
NOSQL Database generally refers as "Not Only SQL" or "non-Relational". NoSQL database provides a mechanism for storage and retrieval of data which is modeled in means other than the tabular relations used in relational databases.

Basic Concept and Technique Used in NoSQL

1. Distribution Model: How to distribute data when scaling out, basically two type:- Sharding and Replication.
2. Consistency: NoSQL databases use CAP theorem to define consistency.
3. Data Model: In NoSQL databases are basically categories on the basis how data is actually stored, like- Column wise, Document, Graph, Key/Value Store.

1. Distribution Model :
Because of their architecture differences, NoSQL databases differ on how they support the reading, writing, and distribution of data. Some NoSQL platforms like Cassandra support writes and reads on every node in a cluster and can replicate / synchronize data between many data centers and cloud providers.
1.1. Sharding: Sharding distributes different data across multiple servers, so each server acts as the single source for a subset of data.
1.2. Replication: Replication copies data across multiple servers, so each bit of data can be found in multiple places.
Replication comes in two forms:
- Master-slave replication makes one node the authoritative copy that handles writes while slaves synchronize with the master and may handle reads.
- Peer-to-peer replication allows writes to any node; the nodes coordinate to synchronize their copies of the data.

2. CAP Theorem :
In distribution system, managing Consistency, Availability, Partition Tolerance exist in a mutually dependent relationship. The CAP theorem which states that in any distributed system we can choose only two of consistency, availability or partition tolerance. The relational database is in favour of consistency and availability.

- C onsistency : All nodes have the same data at the same time.
- A vailability : Every request gets a response on success/failure. Every Achieving availability in a distributed system requires that the system remains operational 100% of the time.
- P artition Tolerance : System continues to work despite message loss or partial failure. A system that is partition-tolerant can sustain any amount of network failure that doesn't result in a failure of the entire network.

All the three combinations can be defined as:
CA – data should be consistent between all nodes. As long as all nodes are online, users can read/write from any node and be sure that the data is the same on all nodes.
CP – data is consistent between all nodes and maintains partition tolerance by becoming unavailable when a node goes down.
AP - nodes remain online even if they can’t communicate with each other and will re-sync data once the partition is resolved, but you aren’t guaranteed that all nodes will have the same data (either during or after the partition)


3. Data Model :
Types of NoSQL Databases
, There have been various approaches to classify NoSQL databases, each with different categories and subcategories. Here is a basic classification by data model, with examples:

3.1. Wide Column Stores:
A column of a distributed data store is a NoSQL object of the lowest level in a keyspace. It is a tuple (a key-value pair) consisting of three elements:
- Unique name: Used to reference the column
- Value: The content of the column. It can have different types, like AsciiType, LongType, TimeUUIDType, UTF8Type among others.
- Timestamp: The system timestamp used to determine the valid content.
Accumulo, Cassandra, Druid, HBase, Vertica are the example of Column database.

3.2. Document Stores:
The central concept of a document store is the notion of a "document". While each document-oriented database implementation differs on the details of this definition, in general, they all assume that documents encapsulate and encode data (or information) in some standard formats or encodings. Encodings in use include XML, YAML, and JSON as well as binary forms like BSON.
Apache CouchDB, Clusterpoint, Couchbase, DocumentDB, HyperDex, Lotus Notes, MarkLogic, MongoDB, OrientDB, Qizx, RethinkDB are the example of Document database.

3.3. Key-value Stores:
Key-value (KV) stores use the associative array (also known as a map or dictionary) as their fundamental data model. In this model, data is represented as a collection of key-value pairs, such that each possible key appears at most once in the collection.
Aerospike, CouchDB, Dynamo, FairCom c-treeACE, FoundationDB, HyperDex, MemcacheDB, MUMPS, Oracle NoSQL Database, OrientDB, Redis, Riak, Berkeley DB are the example of Key-Value database.

3.4. Graph Databases:
A graph database is a database that uses graph structures for semantic queries with nodes, edges and properties to represent and store data.
AllegroGraph, InfiniteGraph, MarkLogic, Neo4J, OrientDB, Virtuoso, Stardog are the example of Graph database.