AWS : How to connect S3 Bucket from EC2 Instance which is in Private VPC without Internet access?

Problem:

How to upload the files to S3 bucket from the Win Server (i.e. EC2 Instance) which is in private vpc with no general internet access?

 

Solution:

First of all, as you already know to enable an upload you need to ensure that there is a connectivity between Server and S3 Buckets.

So there are different approaches based upon Setup and Configuration, few options explained as follows:

1. If the EC2 instance and the S3 bucket are in the same AWS region, Then Using VPC Endpoint : You can use VPC Endpoints and associate it with the route table of the EC2 instance's subnet which would allow traffic to the S3 service in that region even if you don't have a NAT or Internet gateway associated with your subnet.

For more about S3 VPC endpoints:
    S3 VPC endpoint in detail:
        [+] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-s3.html
    create a VPC Endpoint:
        [+] https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

2. If the EC2 instance and the S3 bucket are in different AWS regions, Then Using NAT Gateway or Internet Gateway: You have to make use of either NAT Gateway or Internet gateway to allow traffic from your instance to the S3 service in that region.
If you dont want to keep the internet open on the server, you can later deny traffic between your EC2 Instance and S3 Service after copying the backup file on to S3 Bucket.

For more about NAT\Internet Gateways:
    NAT gateways :
     [+] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
    Internet gateways :
     [+] https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html
 

AWS : Relational Database Service (RDS) - Basics Part 4

 ==============================================================
How do you maintain my database ? How can I plan for it?

• Any maintenance that causes downtime (typically only a few times per year) will be scheduled in your maintenance window
• Operating system or Amazon RDS software patches are usually performed without restarting databases
• Database engine upgrades require downtime
    • Minor version upgrades — automatic or manually applied (Automatic not applicable in case of SQL Server, always manual)
    • Major version upgrades — manually applied
    • Version deprecations—three-to six-month notification before scheduled upgrades
• View upcoming maintenanceevents in your AWS Personal Health Dashboard

==============================================================
How am I charged for Amazon RDS ?

• Database instance (instance hours)
    • Combination of Region + instance type + database engine + license (optional)
• Database storage (GB-mo)
    • Can be either provisioned (Amazon EBS) or consumed (Amazon Aurora)
    • Provisioned IOPS (IOPS-Mo) for IO1 storage type
    • Database I/O requests (IOs) for Amazon Aurora and Amazon EBS magnetic-storage types
• Backup storage (GB-mo
    • Size of backups and snapshots stored in Amazon S3
    • No charge for backup storage up to 100% of total database storage
• Data transfer (GB-mo)
    • Uses AWS regional data-transfer pricing

==============================================================
How can I save money on my database?

• Amazon RDS Reserved Instances (RIs) provide a discount over on-demand prices
• Region, instance family, and engine of on-demand usage must match to apply benefit
• Amazon RDS RIs offer size flexibility for open-source and Oracle BYOL engines
• By default, RIs are shared between allaccounts in consolidated billing
• Use the RI utilization report to determinehow your RIs are being used
    • Support for RI coverage reportcoming soon

==============================================================
Can I stop my database when it’s not in use?

• Stop and start a running database instance from the console or AWS CLI
• Available for single-AZ DB instances
• While instance is stopped, you only pay for storage
• Backup retention window is maintained while stopped
• Instances are restarted after 7 days
    • Pending maintenance operations are applied
    • Instances can be stopped again if desired

==============================================================

AWS : Relational Database Service (RDS) - Basics Part 3

 ==============================================================
How do I secure my Amazon RDS database?

• Amazon RDS is designed to be secure by default
• Network isolation with Amazon Virtual Private Cloud (Amazon VPC)
• AWS Identity and Access Management (IAM)-based resource-level permission controls
• Encryption at rest using AWS KMS (all engines) or Oracle/Microsoft TDE
• Use SSL protection for data in transit

==============================================================
What does Amazon VPC provide?

• Places your instance in a private subnet, making it secure from public routes on the Internet
• Database instance IP firewall protection lets you securely control network configuration
• Turn off Public Accessibility in DB instance settings to restrict access outside Amazon VPC
• Use ClassicLink to network with non-VPC resources

==============================================================
How do I grant access to my database?

• Use IAM to control who can perform actions on RDS resources
• Do not use AWS root credentials to manage Amazon RDS resources—you should create an IAM user for everyone, including yourself
• Can use AWS Multi-Factor Authentication (MFA) to provide extra level of protection

==============================================================
How do I encrypt my database?

- Use AWS KMS-based encryption in the AWS console
- No performance penalty for encrypting data
- Centralized access and audit of key activity
- Best practices
    • Encryption cannot be removed from DB instances
    • If source is encrypted, Read Replicas must be encrypted
    • Add encryption to an unencrypted DB instance by encrypting a snapshot copy

==============================================================
How do I monitor my Amazon RDS database ?

- Amazon CloudWatch metrics
    • CPU/Storage/Memory•Swap usage
    • I/O (read and write)
    • Latency (read and write)
    • Throughput (read and write)
    • Replica lag
- Amazon CloudWatch Alarms
    • Similar to on-premises monitoring tools
- Enhanced monitoring for Amazon RDS
    • Access to over 50 CPU, memory, file system, and disk I/O metrics
    • Low as 1-second intervals
-Integration with third-party monitoring tools

==============================================================
How do I improve database performance?

• Introducing Amazon RDS Performance Insights
• Measures DB Load: Average Active Sessions (AAS)
• Identifies database bottlenecks (Top SQL):
    • Easy
    • Powerful
• Identifies source of bottlenecks
• Enables problem discovery
• Adjustable time frame
    • Hour, day, week, and longer
• Coming soon for Amazon EBS-based Amazon RDS engines

==============================================================
Can I know when service events happen?

• Amazon RDS uses Amazon SNS to receive notification when an event occurs
• Notifications can be in any form supported by Amazon SNS (email, a text message, or a call to an HTTP endpoint)
• Six different source types (DB instance, DB parameter group, DB security group, DB snapshot, DB cluster, DB cluster snapshot)
• 17 different event categories (availability, backup, deletion, configuration change, etc.)

==============================================================

AWS : Relational Database Service (RDS) - Basics Part 2

 ==============================================================
How do I ensure database high availability?

• Multi-AZ provides enterprise-grade fault-tolerance solution for production databases
    • Automatic failover
    • Synchronous replication
    • Inexpensive and enabled with one click

==============================================================
What happens during a Multi-AZ failover? How long does it take?

• Each host manages set of Amazon EBS volumes with a full copy of the data
• Instances are monitored by an external observer to maintain consensus over quorum
• Failover initiated by automation or through the Amazon RDS API
• Redirection to the new primary instance is provided through DNS

==============================================================
Why would I use Read Replicas?

• Relieve pressure on your source database with additional read capacity
• Bring data close to your applications in different regions
• Promote a Read Replica to a master for faster recovery in the event of disaster
• Upgrade a Read Replica to a new engine version•Supported for MySQL, MariaDB, and PostgreSQL

==============================================================
When should I use Multi-AZ ?

 • Synchronous replication—highly durable
 • Only primary instance is active at any point in time
 • Backups can be taken from secondary
 • Always in two Availability Zones within a Region•Database engine version upgrades happen on primary
 • Automatic failover when a problem is detected

==============================================================
When should I use Read Replicas ?

 • Asynchronous replication—highly scalable
 • All replicas are active and can be used for read scaling
 • No backups configured by default•Can be within an Availability Zone, cross-AZ, or cross-region
 • Database engine version upgrades independently from source instance
 • Can be manually promoted to a standalone databaseRead ReplicasMulti-AZ

==============================================================
How does Amazon RDS manage backups?

• Two options –automated backups and manual snapshots
• Amazon RDS backups leverage Amazon EBS snapshots stored in Amazon S3
• Transaction logs are stored every 5 minutes in Amazon S3 to support point-in-time recovery (PITR)
• No performance penalty for backups
• Snapshots can be copied across regions or shared with other accounts

==============================================================
When should I use automated backups?

• Specify backup retention window per instance (7-day default)
• Kept until outside of window (35-day maximum) or instance is deleted
• Supports PITR
• Good for disaster recovery

==============================================================
When should I use snapshots?

• Manually created through AWS console, AWS CLI, or Amazon RDS API
• Kept until you delete them
• Restores to saved snapshot
• Use for checkpoint before making large changes, non-production/test environments, final copy before deleting a databaseManual snapshots

==============================================================
How do I restore a backup? Why does it take so long?

- Restoring creates an entirely new database instance
    • Define the instance configuration just like a new instance
    • Will get the default parameter, security, and option groups

- New volumes are hydrated from Amazon S3
    • While the volume is usable immediately, full performance requires the volume to warm up until fully instantiated
    • Migrate to a DB instance class with high I/O capacity
    • Maximize I/O during restore process

==============================================================

AWS : Relational Database Service (RDS) - Basics Part 1

==============================================================
What is Amazon RDS?

• Managed relational database service in the AWS cloud
• Multi-engine support: Amazon Aurora, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server
• Automated provisioning, patching, scaling, replicas, backup/restore
• Easily scales to handle growth•High availability with Multi-AZ and Amazon Aurora

==============================================================

Why use Amazon RDS?

• Lower TCO because RDS manage “the muck”
    • Get more leverage from your teams
    • Focus on the things that differentiate you
• Built-in high availability and cross-region replication across multiple data centers
• Even a small startup can leverage multiple data centers to design highly available apps with over 99.95% availability

==============================================================
Which RDS engine should I use?

Amazon EBS-based Storage :
- Commercial : SQL Server and Oracle
- Open Source : MySQL , PostgreSQL and MariaDB
Aurora Storage System :
- AWS Cloud Native : MySQL CompatiblePostgreSQL Compatible

==============================================================
Which instance type should I choose?

- T2 Family
    • Burstable instances
    • 1 vCPU/1 GB RAM > 8  vCPU 32 GB RAM
    • Moderate networking performance
    • Good for smaller or variable workloads
    • Monitor CPU credit metrics in Amazon CloudWatch
    • T2.micro is eligible for free tier

- M3/M4 Family
    • General-purpose instances
    • 2 vCPU/8 GiB RAM > 64 vCPU 256 GiB RAM
    • High-performance networking
    • Good for running CPU intensive workloads (e.g., WordPress)

- R3/R4 Family
    • Memory-optimized instances
    • 2 vCPU/16 GiB RAM > 64 vCPU 488 GiB RAM
    • High-performance networking
    • Good for query-intensive workloads or high connection counts
==============================================================
Configuring DB Instance Class Processor ?

Modify processor features to optimize DB instance for specific workloads:
• Number of CPU Cores –Optimize licensing costs for instance with sufficient RAM for memory-intensive workloads, but fewer CPU cores
• Threads per Core –Disable Hyper-Threading for, e.g. HPC workloads

Changes can be applied when you:
• Create a DB instance
• Modify a DB instance
• Restore a DB instance from a snapshot
• Restore a DB instance to a point in time
==============================================================
Which AWS EBS storage type should I choose?

- General purpose (GP2)
    • SSD storage
    • Maximum of 16 TB
    • Leverages Amazon EBS Elastic Volumes
    • IOPS determined by volume size
    • Minimum of 100 IOPS (below 33.33GiB)
    • Bursts to 3,000 IOPS (applicable below 1.3 TB)
    • Baseline of 16,000 IOPS (at 3.3 TB and above)
    • Affordable performance

- Provisioned IOPS (IO1)
    • SSD storage•Maximum of 16 TB
    • Leverages Amazon EBS Elastic Volumes
    • Maximum of 40K IOPS (32K on SQL Server)
    • Delivers within 10% of the IOPS performance 99.9% of the time
    • High performance and consistency

- Magnetic
    • Magnetic storage
    • Maximum of 1 TB
    • Supported for legacy databases

- General purpose (GP3), is coming soon.
==============================================================
How do I decide between GP2 and IO1 AWS EBS Storage Type? Why am I not seeing less IOPS or only 30 K IOPS?

• GP2 is a great choice, but be aware of burst credits on volumes < 1 TB
    • Hitting credit-depletion results in IOPS drop—latency and queue depth metrics will spike until credits are replenished
    • Monitor BurstBalance to see percent of burst-bucket I/O credits available
    • Monitor read/write IOPS to see if average IOPS is greater than the baseline

==============================================================
How do I scale my database instance? Will there be downtime ?

• Scale compute/memory vertically up or down
    • Handle higher load to grow over time
    • Lower usage to control costs
    • New host is attached to existing storage with minimal downtime

• Scale up Amazon EBS storage (now up to 16 TB)
    • Amazon EBS engines now support Elastic Volumes for fast scaling (now including SQL Server)
    • No downtime for storage scaling
    • Initial scaling operation may take longer, because storage is reconfigured on older instances
    • Can re-provision IOPS on the fly

==============================================================

Cloud Services Comparison: AWS Vs Azure Vs Google: Pricing Model

Cloud Services Comparison: AWS Vs Azure Vs Google:

Pricing Model The pricing model of these cloud service is to pay as you go; it means pay on the basis of usage.

Considering AWS Vs Azure Vs Google:
Amazon charges on an hourly basis, While Azure and Google charge on the minute basis.
Also can choose to make upfront payments - prepaid or monthly payments i.e postpaid 

Pricing Model
	AWS	Azure	Google
Pricing	Per Hour - Rounded Up	Per Minute - Rounded Up Commitments (Prepaid or Monthly)	Per Minute - Rounded Up (Minimun 10 Minutes)
Model	On-Demand Spot Reserved	On-Demand Short Term Commitments (Pre-Paid or Monthly)	On-Demand - Sustained Use

Cloud Services Comparison: AWS Vs Azure Vs Google: Networking and Content Delivery & Security

 Cloud Services Comparison: AWS Vs Azure Vs Google: Networking and Content Delivery

Networking and Content Delivery
Services	AWS	Azure	Google
Load Balancing Configuration	Elastic Load Balancing	Load Balancer Application Gateway	Cloud Load Balancing
Global Content Delivery Networks	CloudFront	Content Delivery Network	Cloud Interconnect
Manage DNS Name and Records	Route 53	Traffic Manager Azure DNS	Google Cloud DNS
Cross Premises Connectivity	API Gateway	VPN Gateway	Cloud VPN
Virtual Networking	Virtual Private Cloud	Virtual Network	Subnet
Dedicated Private Network Connection	Direct Connect	Express Route	-

 
Cloud Services Comparison: AWS Vs Azure Vs Google: Security

Security
Services	AWS	Azure	Google
Authentication and Authorization	Identity and Access Management (IAM)	Active Directory Active Directory Permium	Cloud IAM Cloud Identity-Aware Proxy
Protection with Data Encryption	Key Management Service	Storage Service Encryption	-
Firewall	Web Application Firewall	Application Gateway	-
Identity Management	Cognito	Active Directory B2C	-
Cloud Services with Protection	Shield	DDoS Protection Service	-

 

Cloud Services Comparison: AWS Vs Azure Vs Google: Development Tools

 Cloud Services Comparison: AWS Vs Azure Vs Google: Development Tools

Development Tools
Services	AWS	Azure	Google
Media Transcoding	Elastic Transcoder	Media Services	-
App Testing	Device Farm	DevTest Labs (Backend)	Cloud Test Lab
DevOps	Code Build	Visual Studio Team Services	-
Developer Tools	Developer Tools	Developer Tools	-
Git Repositories	AWS Source Repositories	Azure Source Repositories	Cloud Source Repositories

Cloud Services Comparison: AWS Vs Azure Vs Google: Management and Monitoring

Cloud Services Comparison: AWS VS Azure VS Google : Management and Monitoring

Management and Monitoring
Services	AWS	Azure	Google
Billing	Billing API	Billing API	Cloud Billing API
Cloud Advisor Capabilities	CloudWatch X-Ray Management Console	Portal Monitor Application Insights	Stackdriver Monitoring CloudShell Debugger Trace Error Reporting
DevOps Deployment Orchestration	OpsWorks (Chef-based) Cloud Formation	Automation Resource Manager VM Extensions	Cloud Deployment Manager
Cloud Resources Management and Monitoring	Trusted Advisor	Advisor	Cloud Platform Security

 

 

Cloud Services Comparison: AWS VS Azure VS Google : Database Services

Cloud Services Comparison: AWS VS Azure VS Google : Database Services

 

DATABASE
Services	AWS	Azure	Google
Caching	ElasticCache	RedisCache	CloudCDN
Block Storage	Elastic Blob Store (EBS)	Page Blobs	Persistent Disks
Object Storage	Simple Storage Services (S3)	Blobs and Files	Google Cloud Storage Block
Relational Database as a Service	AWS RDS	SQL Database Database for MySQL Database for PostgreSQL	Google Cloud SQL Cloud Spanner
NoSQL (Indexed)	DynamoDB	CosmosDB	Cloud Datastore Cloud BigTable
NoSQL (Key Value)	DynamoDB SimpleDB	Table Storage	Cloud Datastore
Managed Data warehouse	RedShift	SQL Data Warehouse	-
Database Migration Solution	Database Migration Service (DMS)	Database Migration Service	-

Cloud Services Comparison: AWS VS Azure VS Google : Compute and Storage Services

The cloud services market is governed by Amazon Web Services (AWS), Microsoft Azure, and Google Cloud platform cloud services providers. 

Cloud Services Comparison: AWS VS Azure VS Google : Compute and Storage Services Comparison

COMPUTE
Services	AWS	Azure	Google
Virtual Server	Elastic Compute Cloud (EC2)	Virtual Machines	Compute Engine
Platform as a Service (PaaS)	Elastic Beanstalk	Cloud Services	Google App Engine
Kubernetes Containers	Elastic Container Service (ECS) Elastic Kubernetes Service (EKS)	Container Service Azure Kubernetes Service (AKS)	Kubernetes Engine
Docker Container Resgistry	EC2 Container Registry (ECR)	Container Registry	Container Registry
Integrated Systems and Run Backend Logic Processes	Lambda	Functions Event Grid Web Jobs	Cloud Functions (Beta)
Automatic Scale Instances	Auto Scaling	Azure App Service Scale Capability (PaaS) Auto Scaling Virtual Machine Scale Sets	Instance Group
STORAGE
Object Storage Services for use Cases	Simple Storage Services (S3)	Storage (Block Blob)	Cloud Storage
Archive Storage	S3 Infrequent Access S3 Glacier S3 Glacier Deep Archive	Azure Storage (Cool) Azure Storage (Archive)	Nearline Coldline
Hybrid Storage	Storage Gateway	StorSimple	Egnyte Sync
Bulk Data Transfer Solutions	Import/Export Disk Snowball Edge SnowMobile	Import/Export Azure Data Box	Storage Transfer Service
Backup Solutions	Object Storage Cold Archive Storage Storage Gateway	Backup	-
Automatic Protection Disaster Recovery	Disaster Recovery	Azure Site Recovery (ASR)	-