Monday, 22 January 2024

AI Fundamentals Part 4 : Introduction to LLMs(Large Language Models)


What is Large Language Models (LLMs)?


Artificial Intelligence (AI) is the simulation of human intelligence in machines that are programmed to think and learn. Large Language Models (LLMs) are a type of AI that processes, understands, and generates human-like text based on vast datasets.

History of Large Language Models



- LLMs have a fascinating history that dates back to the 1960s with the creation of the first-ever chatbot, Eliza.
- Over the years, several significant innovations have propelled the field of LLMs forward. One such innovation was the introduction of Long Short-Term Memory (LSTM) networks in 1997, which allowed for the creation of deeper and more complex neural networks capable of handling more significant amounts of data.
- Another pivotal moment came with Stanford’s CoreNLP suite, which was introduced in 2010. The suite provided a set of tools and algorithms that helped researchers tackle complex NLP tasks such as sentiment analysis and named entity recognition.
- In 2011, Google Brain was launched, providing researchers with access to powerful computing resources and data sets along with advanced features such as word embeddings, allowing NLP systems to better understand the context of words.
- Google Brain’s work paved the way for massive advancements in the field, such as the introduction of Transformer models in 2017.
- The transformer architecture enabled the creation of larger and more sophisticated LLMs such as OpenAI’s GPT-3 (Generative Pre-Trained Transformer) which served as the foundation for ChatGPT and a legion of other incredible AI-driven applications.

In recent years, solutions such as Hugging Face and BARD have also contributed significantly to the advancement of LLMs by creating user-friendly frameworks and tools that enable researchers and developers to build their own LLMs.

Types of Large Language Models(LLMs)


Three major categories can be used to categorize large language models (LLMs): multimodal, fine-tuning, and pre-training models.
- Pre-training models: with large data sets, such as GPT-3/GPT-3.5, T5, and XLNet, are able to acquire a broad knowledge base of linguistic patterns and structures. These models are quite good at producing grammatically sound and coherent content on a wide range of subjects. They serve as a foundation for additional instruction and task-specific fine-tuning.
- Fine-tuning models: models such as ALBERT, RoBERTa, and BERT are trained beforehand on a big dataset and then adjusted for a particular job on a smaller dataset. For applications such as text classification, question answering, and sentiment analysis, these models work incredibly well. When task-specific language models are required in industrial applications, they are frequently employed.
- Multimodal models: to produce more reliable language models, models such as CLIP and DALL-E mix text with additional modalities like images or video. These models are able to provide text descriptions of images or even produce images from textual descriptions since they are able to comprehend the links between images and words.



Applications of Large Language Models


Large Language Models (LLMs) have demonstrated remarkable performance in a wide range of natural language processing (NLP) tasks like :
- Content Generation: Articles, stories, code.
- Translation: Converting text from one language to another.
- Customer Service: Automated responses and support bots.

Sunday, 21 January 2024

AI Fundamentals Part 3 : Generative AI (GenAI)


What is Generative AI?

Generative AI, also referred to as "GenAI" is a technology that lets users enter different kinds of prompts to create new media, including text, photos, movies, sounds, code, and 3D designs.
As generative AI is trained on additional data, it develops. It uses artificial intelligence (AI) models and algorithms that are trained on big, unlabeled data sets, which demand sophisticated mathematics and a lot of processing power to produce. These sets of data teach the AI to forecast results in ways that resemble actions or creations made by humans.

How does generative AI work?

Neural networks are used by generative AI models to find patterns in current data in order to produce new material. Organizations can build foundation models from sizable, unlabeled data sets by training them on unsupervised and semi-supervised learning methodologies. This effectively creates a foundation for AI systems to execute tasks.
Foundation models, which power tools like ChatGPT, DALL-E, and more, include LLMs, GANs, VAEs, and Multimodal. ChatGPT allows users to create stories in response to prompts by utilizing data from GPT-3. Stable Diffusion is another foundation model that lets people create realistic visuals from text input.

How to Evaluate Generative AI Models?

Three essential elements are needed for a generative AI model to be successful:
- Quality: Having high-quality generation outputs is essential, particularly for apps that engage directly with consumers. For instance, low-quality speech is hard to understand when it comes to speech creation. Similar to this, while creating photographs, the intended results should be identical to natural photos in terms of appearance.
- Diversity: Without compromising the quality of its generation, a good generative model captures the minority modes in its data distribution. As a result, the trained models' undesirable biases are lessened.
- Speed: In order to be used in content development workflows, many interactive apps, including real-time image editing, require quick generating.


Generative AI use cases


These use cases could inspire you to think of innovative ways that generative AI can help you and your company after you've determined which AI generator best meets your requirements.
- Writing or refining material by creating a draft copy with a particular length or style.
- Including subtitles or translating movies, TV shows, and other stuff into several languages.
- Writing summaries for term papers, resumes, briefings, and more.
- Getting a generic code to work with and modify.
- Condensing reports, emails, and articles.
- Enhancing videos for explanations or demonstrations.
- Composing in a certain style or tone.

AI Fundamentals Part 2 : Introduction to NLP (Natural Language Processing)


What is Natural Language Processing (NLP)?


Natural Language Processing (NLP) is a branch of Artificial Intelligence (AI) that helps computers understand, interpret and manipulate human language.

NLP involves the application of computational techniques to the analysis and synthesis of natural language and speech.

To analyze natural language and determine the true meaning of text or audio data, it combines machine learning, statistical, and deep learning models with computational linguistics—ruled-based modeling of natural language. It even comprehends the sentiment and intention of the writer or speaker.

NLP processes text and voice data, extracts meaning, determines intent and sentiment, and forms an input or response using computational linguistics, artificial intelligence, and machine learning. Since humans communicate in binary (1 and 0), computers are unable to comprehend or interpret text or words in the same way. As a result, NLP gives computers the ability to comprehend, mimic, and react to people intelligently.

Why is Natural Language Processing important?

Natural language processing is crucial because it aids in the analysis and interpretation of enormous amounts of data. It assists in processing both text and voice data, comprehends intents and feelings, and even aids in the extraction of important insights from the data.

The significance of natural language processing (NLP) lies in its ability to organize unstructured data and lessen ambiguity in natural language. It even assists companies in better organizing their data for their own benefit.

How does Natural Language Processing (NLP) work?

NLP tools use text vectorization to turn text into a machine-readable format. Machine learning algorithms are then fed training data and predicted outputs (tags) to teach machines to associate a given input with its associated output. Machines utilize statistical analysis techniques to create their own "knowledge bank" and identify which qualities best describe the texts before predicting outcomes for unknown data.

NLP software carries out three primary tasks:
- It starts by dissecting the language (text or voice) into its basic components.
- It then attempts to determine the relationships between these pieces of information.
- Lastly, it tries to explain these relationships.

Each of these operations, of course, involves a great deal more stages. Language proficiency is necessary, as well as expertise in algorithms, statistics, and programming.

What are the applications of Natural Language Processing?


- Sentiment analysis is the use of NLP techniques to extract subjective information from text, often to determine the attitude of a speaker or writer.
- Machine translation involves the use of software to translate text or speech from one language to another.
- Chatbots are AI programs that can conduct conversations with human users through text or voice interfaces, often using NLP.

Wednesday, 10 January 2024

AI Fundamentals Part 1: Introduction to AI


What is AI?

Artificial Intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and learn like humans.

History of AI

The concept of artificial intelligence dates back to ancient civilizations, but the term "AI" was formally coined in 1956 at the Dartmouth Conference.
AI can throw around some intimidating terms, so let’s break down a few key jargon :

Types of AI

AI can be categorized into Narrow AI (or Weak AI), which is designed for specific tasks, and General AI (or Strong AI), with human-like cognitive abilities.
Note: Chat-GPT is not classified as ASI or Strong AI. A sophisticated illustration of Narrow or Weak AI is GPT-4. Please feel free to get this confirmed from ChatGPT directly.

Now, What is Machine Learning?


Machine Learning is a subset of AI that involves the use of data and algorithms to imitate the way humans learn, gradually improving accuracy.

Types of ML

There are three main types of machine learning: supervised, unsupervised, and reinforcement learning.

Supervised Learning

Supervised learning involves learning a function that maps an input to an output based on example input-output pairs. It's like “Teach me what to learn”.

Unsupervised Learning

Unsupervised learning involves learning patterns from untagged data, used for clustering, association, and dimensionality reduction tasks. It's like “I will find what to learn”.

Reinforcement Learning

Reinforcement Learning is a technique that teaches computer programs to make decisions in order to produce the best possible outcomes. It's like “I’ll learn from my mistakes at every step (Hit & Trial!)”.

Now, What is Deep Learning?


Deep Learning is a subset of machine learning that uses neural networks with many layers (deep nets) to learn from data.

Neural Networks

Neural networks are computing systems vaguely inspired by the biological neural networks that constitute animal brains.

Convolutional Neural Networks (CNNs)

CNNs are deep neural networks used primarily to classify images, cluster them by similarity, and perform object recognition within scenes.

Recurrent Neural Networks (RNNs)

RNNs are networks with loops in them, allowing information to persist, making them ideal for sequence prediction tasks.

AI vs. ML vs. DL

Wednesday, 1 February 2023

AWS Kinesis Overview

Amazon Kinesis is a family of services provided by Amazon Web Services (AWS) for processing and analyzing real-time streaming data at a large scale. Amazon Kinesis is composed of four main services: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams.
- Kinesis is a managed “data streaming” service
- Great for application logs, metrics, IoT, clickstreams
- Great for “real-time” big data
- Great for streaming processing frameworks (Spark, NiFi, etc...)
- Data is automatically replicated synchronously to 3 AZ.
- Kinesis Data Streams: low latency streaming ingest at scale.
- Kinesis Data Analytics: perform real-time analytics on streams using SQL.
- Kinesis Data Firehose: load streams into S3, Redshift, ElasticSearch & Splunk.
This is how it Amazon Kinesis looks like, Source can be Click Streams, IoT devices and Metrics & logs; Then do some Amazon Kinesis for processing and analyzing real-time streaming of data at a big scale and then data can be pushed to Amazon S3 and Redshift.

Kinesis Streams Overview

- Streams are divided in ordered Shards / Partitions

Producers --> Shard 1/Shard 2/Shard 3 --> Consumers

- Data retention is 24 hours by default, can go up to 365 days
- Ability to reprocess / replay data
- Multiple applications can consume the same stream
- Real-time processing with scale of throughput
- Once data is inserted in Kinesis, it can’t be deleted (immutability)

Kinesis Producers & Consumers


KINESIS PRODUCERS
- AWS SDK: simple producer.
- Kinesis Producer Library (KPL): batch, compression, retries, C++, Java.
- Kinesis Agent:
- Monitor log files and sends them to Kinesis directly.
- can write to Kinesis Data Streams
AND Kinesis Data Firehose.
KINESIS CONSUMERS
- AWSSDK:simpleconsumer
- Lambda:(throughEventsourcemapping)
- KCL:checkpointing,coordinatedreads

Monday, 30 January 2023

Amazon Athena Overview

Amazon Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL.
Use cases : Buisness intelligence / analytics / reporting, analyze, & query VPC Flow logs, ELB logs, CloudTrails, etc.
- Serverless query service to analyse data stored in Amazon S3.
- Use standard SQL language to query files(built on pesto).
- Support CSV, JSON, ORC, Avro and Parquet.
- Pricing $5.00 per TB of data scanned.
- Commonly used with Amazon Quicksight for reporting/dashboards.


Amazon Athena : Performance Improvement

- Use Columnar Data for cost savings (by doing less scan).
- Apache Parquet or ORC is recommended.
- Huge performance improvement.
- Use Glue to convert your data to Parquet or ORC format.
- Compress data for smaller retrievals (bzip2, gzip, Iz4, snappy, zlip...)
- Partition datasets in S3 for easy querying on virtual columns
- s://yourbucketname/pathtotable
/[PARTITION_COLUMN_NAME]=[VALUE]
/[PARTITION_COLUMN_NAME]=[VALUE]
etc..

- Example s3://athenabucket/flight/parquet/year=1999/month=1/day=1/
- Use larger files (>128MB) to minimize overhead.

Amazon Athena : Federated Query


- Allows you to run SQL queries across data stored in relation, non-relational, objects or custom data sources (AWS or on-premises).

- Uses Data Sources Connectors that run on AWS Lambda to run federated queries (like on Cloudwatch logs, DynamoDB, RDS DB, Elasticache etc..)

- Store results back in Amazon S3.


Monday, 23 January 2023

Amazon DocumentDB Overview

We will talk about DocumentDB so we all know about Aurora RDS and Aurora RDS is AWS implementation of Postgresql or MySQL because it has the same API but it's built on AWS and it has some nice benefits to be cloud native.
DocumentDB is the same as Aurora RDS but for MongoDB (Which is NoSQL database).
- Mongodb is used to store, query and index JSON data so thanks to DocumentDB.
- Similar "deplyement concept" as Aurora RDS.
- Fully Managed, highly available with replication across 3 AZ.
- DocumentDB storage automatically grows in increments of 10GB.
- Automatically scales to workloads with millions of requests per seconds.
If using MongoDB on AWS so you pay for what you use in DocumentDB and there is no upfront cost but you need to understand the Architecture of DocumentDB, which is very similar to Aurora in order to understand its pricing.


DocumentDB - Pricing

- Pay as you Go, so pay for what you use. No up front cost.
- On-Demand Instances (pay per second with the minimum of 10 minutes)
- Database I/O: amount of IO used when Read/Write (per million IOs)
- Database Storage (per GB/month)
- Backup Storage (per GB/month)

We have database storage and then we have On Demand instances. We have primary instance and then replica instances that's up to you're going to pay for these on-demand instances, you can pay per second with the minimum of 10 minutes then these instances are going to do a read and write operations against your database storage so you're going to base for database IOs and this going to be counted per million IOs then of course the data is being stored on your database storage and therefore you're going to pay for the derby storage itself in gigabytes per month.

Monday, 9 January 2023

Amazon Timestream Overview

As name indicates, Amazon Timestream is a time series database so it's fully managed, fast, scalable, serverless time series database. Use cases for timestream would be to have an IoT application, operational applications, real-time Analytics or everything related to a Time series database.
- Automatically scale up/down to scale capacity.
- Store and analyse trillions of events per day.
- 1000s times faster & 1/10th the cost of relation databases.
- Schedule queries, multi-measure records and SQL Compatibility.
- Data storage tiering : Recent data kept in memory and historical data kept in a cost-optimised storage.
- Built-in time series analytics functions (helps you identify patterns in your data in real-time.
- Supports Encryption in-transit and at rest.

Amazon Timestream Architecture


Timeseries database can receive data from AWS IoT so internet of things, the Kineses Data Streams through them that can receive data as well, Prometheus, Telegraf, their integrations for that case the streams as well through conditions data analytics for Apache link can receive data into Amazon time stream and Amazon msk as well through the same process.

In terms of what can connect to do timesstream where can build a dashboards using Amazon Quicksight and machine learning using Amazon SageMaker, can connect to grafana or because there is a standard jdbc connection into your database and the application that is compatible with jdbc and SQL can leverage Amazon timestream.

Tuesday, 3 January 2023

Amazon Redshift Overview

Amazon Redshift is a data warehouse product which forms part of the larger cloud-computing platform Amazon Web Services.
- Redshift is based on PostgreSQL, but it's not used for OLTP. 
- It's OLAP - Online Analytical Processing (analysis and data warehousing).
- Built on top of technology from the massive parallel processing data warehouse company ParAccel, to handle large scale data sets and database migrations.
- 10X better performance than other data warehouses, Scale to PBs of data.
- Columnar storage of data (instead of row based data).
- Pay as you go based on the instances provisioned.
- Can use SQL interface for performing the queries.
- BI tools such as AWS QuickSight or Tableau integrate with it.
- How to load data into Redshift?
Data is loaded from Amazon S3(using COPY command), Kinesis Firehose (To load data near real-time), DynamoDB, DMS (Data Migration Service)...
- Based on node type : up to 100+ nodes, each node can have up to 16 TB of storage space.
- Can provision multiple nodes, with Multi-AZ only for some clusters.
- There are two types of nodes in Redshit :
- Leader Node : For query planning and result aggregation from compute node.
- Compute Node : For performing the queries, and send result to leader node.
- As Redshit is a managed service; so we only get Backup & Restore, Security VPC enhancements /IAM (for accessing cluster)/KMS (for encryption), Monitoring using Cloudwatch.
- Redshift Enhanced VPC routing: COPY / UNLOAD goes through VPC (for better performance and lower cost).
- Redshift is provisioned, so it's worth it when you have a sustained usage (use Athena if the queries are sporadic instead).

Redshift - Snapshot & DR

- Snapshots are point-in-time backups of your cluster, stored internally in S3.
- Snapshots are incremental (Only what has changed in your redshift cluster will be saved).
- You can restore a snapshot into a new cluster (You've to create new cluster to restore the data).
- There are two types of snapshots :
- Automated Snapshot : Happens every 8 hours or every 5 GB of data change in the cluster or on a schedule you set. And set retention ( for example, like for 30 Days and after 30 days snapshot will be automatically deleted).
- Manual Snapshot : On-demand snapshot, retained until you delete it.
It is very similar, how RDS works. But one cool thing in Redshift, you can configure Amazon Redshift to automatically copy snapshots(automated or manual) of a cluster to another AWS Region and this is very useful to setup disaster recovery for your Redshift cluster.

Cross-Region Snapshot Copy for an KMS-Encrypted Redshift Cluster

How to copy a cross-regions snapshot for kms encrypted redshift snapshots?

You have the source and your snapshot is in your source region it's encrypted using the KMS Key A, and you want to copy it into destination came as KMS key B so what we need to do is call a redshift "snapshot copy grant" and that it will allow redshift the service to perform encryption operations in the destination region when this is done then you can copy your snapshot from the original one into the region 2 and then it will be encrypted by redshift with the correct kms key so the the really magical thing here that happens is that you need to have and create a snapshot to copy.

Redshift Spectrum

- Query data that is already in S3 without loading it. - Must have a Redshift cluster available to start the query. - The query is then submitted to thousand of Redshift Specturm nodes.
How does that work?
Here, our existing redshift cluster has a leader node and has a bunch of compute nodes then if we do a query on data on Amazon S3 that will look like above image then what's going to happen is that redshift is going to spin up a lot of virtual spectrum nodes they will do the computation and the query over the data set in Amazon S3 and then once the result is there is going to be sent back to the compute nose for aggregation and then will be rolled back into the leader node.

Redshift WorkLoad Management (WLM)

- It enables you to flexibly manage queries priorities within workloads.
- Use case for this is to prevent short files from inquiries from getting stuck behind long running queries.
- You can define multiple query queues you get a very simple super user queue and user defined queue and so on and then,
- Route your query to the proper queue at runtime.
Lets take an example, here in Amazon redshift and have three queues. One is the Super User Queue, the short running queue and the long running queue the names are pretty explicits and say we have two kinds of users we have Admin and User.


Now, they want their queries to be done as soon as possible right so they will do system queries and then these queries will go directly into the Super User Queue.
And this will have priority the user may have some short-running queries and these we want to have into their short-running query queue this way we are sure that it will just be a lot of short-running queries.
But if the user is submitting a longer running query and we're not gonna take a lot of time then we should send it directly into the long running queue to make sure not to block the short running queries.
So we have two kind of work load management :
- Automatic WLM : where the cues and resources are managed by shift.
- Manual WLM : which accusing resources are managed by you (i.e. the user).

Redshift Concurrency Scaling Cluster

- It enables you to provide closely fast performance with virtually and limited number of users and queries.
- Redshift automatically adds additional cluster capacity on the fly so this additional concert capacity is called Concurrencies Scaling Cluster and allows you to process an increase in request.
- Ability to decide which queries sent to the Concurrencies Scaling Cluster using WLM.

So let's say in the below image, we have a normal version cluster with a few nodes under and then this sustains you know a bunch of users that we have normally let's say tomorrow we get a lot more users submitting queries into Amazon redshift then automatically the concurrency scaling cluster will add those automatically to accommodate for these number of users.
Now we get the ability to decide which queries get sent to the concurrency is getting cluster using the WLM feature we just saw in previous discussion and this feature is charged per second.

Choosing the Right Cloud Database Service offered by cloud providers : AWS, Azure and GCP


The majority of cloud providers started off treating databases as application that could be run on general-purpose compute instances. However, they quickly started adding higher-level application services to their IaaS offerings. Cloud databases have grown to be a crucial area of technological advancement, with manufacturers battling for customers by introducing several database kinds.

Architects, administrators, and developers must be aware of the various database options available because the IT industry is currently experiencing a significant displacement of data-centric workloads to the cloud.
Ensure that the IT team (including DBA and App Team) choose the cloud database provider that best suits the requirements of your business by following the below tips :

1. Become familiar with the database's options:

The deployment architecture and the database requirements are the two main factors to take into account while selecting cloud database services. Each element will include parameters specific to a given application.

The following are the variables that influence cloud database selection:
- Deployment environment: on-site, in a public cloud or clouds, or a combination of hosted and private resources.
- Database manager: specifically, whether the database will be self-managed or a cloud service (DBaaS).
- Database type: including relational SQL (RDBS), distributed SQL, NoSQL variants like a column (key-value) store, wide column or document DB, cache, graph, or various specialty databases like time series and quantum ledger.
- Availability and resilience requirements: these will dictate whether many replicated and synchronized instances are required.
- Performance and capacity requirements: these include transactions per second, query latency, and scalability of both database throughput and capacity.
- Flexibility: defining precise version criteria and the degree of authority required for an IT staff to manage more intricate elements like database setup and configuration.

IT teams including DBA's and Application team should investigate the spectrum of database services offered by AWS, Microsoft Azure, and Google Cloud Platform to suit their individual requirements.

2. Examine cloud trade-offs:

Cloud database services are more cost-effective for businesses looking to cut costs than self-managed internal database systems because they do not require hardware, system administration, or any DBA overhead. Using cloud goods built on open source or in-house produced software, which does not require license costs, also increases the savings for businesses.

Cloud database services work well for businesses that:
- Desire the simplest, fastest implementation possible.
- Desire access to a large selection of database products without having to set up and maintain each one.
- Reluctant to oversee the software.
- Require maximum expandability.
- Desire high availability through a number of widely separated instances.
- Recognize the benefits of the cloud's intrinsic security and its shared security paradigm; additionally, you don't have to micromanage every database option.

In general, cloud users need to understand the limitations on the capacity and performance of specific database instances. Nonetheless, problems may frequently be handled using a distributed architecture that takes advantage of networking and replication capabilities included in cloud solutions.

3. Analyze your consumption and business:

Consider some of the following aspects while making the best choice for the workloads at your organisation.
First, database and cloud infrastructure products are better suited for new applications than for aging ones. IT departments may decide to shift old systems to the cloud, meanwhile, if they wish to get off outdated hardware, experience utilization spikes, and anticipate cost savings from the cloud's consumption-based pricing. IT teams may need to relocate a historical database in order to connect it with more recent apps if an organization is pursuing a cloud-native strategy.

However, a hybrid cloud deployment—where on-premises data is replicated to or accessed from the cloud—is typically a better starting point and provides better risk management than a lift-and-shift database migration if you're hesitant to jump right into the cloud. By using a hybrid strategy, IT can also move its primary system from the on-premises database to the cloud, with the on-premises database serving as a backup.

Systems architects and application developers need to be especially mindful of service costs when designing cloud-based environments, regardless of the cloud database they use. This is because careless usage and setups can quickly negate any potential savings from cloud deployments.

Thursday, 29 December 2022

SQL Server : When was my SQL Server Database Last Accessed?


Your client come up to you and says , “Hey, I wanna list of all of our databases and when they were last accessed”.
You might be okay if you have auditing enabled or if there is a trace or xevent capturing this kind of data, but I'm guessing none of them exist. That's great, I don't usually monitor things like that either.

Is there a method for you to obtain that information if you're not keeping an eye on it? Nothing particularly clear comes from SQL; there is no magic in the "last_access_date" column in sys.databases or anywhere else for that matter.

It's possible to work around this issue; have a look at sys.dm_db_index_usage_stats. For every index in a database, you will observe the following intriguing columns: last_user_seek, last_user_scan, last_user_lookup, and last_user_update.
You may find out when the database was last visited by using the code below, which will provide the most recent date that any index in the database was modified.
SELECT db_name(databases.database_id) AS DBName,
(SELECT MAX(last_user_access)
FROM (VALUES (MAX(last_user_seek)),(MAX(last_user_scan))
,(MAX(last_user_lookup))) AS value(last_user_access)) AS last_user_access
FROM sys.dm_db_index_usage_stats indexstats
RIGHT OUTER JOIN sys.databases databases
ON indexstats.database_id = databases.database_id
GROUP BY databases.database_id
ORDER BY db_name(databases.database_id)
Note: sys.dm_db_index_usage_stats like many DMVs is cleared out on a SQL restart.

Thursday, 22 December 2022

RDS SQL Server : Enable and Configure Database mail profile


In this article will discuss about enabling and configuring Database Mail on AWS RDS SQL Server.

Before configuring Database Mail, you've to first enable it through a database parameter group.
In Amazon RDS, parameter groups act as a container for engine configuration values that are applied to one or more DB instances. Each RDS instance comes with an associated default parameter group; however, we can’t modify it.

You can either use a new parameter group or an existing created parameter group. If choosing an existing parameter group, it must support your SQL Server instance edition and version.
To enable Database Mail through a parameter group, complete the following steps:

- On the Amazon RDS console, choose Parameter groups.
- Choose the parameter group you want to use.
- In the search box, enter database mail xps.
- Choose Edit Parameters to modify the value.
- For Values, choose 1.
- Save your changes.
- On the Amazon RDS console, choose Databases.
- Choose the instance you want to use.
- Choose Modify.
- Under Database options, choose the parameter group that has database mail xps set to 1.
- Choose Continue.
- Under Scheduling of modifications, choose Immediately.
- Choose Modify DB Instance to apply the changes.

Before we can use Database Mail, we need to set up a mail configuration.
- Launch SQL Server Management Studio.
- Connect to the SQL Server engine of the RDS instance that Database Mail is enabled for.
- Open a new query.
Use the following stored procedures to create a simple Database Mail configuration.

- Create a Database Mail profile (a profile is a container used to store email accounts). See the following code:
use msdb
go

EXECUTE msdb.dbo.sysmail_add_profile_sp
@profile_name = 'Notifications',
@description = 'Profile used for sending outgoing notifications using SES.' ;

- Add principles to the profile; use public so any user can access the profile:
use msdb
go

EXECUTE msdb.dbo.sysmail_add_principalprofile_sp
@profile_name = 'Notifications',
@principal_name = 'public',
@is_default = 1 ;

We can grant the permissions on Database Mail objects as needed, but public is fine for now.
- Create the Database Mail account (make sure to enter the correct SMTP credentials):

use msdb
go

EXECUTE msdb.dbo.sysmail_add_account_sp
@account_name = 'Acc1',
@description = 'Mail account for sending outgoing notifications.',
@email_address = 'example@example.com',
@display_name = 'Automated Mailer',
@mailserver_name = 'email-smtp.us-west-2.amazonaws.com',
@port = 587,
@enable_ssl = 1, @username = 'SMTP-username',
@password = 'SMTP-password' ;

- Add Database Mail account to the Database Mail profile:
use msdb
go

EXECUTE msdb.dbo.sysmail_add_profileaccount_sp
@profile_name = 'Notifications',
@account_name = 'Acc1',
@sequence_number = 1 ;

- Sending a test email
EXEC msdb.dbo.sp_send_dbmail
@profile_name = 'Notifications',
@recipients = 'success@simulator.amazonses.com',
@body = 'The database mail configuration was completed successfully.',
@subject = 'Automated Success Message';
GO

Then run this stored procedure to see all email items:
SELECT * FROM msdb.dbo.sysmail_allitems

Please refer the below document for more details on the same : [+] Using Database Mail on Amazon RDS for SQL Server; https://aws.amazon.com/blogs/database/using-database-mail-on-amazon-rds-for-sql-server/

Tuesday, 22 November 2022

RDS SQL Server : Script to automate native backup of all the individual databases using msdb.dbo.rds_backup_database Stored procedure


You can use the below command to automate native backup of all the databases using msdb.dbo.rds_backup_database Stored procedure. You can also use the below script to schedule as a agent job to take native backup (in .bak file) of your all databases.

Script to take Full backup without encryption


DECLARE @BackupFileName varchar(50)
DECLARE @DBName sysname
DECLARE @S3ARN_Prefix nvarchar(100)
DECLARE @S3ARN nvarchar(100)
SET @S3ARN_Prefix = 'arn:aws:s3:::awsbucket_name/' -- update your bucketname

DECLARE DBBackup CURSOR FOR
SELECT name
FROM master.sys.databases
WHERE name NOT IN ('rdsadmin', 'master', 'model', 'msdb', 'tempdb')
AND state_desc = 'ONLINE'
AND user_access_desc = 'MULTI_USER'
AND is_read_only = 0

OPEN DBBackup
FETCH NEXT FROM DBBackup INTO @DBName
WHILE (@@FETCH_STATUS = 0)

BEGIN
SET @BackupFileName = @DBName + '_' + REPLACE(REPLACE(REPLACE(CONVERT(varchar,GETDATE(),20),'-',''),':',''),' ','') + '.bak'
SET @S3ARN = @S3ARN_Prefix + @BackupFileName
EXEC msdb.dbo.rds_backup_database
@source_db_name = @DBName,
@S3_arn_to_backup_to = @S3ARN,
@overwrite_S3_backup_file = 0
FETCH NEXT FROM DBBackup INTO @DBName
END


Script to take Full backup with encryption


DECLARE @BackupFileName varchar(50)
DECLARE @DBName sysname
DECLARE @S3ARN_Prefix nvarchar(100)
DECLARE @S3ARN nvarchar(100)
DECLARE @KMS_master_key_ARN nvarchar(100)
SET @S3ARN_Prefix = 'arn:aws:s3:::awsbucket_name/' -- update your bucketname
SET @KMS_master_key_ARN = 'arn:aws:kms:us-east-1:XXXXXXXXXX:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxx' -- update your kms key ARN

DECLARE DBBackup CURSOR FOR
SELECT name
FROM master.sys.databases
WHERE name NOT IN ('rdsadmin', 'master', 'model', 'msdb', 'tempdb')
AND state_desc = 'ONLINE'
AND user_access_desc = 'MULTI_USER'
AND is_read_only = 0

OPEN DBBackup
FETCH NEXT FROM DBBackup INTO @DBName
WHILE (@@FETCH_STATUS = 0)

BEGIN
SET @BackupFileName = @DBName + '_' + REPLACE(REPLACE(REPLACE(CONVERT(varchar,GETDATE(),20),'-',''),':',''),' ','') + '.bak'
SET @S3ARN = @S3ARN_Prefix + @BackupFileName
EXEC msdb.dbo.rds_backup_database
@source_db_name = @DBName,
@S3_arn_to_backup_to = @S3ARN,
@KMS_master_key_arn = @KMS_master_key_ARN,
@overwrite_S3_backup_file = 0
FETCH NEXT FROM DBBackup INTO @DBName
END

To track the status of the job, use this SQL statement:


exec msdb..rds_task_status @task_id= 5 -- 5 as an example of your task id

Saturday, 22 October 2022

SQL Server : Performance degraded after upgrading the compatibility level from 2008 to 2014/2017/2019 (i.e. 100 to 120 Compatibility version).


In this article, we will discuss about the performance impact after change in compatibility level of your databases.

Please allow me to shed some light about the concept of Compatibility Level, as you know prior to SQL Server 2014, the database compatibility level of your user databases was not typically an important property that you had to be concerned with, at least from a performance perspective. Unlike the database file level (which gets automatically upgraded when you restore or attach a down-level database to an instance running a newer version of SQL Server, and which can never go back to the lower level), the database compatibility level can be changed to any supported level with a simple ALTER DATABASE SET COMPATIBILITY LEVEL = xxx command.

With SQL Server 2012 and older, the database compatibility level was mainly used to control whether new features introduced with a particular version of SQL Server were enabled or not and whether non-supported old features were disabled or not. The database compatibility level was also used as a method to maintain better backwards application compatibility with old versions of SQL Server. If you didn’t have time to do full regression testing with the newest compatibility level, you could just use the previous compatibility level until you could test and modify your applications if needed.

Now when you create a new user database in SQL Server, the database compatibility level will be set to the default compatibility level for that version of SQL Server. So for example, a new user database that is created in SQL Server 2017 will have a database compatibility level of 140. The exception to this is if you have changed the compatibility level of the model system database to a different supported database compatibility level, then a new user database will inherit its database compatibility level from the model database.

But Database Compatibility Level 120, this was when the “new” cardinality estimator (CE) was introduced. In many cases, most of your queries ran faster when using the new cardinality estimator, but it was fairly common to run into some queries that had major performance regressions with the new cardinality estimator. Using database compatibility level 120 means that you will be using the “new” CE unless you use an instance-wide trace flag or a query-level query hint to override it.

You can refer the below article from MS on the Differences between lower compatibility levels and level 120; https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-database-transact-sql-compatibility-level?view=sql-server-2017#differences-between-lower-compatibility-levels-and-level-120
Also there classic whitepaper wrote by Joe Sack on “Optimizing Your Query Plans with the SQL Server 2014 Cardinality Estimator” that explains the background and behavior of this change back in April of 2014. ; https://docs.microsoft.com/en-us/previous-versions/dn673537(v=msdn.10)?redirectedfrom=MSDN

Microsoft’s recommended upgrade process is :
[+] Change the Database Compatibility Level and use the Query Store; https://docs.microsoft.com/en-us/sql/database-engine/install-windows/change-the-database-compatibility-mode-and-use-the-query-store?view=sql-server-2017
- Upgrade to the latest SQL Server version and keep the source (legacy) database compatibility level.
- Enable Query Store, and let it collect a baseline of your workload.
- Change the database compatibility level to the native level for the new version of SQL Server.
- Use Query Store to fix performance regressions by forcing the last known good plan.

Please refer the below articles on the same :
[+] https://docs.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store?view=sql-server-2017

Microsoft also introduce the new tool Query Tuning Assistant; Starting with SQL Server Management Studio v18, the new Query Tuning Assistant (QTA) feature will guide users through the recommended workflow to keep performance stability during upgrades to newer SQL Server versions, as documented in the section Keep performance stability during the upgrade to newer SQL Server of Query Store Usage Scenarios. However, QTA doesn't roll back to a previously known good plan as seen in the last step of the recommended workflow. Instead, QTA will track any regressions found in the Query Store Regressed Queries view, and iterate through possible permutations of applicable optimizer model variations so that a new better plan can be produced.

Please have a look into the below article for more details on The Query Tuning Assistant workflow; https://docs.microsoft.com/en-us/sql/relational-databases/performance/upgrade-dbcompat-using-qta?view=sql-server-2017#the-query-tuning-assistant-workflow

Sunday, 4 September 2022

RDS SQL Server : How to reduce the size on disc for RDS SQL Server database?


First of all I would like to explain you that SQL Server databases have two types of files; Data File and Log File.

1. Data files contain data and objects such as tables, indexes, stored procedures, and views.
Data files can be shrink, however it is not a good practice to shrink the data file unless necessary as it can result into blocking, dead locks and index fragmentation. The below documentation to provides more insights on why shrink data files should be avoided.

[+] https://littlekendra.com/2016/11/08/shrinking-sql-server-data-files-best-practices-and-why-it-sucks/
[+] https://www.brentozar.com/archive/2009/08/stop-shrinking-your-database-files-seriously-now/
2. Log files contain the information that is required to recover all transactions in the database.

Log Files can be shrink, based upon the log space utilization and on left a transaction open and running.
Now if you would like to know which file in the database is taking more space can use below command to identify file utilization:
USE [your_database_name]
GO

SELECT DB_NAME() AS DbName, 
    name AS FileName, 
    type_desc,
    size/128.0 AS CurrentSizeMB,  
    size/128.0 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128.0 AS FreeSpaceMB
FROM sys.database_files
WHERE type IN (0,1);
GO
Then accordingly perform the shrink, Here are the steps to perform the shrink for database files:

For Log files:
Firstly, I would humbly request you to check if you have an open or active transaction. The following queries may help:

1. DBCC SQLERF(LOGSPACE)
>> Provides the log space utilization
[+] How can I troubleshoot storage consumption in my Amazon RDS DB instance that is running SQL Server? -
https://aws.amazon.com/premiumsupport/knowledge-center/rds-sql-server-storage-optimization/
2. SELECT name, log_reuse_wait_desc FROM sys.databases
>> Provides why each database's log file is not clearing out
[+] sys.databases (Transact-SQL)
- https://docs.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-databases-transact-sql?view=sql-server-ver15
>> Includes the possible reasons[+]:
a. Log backup needs to be run
b. Active backup running - because the full backup needs the transaction log to be able to restore to a specific point in time
c. Active transaction - someone typed BEGIN TRAN and locked their workstation for the night

[+] My Favorite System Column: LOG_REUSE_WAIT_DESC
- https://www.brentozar.com/archive/2016/03/my-favorite-system-column-log_reuse_wait_desc/
As already mentioned above, you can perform the shrink on the transaction log using the below command.

Step 1: Please run the below command to get the used space and free space of transaction log in the database [+]
DBCC SQLPERF(LOGSPACE); --> review transaction log file utilization per database level, if it is more than 80% free then only perform step2
[+] https://docs.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-sqlperf-transact-sql?view=sql-server-2017
Step 2: Query to shrink transaction log file
USE [userdb-name]
GO
DBCC SHRINKFILE (logfilename, sizeinmb)

If this instance is vital to production, I would also like to recommend using CloudWatch Alarms [+] to monitor your 'FreeStorageSpace' to
prevent Storage Full issues. The alarm will monitor this metric and notify you when the defined threshold is breached, so you may take
immediate actions to increase storage.
[+] How can I create CloudWatch alarms to monitor the Amazon RDS free storage space and prevent storage full issues?
- https://aws.amazon.com/premiumsupport/knowledge-center/storage-full-rds-cloudwatch-alarm/
For Data Files:

You can use same 'DBCC SHRINKFILE' command for shrinking a data file to a specified target size.

The following example shrinks the size of a data file named 'DataFile1' in the user database to 1024 MB.
USE [userdb-name];
GO
DBCC SHRINKFILE (DataFile1, 1024);
GO
OR From GUI- right click on database>click task> shrink> files > select data file from drop down.
I would also like you to know that data that is moved to shrink a file can be scattered to any available location in the file. This causes index fragmentation and can slow the performance of queries that search a range of the index. To eliminate the fragmentation, consider rebuilding the indexes on the file after shrinking.
To know fragmentation information for the data and indexes of the specified database ‘DB_name’
  SELECT OBJECT_NAME(OBJECT_ID), index_id,index_type_desc,index_level,
	avg_fragmentation_in_percent,avg_page_space_used_in_percent,page_count
	FROM sys.dm_db_index_physical_stats
	(DB_ID(DB_name'), NULL, NULL, NULL , 'SAMPLED')
	ORDER BY avg_fragmentation_in_percent DESC

To rebuild all indexes in a table, here is the sample query:
	ALTER INDEX ALL ON DatabaseName.SchemaName
	REBUILD WITH (FILLFACTOR = 80, SORT_IN_TEMPDB = ON, STATISTICS_NORECOMPUTE = ON);
[+] https://docs.microsoft.com/en-us/sql/relational-databases/indexes/reorganize-and-rebuild-indexes?view=sql-server-ver15
These suggestions are based on best effort basis, to conclude I would highly recommend you to please test these thoroughly before implementing them on any production environment. This will help in proactive testing to avoid any unforeseen issues during actual implementation.

Thursday, 1 September 2022

SQL Server : Capture expensive queries using Server side trace


You can follow the below steps to capture the expensive queries by creating the trace and also from SQL cache plan DMV's :

1. Create Server Side trace: by defining a trace, its the events and columns to captured, and add the filter condition.

You can use the below Script to create the trace:

-- Create Server-side trace
-- Create a Queue
DECLARE @rc int
DECLARE @TraceID int
DECLARE @maxfilesize bigint
set @maxfilesize = 2048

-- Please replace the text InsertFileNameHere, with an appropriate
-- filename prefixed by a path, e.g., c:\MyFolder\MyTrace. The .trc extension
-- will be appended to the filename automatically. If you are writing from
-- remote server to local drive, please use UNC path and make sure server has
-- write access to your network share

DECLARE @MyFileName nvarchar(256)
SELECT @MyFileName='D:\RDSDBDATA\Log\RDSSQLTrace_' + CONVERT(nvarchar(10),GetDate(),121) + '_' + REPLACE(CONVERT(nvarchar(10),GetDate(),108),':','-')


exec @rc = sp_trace_create @TraceID output, 0, @MyFileName, @maxfilesize, NULL
if (@rc != 0) goto error

-- Client side File and Table cannot be scripted
-- Set the events
DECLARE @on bit
set @on = 1 --sp_trace_setevent @traceid ,@eventid ,@columnid,@on

-- Capturing the RPC:Completed event

EXEC sp_trace_setevent @TraceID, 10, 16, @on
EXEC sp_trace_setevent @TraceID, 10, 1, @on
EXEC sp_trace_setevent @TraceID, 10, 17, @on
EXEC sp_trace_setevent @TraceID, 10, 14, @on
EXEC sp_trace_setevent @TraceID, 10, 18, @on
EXEC sp_trace_setevent @TraceID, 10, 12, @on
EXEC sp_trace_setevent @TraceID, 10, 13, @on
EXEC sp_trace_setevent @TraceID, 10, 8, @on
EXEC sp_trace_setevent @TraceID, 10, 10, @on
EXEC sp_trace_setevent @TraceID, 10, 11, @on
EXEC sp_trace_setevent @TraceID, 10, 35, @on

--Capturing the SQL:BatchCompleted event

EXEC sp_trace_setevent @TraceID, 12, 16, @on
EXEC sp_trace_setevent @TraceID, 12, 1, @on
EXEC sp_trace_setevent @TraceID, 12, 17, @on
EXEC sp_trace_setevent @TraceID, 12, 14, @on
EXEC sp_trace_setevent @TraceID, 12, 18, @on
EXEC sp_trace_setevent @TraceID, 12, 12, @on
EXEC sp_trace_setevent @TraceID, 12, 13, @on
EXEC sp_trace_setevent @TraceID, 12, 8, @on
EXEC sp_trace_setevent @TraceID, 12, 10, @on
EXEC sp_trace_setevent @TraceID, 12, 11, @on
EXEC sp_trace_setevent @TraceID, 12, 35, @on

-- Set the Filters
EXEC sp_trace_setfilter @TraceID,11,0,7,N'%NT AUTHORITY%' --SQL login
EXEC sp_trace_setfilter @TraceID, 10, 0, 7, N'RdsAdminService'

-- Set the trace status to start
EXEC sp_trace_setstatus @TraceID, 1

-- display trace id and file name, Note these details for future references
SELECT @TraceID,@MyFileName

goto finish

error:
select ErrorCode=@rc

finish:
go

2. Start the Trace using below command :

-- To Start the trace (Please update the trace_id as you may recived at the time of creation)
Exec sp_trace_setstatus @traceid = 2 , @on = 1 -- start

3. Start the application test run

4. Once test run is completed, Stop the trace using below command:

-- To Stop the trace (Please update the trace_id as you may recived at the time of creation)

Exec sp_trace_setstatus @traceid = 2 , @on = 0 -- stop
-- To Delete the trace, once application test is completed.
--Exec sp_trace_setstatus @traceid = 2 , @on = 0 -- Delete definition from server

5. Insert the trace output in dummy table, this helps to get the data in easy to read format.

USE [DB_name]
GO
SELECT * INTO RDSSQLTrace_20210722_1215 FROM ::fn_trace_gettable ('D:\RDSDBDATA\Log\RDSSQLTrace_2021-07-22_12-15-19.trc', default)
6. Capture the output of below commands and update the output in the attached excel sheet template in the TraceData sheet (File Name : Capture_ExpensiveQueries_Template.xlsx)

--To collect queries based on max Duration 
SELECT TOP 10 StartTime,EndTime,Reads, Writes, 
Duration/1000000 DurationSec,CPU, TextData,RowCounts,sqlHandle,[DatabaseName],
SPID,LoginName,EventClass FROM [gptst].[dbo].[RDSSQLTrace_20210722] 
ORDER BY Duration DESC

--To collect queries based on max CPU utilization
SELECT TOP 10 StartTime,EndTime,Reads, Writes, 
Duration/1000000 DurationMs,CPU, TextData,RowCounts,sqlHandle,[DatabaseName],
SPID,LoginName,EventClass FROM [gptst].[dbo].[RDSSQLTrace_20210722] 
ORDER BY CPU DESC

--To collect queries based on max Reads,Writes
SELECT TOP 10 StartTime,EndTime,Reads, Writes, 
Duration/1000000 DurationSec,CPU, TextData,RowCounts,sqlHandle,[DatabaseName],
SPID,LoginName,EventClass FROM [gptst].[dbo].[RDSSQLTrace_20210722]
ORDER BY Reads,Writes DESC
7. Also, capture the output of the below scripts as well and update the same in the attached excel sheet template in the CacheData sheet (File Name : Capture_ExpensiveQueries_Template.xlsx)
--TOP 10 Queries by Duration (Avg Sec) from system dmv's		
			
SELECT TOP 10 DB_NAME(qt.dbid) AS DBName,
o.name AS ObjectName,
qs.total_worker_time / 1000000 / qs.execution_count As Avg_CPU_time_sec,
qs.total_worker_time / 1000000 as Total_CPU_time_sec,
qs.total_elapsed_time / qs.execution_count / 1000000.0 AS Average_Seconds,
qs.total_elapsed_time / 1000000.0 AS Total_Seconds,
'',
qs.execution_count as Count,
qs.last_execution_time as Time,
SUBSTRING (qt.text,qs.statement_start_offset/2,
(CASE WHEN qs.statement_end_offset = -1
THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
ELSE qs.statement_end_offset END - qs.statement_start_offset)/2) AS Query
--,qt.text [MainQuenry]
--,qp.query_plan
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp
LEFT OUTER JOIN sys.objects o ON qt.objectid = o.object_id
where qs.execution_count>5 and DB_NAME(qt.dbid) not in ( 'master','msdb','model')
--and DB_NAME(qt.dbid) is not null
ORDER BY average_seconds DESC
--TOP 10 Queries by CPU (Avg CPU time) from system dmv's
SELECT TOP 10 DB_NAME(qt.dbid) as DBName,
	qs.total_worker_time / 1000000 / qs.execution_count AS Avg_CPU_time_sec,
	qs.total_worker_time / 1000000 As Total_CPU_time_sec,
	qs.total_elapsed_time / 1000000 / qs.execution_count As Average_Seconds,
	qs.total_elapsed_time / 1000000 As Total_Seconds,
    (total_logical_reads + total_logical_writes) / qs.execution_count as Average_IO,
	total_logical_reads + total_logical_writes as Total_IO,	
    qs.execution_count as Count,
    qs.last_execution_time as Time,
	SUBSTRING(qt.[text], (qs.statement_start_offset / 2) + 1,
		(
			(
				CASE qs.statement_end_offset
					WHEN -1 THEN DATALENGTH(qt.[text])
					ELSE qs.statement_end_offset
				END - qs.statement_start_offset
			) / 2
		) + 1
	) as Query
   ,qt.text
	,qp.query_plan
FROM sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.[sql_handle]) AS qt
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp
LEFT OUTER JOIN sys.objects o ON qt.objectid = o.object_id
where 
DB_NAME(qt.dbid) not in ( 'master','msdb','model')
ORDER BY Avg_CPU_time_sec DESC

-- Top 10 query by IO (Avg IO) from system dmv's

SELECT TOP 10 DB_NAME(qt.dbid) AS DBName,
o.name AS ObjectName,
qs.total_elapsed_time / 1000000 / qs.execution_count As Average_Seconds,
qs.total_elapsed_time / 1000000 As Total_Seconds,
(total_logical_reads + total_logical_writes ) / qs.execution_count AS Average_IO,
(total_logical_reads + total_logical_writes ) AS Total_IO,
'',
qs.execution_count AS Count,
last_execution_time As Time,
SUBSTRING (qt.text,qs.statement_start_offset/2,
(CASE WHEN qs.statement_end_offset = -1
THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
ELSE qs.statement_end_offset END - qs.statement_start_offset)/2) AS Query
--,qt.text
--,qp.query_plan
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp
LEFT OUTER JOIN sys.objects o ON qt.objectid = o.object_id
where last_execution_time > getdate()-1 
and DB_NAME(qt.dbid) not in ( 'master','msdb','model')
--and DB_NAME(qt.dbid) is not null
and qs.execution_count > 5 
ORDER BY average_IO DESC
Note : Please do capture the output after each run if running the test multiple times, to compare the result set.

Monday, 22 August 2022

RDS SQL Server : Unable to restore and getting an error message : "There is not enough space on the disk to perform restore database operation."


If you are getting an error message as "There is not enough space on the disk to perform restore database operation."

-- This issue appears when rds restore starts, RDS makes an estimate about how much disk space will be required by the database that is going to be restored. If that estimate exceeds the amount of free space left on the disk, you will see this error.

Kindly note that as part of the restore process is trying to look for the data and log file size as was configured on the source and if the amount of space is not available on disk upfront its failing the restore process in the first place while trying perform the restore and also the SQL server restore process will not initiate the Storage Autoscaling.

For example: lets say the source database (On-prem\EC2\RDS sql server instance) is configured with 30 GB of data file and 30 GB of log file sizes and the used space inside the 30 GB data file is only 10 GB and used space inside the 30 GB log file is only 10 GB.The backup size of the database in this case would be only 20 GB(with compression this backup file size can further be reduced) out of actual 60 GB configured on the source database.During the restore of this 20 GB backup file on the RDS SQL server instance the process looks for the 60 GB of space on the underlying storage per the source data and log file configurations.In this case if the Target RDS database instance if it has 50 GB of underlying storage so restore process would fail as it requires 60 GB of storage space to proceed with the restore and it fails with the space issue.

Now I would suggest you to please check the actual database size on the source database using below script:
--Useful script to know the current size of database files and the free space.
USE [DBNAME] -- update your db name
GO
SELECT DB_NAME() AS DbName,
name AS FileName,
size/128.0 AS CurrentSizeMB,
size/128.0 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128.0 AS FreeSpaceMB
FROM sys.database_files;

If log is using the most of the storage space then you can try to shrink the log file and then again the backup and try to restore again.

Sunday, 22 May 2022

SQL Server : Unable to connect to the database, error message Logon Login failed for user 'USER01'. Reason: Password did not match that for the login provided. [CLIENT: 10.xxx.xxx.xxx]


In this article, you will be able to see how to troubleshoot when you are unable to connect to the database and getting an error message "Logon Login failed for user 'USER01'. Reason: Password did not match that for the login provided. [CLIENT: 10.xxx.xxx.xxx]."

Please allow me to explain you as per the error message it seems that Password you are providing for user "USER01" is not matching with actual one.

You can follow the below steps as a troubleshooting :
1. If you have SQL Server Management Studio, please re-try and test connectivity with the login and correct password.

2. If you are checking from Application Server then check the user / password settings with an udl file from the application server; just follow the steps below :

2.1) Create a new empty text file like "test.txt" (For example, right mouse click on an empty place on your desktop, select "New" and "Text Document")
2.2) Rename the file as "test.udl"
As soon as you renamed the file, text file icon should change to a UDL icon
2.3) Double click on test.udl file and you'll receive a window titled "Data Link Properties"
2.4) Provide the correct username and password; Test Connectivity from application server

3. You can check if there is any orphan user by the same login using below command:
-- command to check orphan user, this will give you the SID:
Use [YourDB_Name]
GO
EXEC sp_change_users_login 'Report'
GO

If that is the orphan user, than to fix the orphan user using below command:

-- First drop the user from login and then re-create using below script.
-- get the sid using above command for corresponding user
use master
go
CREATE LOGIN login name WITH PASSWORD = 'login password', sid='' -- use the same SID given by above script
GO

Tuesday, 2 November 2021

AWS : How to connect S3 Bucket from EC2 Instance which is in Private VPC without Internet access?

Problem:

How to upload the files to S3 bucket from the Win Server (i.e. EC2 Instance) which is in private vpc with no general internet access?

 

Solution:

First of all, as you already know to enable an upload you need to ensure that there is a connectivity between Server and S3 Buckets.

So there are different approaches based upon Setup and Configuration, few options explained as follows:

1. If the EC2 instance and the S3 bucket are in the same AWS region, Then Using VPC Endpoint : You can use VPC Endpoints and associate it with the route table of the EC2 instance's subnet which would allow traffic to the S3 service in that region even if you don't have a NAT or Internet gateway associated with your subnet.

For more about S3 VPC endpoints:
    S3 VPC endpoint in detail:
        [+] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-s3.html
    create a VPC Endpoint:
        [+] https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

2. If the EC2 instance and the S3 bucket are in different AWS regions, Then Using NAT Gateway or Internet Gateway: You have to make use of either NAT Gateway or Internet gateway to allow traffic from your instance to the S3 service in that region.
If you dont want to keep the internet open on the server, you can later deny traffic between your EC2 Instance and S3 Service after copying the backup file on to S3 Bucket.

For more about NAT\Internet Gateways:
    NAT gateways :
     [+] https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html
    Internet gateways :
     [+] https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html
 

Tuesday, 20 April 2021

AWS : Relational Database Service (RDS) - Basics Part 4

 ==============================================================
How do you maintain my database ? How can I plan for it?

• Any maintenance that causes downtime (typically only a few times per year) will be scheduled in your maintenance window
• Operating system or Amazon RDS software patches are usually performed without restarting databases
• Database engine upgrades require downtime
    • Minor version upgrades — automatic or manually applied (Automatic not applicable in case of SQL Server, always manual)
    • Major version upgrades — manually applied
    • Version deprecations—three-to six-month notification before scheduled upgrades
• View upcoming maintenanceevents in your AWS Personal Health Dashboard

==============================================================
How am I charged for Amazon RDS ?

• Database instance (instance hours)
    • Combination of Region + instance type + database engine + license (optional)
• Database storage (GB-mo)
    • Can be either provisioned (Amazon EBS) or consumed (Amazon Aurora)
    • Provisioned IOPS (IOPS-Mo) for IO1 storage type
    • Database I/O requests (IOs) for Amazon Aurora and Amazon EBS magnetic-storage types
• Backup storage (GB-mo
    • Size of backups and snapshots stored in Amazon S3
    • No charge for backup storage up to 100% of total database storage
• Data transfer (GB-mo)
    • Uses AWS regional data-transfer pricing

==============================================================
How can I save money on my database?

• Amazon RDS Reserved Instances (RIs) provide a discount over on-demand prices
• Region, instance family, and engine of on-demand usage must match to apply benefit
• Amazon RDS RIs offer size flexibility for open-source and Oracle BYOL engines
• By default, RIs are shared between allaccounts in consolidated billing
• Use the RI utilization report to determinehow your RIs are being used
    • Support for RI coverage reportcoming soon

==============================================================
Can I stop my database when it’s not in use?

• Stop and start a running database instance from the console or AWS CLI
• Available for single-AZ DB instances
• While instance is stopped, you only pay for storage
• Backup retention window is maintained while stopped
• Instances are restarted after 7 days
    • Pending maintenance operations are applied
    • Instances can be stopped again if desired

==============================================================