SQL Exec Stats

Copyright © 2006-2010 Elemental Inc.

All rights reserved.

www.qdpma.com

Email: jchang@qdpma.com or jchang6@yahoo.com

Download SQL Exec Stats

License

The license to use this software is free. Elemental Inc. (www.QDPMA.com) assumes no liability for use.

About

The Distributed Management Views and functions (DMV and DMF) have greatly simplified the process of collecting performance data sufficient for basic assessment and SQL query and index tuning. Still, even the modest effort to collecting a sufficient set of information is not completely turn-key that people do this on a regular basis. Also, query tuning requires the additional effort to examine the execution plans.

Index tuning involves both indexes that should be added or expanded, and indexes that should be removed or consolidated. The DMVs provide some recommendations on missing indexes, but this is not always reliable, so a live expert is still essential. Identifying indexes not used is simple. The other part is identifying infrequently used indexes that could also be dropped or consolidated. This would require finding the execution plans that reference the infrequently used indexes to assess the implications or impact.

Much of this can be automated with the SQL Exec Stats so that regular performance data collection is nearly a zero effort routine, including correlation of the execution statistics in the DMV with information from the execution plans, and a cross reference of execution plans that reference each index. Finally, if performance expertise is not available onsite, the set of information is sufficiently compact to be sent by email or ftp for full assessment.

 

Requirements

  1. Either SQL Server 2005 any build, service pack 2 or later prefered (statement level query plan).
  2. or SQL Server 2008 any build, service pack 1 preferred (worker time for parallel execution plans).
  3. .NET Framework 3.0, (older version works with 2.0)

Supported versions are SQL Server 2005 (service pack 2 or later preferred), and SQL Server 2008 (service pack 1 or later preferred). SQL Server 2008 SP1 now correctly reports worker time for parallels execution plans. Previous 2005 and 2008 versions typically reported zero, negating the ability to weight by CPU correctly when parallel plans are present.

References in previous versions to SMO were removed. The SMO dlls are 32-bit only. It is highly desired to run ExecStats in a 64-bit environment as a large virtual address space is desired in order to handle very large amount of plan contents.
A new data structure requires .NET Framework 3.0. The previous requirements were .NET Framework 2.0. A component from Gembox (www.gemboxsoftware.com) is employed to generate Excel files.

 

 

Objectives: What does SQL Exec Stats do?

  1. Automate collection of key DMVs for query tuning.
  2. Collates query exec stats with execution plan details.
  3. Cross reference index usage by execution plan, what plans reference infrequently used indexes.
  4. Saves XML plan for top queries.
  5. Save information directly to Microsoft Excel 2003 file format and in binary.

DMV and DMFs include:

  1. dm_exec_query_stats
  2. dm_exec_query_plan
  3. or dm_exec_text_query_plan
  4. dm_exec_sql_text
  5. dm_db_index_usage_stats
  6. dm_io_virtual_file_stats

The collation involves parsing the XML plan so that plan information can be displayed along with execution statistics, and to build a cross reference between index usage statistics and the execution plans that reference each index.

The performance data is saved as a compressed binary file and as a Microsoft Excel 2003 file. The top XML plans can be saved in the Sqlplan XML format that can be opened from SSMS.

Comments

The DMV dm_exec_query_stats does not a replace for SQL Server Profiler and Trace. the Profiler/Trace can capture every statement or batch. The DMV will only maintain the overall summary. The DMV may miss SQL and RPC evicted from the procedure cache or will lose history for recompiles. Still, dm_exec_query_stats is easy to use, the data is already available on an active system, and is sufficiently useful in many situations.

Sure, SQL Server 2008 has the Management Data Warehouse along with several built-in reports. My tools does some similar functions, some different, but the main difference is portatibility of data for remote tuning.

 

Purpose: Why do we need or want this?

  1. Simplifies performance data collection (sufficiently complete)
  2. Encourages regular collection of performance data, i.e. keep a history of system health
  3. Automates several elements of performance analysis (reduces time and effort)
  4. Enable remote tuning assistance, key information is compact enough to be sent by email

The intent is that later versions of this utility will have the ability to compare execution and index usage statistics on a day-to-day basis. Another possibility is to have some of the capability of Database Tuning Assistant, with the ability to identify situations where rewriting SQL is the proper resolution, not just index or partition tuning.

 

Introduction

Everyone should know by now how really useful the DMVs dm_exec_query_stats, and dm_db_index_usage_stats and the associated DMFs for obtaining the SQL and XML plan: dm_exec_sql_text, dm_exec_query_plan, and dm_exec_text_query_plan. Of course it has been explained that dm_exec_query_stats is not a replacement for SQL Server Profiler and SQL Trace. The DMV is only reliable if execution plans are not frequently evicted from the procedure cache and if SQL is not frequently recompiled.

Still the DMV dm_exec_query_stats is popular because the results are available with a simple query. The complexity of setting up a trace, making sure it is not adversely impacting server performance, making sure space is available, and most of all, the effort of parsing the trace are all avoided.

So what do we do next? From the top query statistics we can start looking at the SQL and the execution plans. One minor annoyance is that if we use CROSS APPLY to dm_exec_query_plan, each XML plan has to be saved individually. This can be annoying if one has to remote into to a PC in the server room and this connection is slow.

There is much information in the XML plan that would be helpful if we could see it in a grid with the execution statistics. From the index usage statistics, we can eliminate unused indexes. There might also be indexes that are infrequently used. Sometimes it is obvious that certain indexes can be consolidated. Other times, it is necessary to examine the execution plan to determine if another index is sufficient. The XML plan analysis cross-references in which plans each index is used. All of this is tedious work which can be automated. So this is the purpose of the SQLExecStats tools.

Keep in mind the missing index information is not an optimized list. DTA would do a proper analysis, while the missing indexes generator may apply more indexes than necessary. It would be nice to have a magic index tuning wizard, but nothing today replaces a good expert. Also, an automated tuning tool cannot question the developer as to what was meant by a specific query.

 

Configuration:

The file SQLStatsExec.exe.config contains the configuration values. Disregard the entry for Target. Source is the preferred SQL Server Instance. Database is for the preferred database. Detailed table and index information is extracted for that database. Execution statistics are instance-wide.

XML config

 

Main Screen

Below is the main screen. Target SQL Servers and Databases need to be set. Default security is SSPI, so the client must have proper permissions. Uncheck the SSPI box to set SQL Server user and password if desired. After the Target SQL Server is set, the button to the right of Databases can bring up a list of databases.

There are options for collecting the full XML Execution Plans (Exec Plans), Statistics Header (Stat Hdr) which contains the date each data distribution statistics was generated. Histogram information is currently not enabled.

The execution statistics can be statement level, batch level aggregation. A query hash aggregation will be considered for later addition. Another possibility is simply generating the execution plans for all stored procedures.

The default mode collects table and index information on one database, using the database in the config file or manually entered in the Databases drop box to the right of the Server box.

Main Screen

New in Build 2010-02-0x
On startup, the Debug Messages group Textbox at bottom will now display the host operating system and .NET Framework version, example
Microsoft Windows NT 5.1.2600 Service Pack 3, Win32NT, 5.1.2600.196608, Service Pack 3
2.0.50727.3603, 2, 0, 50727, Size of IntPtr is 4

The Size of IntPtr should indicate whether the current process is running as full 64-bit or 32-bit, even if the underlying OS is 64-bit.

Step 1
Click the button ... to the right of the Databases box or the Multi-DB button

Databases

Select the database(s) for which detailed table and index information should be collected.

Step 2
Click the “Connect Test” button at the lower left to start data collection. The SQL Exec Stats utility will collect all information and save the results in Excel and binary. Later versions may allow manual selection of which components to execute.

 

General Information

General system information is displayed in the grid to right of the Connection tab. The information include xp_msver, sys.configurations, and dm_os_sys_info.

General

The tab “OS Perf Ctrs” collects the point in time OS Performance Counters. Hopefully I can add regular sampling to display key performance counters over a period.

OS Perf Counters

 

Databases

The “Databases” tab is point in time file IO information from dm_io_virtual_file_stats. Later versions should show file IO over time.

Databases

 

File IO

The “File IO” tab is point in time file IO information from dm_io_virtual_file_stats. Later versions should show file IO over time.

File IO

The SQL for this is in the text file output. Example below:

--File IO Stats

SELECT DB_NAME(v.database_id) AS [Database], m.name AS [Filename]

, v.database_id AS dbid, v.file_id AS [file], m.type, m.data_space_id as dsid, m.state

, CONVERT(real, (m.size*8./1024.)) AS [Size]

, num_of_reads AS Reads, num_of_bytes_read AS ReadBytes, io_stall_read_ms AS RdIOStall

, num_of_writes AS Writes, num_of_bytes_written AS WrBytes, io_stall_write_ms AS WrIOStall

, ISNULL(p.io_pending_ms_ticks, 0) AS IOPen_ms, ISNULL(p.io_pending, 0) AS IOPending

, ISNULL(IOPenCnt,0) AS IOPenCnt

FROM sys.dm_io_virtual_file_stats(NULL,NULL) v

INNER JOIN sys.master_files m ON m.database_id = v.database_id AND m.file_id = v.file_id

LEFT JOIN (

SELECT io_handle, COUNT(*) AS IOPenCnt , SUM(io_pending_ms_ticks) AS io_pending_ms_ticks

, SUM(io_pending) AS io_pending

FROM sys.dm_io_pending_io_requests

GROUP BY io_handle

) p ON p.io_handle = v.file_handle

ORDER BY v.database_id, v.file_id

 

Table Space

The tab “Table Space” lists space usage by table for the selected database.

Table Space

The query for table space is as follows:

--Table Space used:

DECLARE @dbid int SELECT @dbid = DB_ID()

SELECT @dbid AS dbid , [Schema], [Object], object_id, [Type]

, [Rows], Reserved, Data, [Index] = index2, Unused = Reserved - Used

, Clust, IX_Cnt, StatCnt, Partitions, Hypothetical, create_date

FROM(

 SELECT p.object_id, o.schema_id, t.name AS [Schema], o.name AS [Object], o.Type

 , Reserved = SUM(p.reserved), Used = SUM(p.used)

 , Data = SUM( CASE WHEN (p .index_id < 2)

    THEN (p.in_row_data + p.lob_used + p.row_overflow_used)

    ELSE p.lob_used + p.row_overflow_used END )

 , index2 = SUM(CASE WHEN (p.index_id > 1) THEN (p.in_row_data) ELSE 0 END)

 , Rows = SUM( CASE WHEN (p.index_id < 2) THEN p.row_count ELSE 0 END )

 , Clust = SUM (CASE p.index_id WHEN 1 THEN 1 ELSE 0 END)

 , IX_Cnt = SUM (CASE WHEN (p.index_id>1) THEN 1 ELSE 0 END)

 , StatCnt = (SELECT COUNT(*) FROM sys.stats s WHERE s.object_id = p.object_id)

 , Partitions = MAX(Partitions), Hypothetical = SUM(CASE i.is_hypothetical WHEN 1 THEN 1 ELSE 0 END)

 , o.create_date

 FROM (

  SELECT object_id, index_id, COUNT(*) AS Partitions

  , reserved = 8*SUM(reserved_page_count), used = 8*SUM(used_page_count)

  , in_row_data = 8*SUM(in_row_data_page_count) , lob_used = 8*SUM(lob_used_page_count)

  , row_overflow_used = 8*SUM(row_overflow_used_page_count)

  , row_count = SUM(row_count)

  FROM sys.dm_db_partition_stats GROUP BY object_id, index_id

  ) p

 INNER JOIN sys.indexes i ON i.object_id = p.object_id AND i.index_id = p.index_id

 LEFT JOIN sys.objects o ON o.object_id = p.object_id

 LEFT JOIN sys.schemas t ON t.schema_id = o.schema_id

 WHERE o.type NOT IN ('S', 'IT', 'TF')

 GROUP BY o.schema_id, p.object_id, o.name, o.type, t.name, o.create_date

) x

ORDER BY Data DESC

 

Index Space and Usage

The Index Usage view includes index space, and index usage system view.

Index Usage 1

The columns to the right indicated the number of execution plan references that include a scan, insert (or update) and delete, along with the specific plans that reference the operation.

Index Usage 2

The queries for index space and usage is as follows:

--Index Space SQL:

DECLARE @dbid int SELECT @dbid = DB_ID()

SELECT @dbid AS dbid, u.name AS [Schema], o.name AS [Object], ISNULL(i.name,'') AS [Index]

, i.object_id, i.index_id, o.type AS otype

, p.reserved, p.used, p.leaf, p.row_count

, i.type AS itype, data_space_id AS dsid, p.Partitions, is_unique, ignore_dup_key, is_primary_key

,is_unique_constraint, is_disabled, is_hypothetical, allow_row_locks, allow_page_locks

FROM sys.indexes i

INNER JOIN sys.objects o ON o.object_id = i.object_id

INNER JOIN sys.schemas u ON u.schema_id = o.schema_id

LEFT JOIN (

 SELECT object_id, index_id, COUNT(*) AS Partitions

 , reserved = 8*SUM(reserved_page_count) , used = 8*SUM(used_page_count)

 , leaf = 8*SUM(in_row_data_page_count+lob_used_page_count+row_overflow_used_page_count)

 , row_count = SUM(row_count)

 FROM sys.dm_db_partition_stats

 GROUP BY object_id, index_id

 ) p ON p.object_id = i.object_id AND p.index_id = i.index_id

WHERE o.type NOT IN ('S','IT','TF')

ORDER BY o.name , i.index_id

 

--Index Usage SQL:

DECLARE @dbid int SELECT @dbid = DB_ID()

SELECT s.database_id, s.object_id, s.index_id

, s.user_seeks, s.user_scans, s.user_lookups, s.user_updates

, s.last_user_seek, s.last_user_scan, s.last_user_lookup, s.last_user_update

, s.system_seeks ASs_seeks, s.system_scans AS s_scans

, s.system_lookups AS s_lookups, s.system_updates AS s_updates

, s.last_system_seek AS ls_seek, s.last_system_scan AS ls_scan

, s.last_system_lookup AS ls_lookup, s.last_system_update AS ls_update

FROM sys.dm_db_index_usage_stats s

 

Exec Stats

The first set columns are aggregations from dm_exec_query_stats. After Plan Generation (PGenN) are information extracted from the XML plan, including: compile cost, plan cost, estimated number of rows, the number of steps in the plan, number of parallel execution steps, the number of scans, inserts, updates and delete, and finally the compile parameters.

Exec Stats Statment 1

Exec Stats 2

-- Exec Stats Stmt:

SELECT  q.statement_start_offset, q.statement_end_offset

, q.plan_handle, q.plan_generation_num, q.creation_time, q.last_execution_time, q.execution_count

, q.total_worker_time, q.max_worker_time

, q.total_physical_reads , q.total_logical_writes , q.total_logical_reads

, q.total_clr_time, q.total_elapsed_time, q.max_elapsed_time

 , q.query_hash, q.query_plan_hash

--, SUBSTRING(s.text, statement_start_offset/2 + 1,  (CASE WHEN statement_end_offset = -1 THEN LEN(CONVERT(nvarchar(max),text)) * 2   ELSE statement_end_offset END - statement_start_offset)/2) AS [Stmt], p.query_plan

FROM(

 SELECT TOP 300 statement_start_offset, statement_end_offset, plan_generation_num, plan_handle

,creation_time, last_execution_time , execution_count, total_worker_time, max_worker_time

,total_physical_reads, total_logical_writes, total_logical_reads , total_clr_time, total_elapsed_time, max_elapsed_time

 , query_hash, query_plan_hash

 FROM sys.dm_exec_query_stats

 WHERE statement_start_offset <> 86

 ORDER BY total_worker_time DESC

 ) q

--CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS s

--CROSS APPLY sys.dm_exec_text_query_plan(plan_handle, statement_start_offset, statement_end_offset) AS p

Below is the output in batch mode.

Exec Stats Batch

-- Exec Stats Batch:

SELECT 0 AS statement_start_offset, -1 AS statement_end_offset

, q.plan_handle, q.plan_generation_num, q.creation_time, q.last_execution_time

, q.execution_count, q.total_worker_time, q.max_worker_time

, q.total_physical_reads, q.total_logical_writes, q.total_logical_reads

, q.total_clr_time, q.total_elapsed_time, q.max_elapsed_time, StmtCnt

-- , t.text, p.query_plan

FROM (

 SELECT TOP 300 q.plan_handle

 , MAX(q.plan_generation_num) AS plan_generation_num , MIN(q.creation_time) AS creation_time

 , MAX(q.last_execution_time) AS last_execution_time

 , AVG(q.execution_count) AS execution_count

 , SUM(q.total_worker_time) AS total_worker_time, MAX(q.max_worker_time) AS max_worker_time

 , SUM(q.total_physical_reads) AS total_physical_reads , SUM(q.total_logical_writes) AS total_logical_writes

 , SUM(q.total_logical_reads) AS total_logical_reads

 , SUM(q.total_clr_time) AS total_clr_time , SUM(q.total_elapsed_time) AS total_elapsed_time

 , MAX(q.max_elapsed_time) AS max_elapsed_time

 , COUNT(*) AS StmtCnt

  FROM sys.dm_exec_query_stats q

  WHERE statement_start_offset <> 86

  GROUP BY q.plan_handle

  ORDER BY SUM(q.total_worker_time) DESC ) q

-- CROSS APPLY sys.dm_exec_sql_text(plan_handle) AS t

-- CROSS APPLY sys.dm_exec_query_plan(plan_handle) AS p

Below is Plan Cache. Notice execution statistics is not available.

Plan Cache

-- Plan Cache:

SELECT  bucketid, refcounts, usecounts, size_in_bytes, objtype, plan_handle

FROM sys.dm_exec_cached_plans WHERE cacheobjtype = 'Compiled Plan'

Clicking on the SQL column in Exec Stats will shift to the tab page XML Plan. Also use the < and > buttons to navigate XML plans. Click the button “Save XML Plans” to save each execution plan as a separate sqlplan file.

XML Plan

The Show Plan tab is similar to the plan in the old SQL Server 2000 tabular format.

Show Plan

 

Data Distribution Statistics Header

Includes date that the column or index statistic was updated.

Dist Stats

 

Run Time Information

A text file name in the format: Server_Instance_YYYYMMDD_hhmm.txt contains runtime information, including some of the queries sent.

_x0000_i1039

 

Excel File

Much of the data collected is saved to an Excel file name format: Server_Instance_YYYYMMDD_hhmm.xls.

 

XML Plan files

XML plan files are not saved by default. Click the “Save XML Plans” button after running the data collection. Plans are saved in a directory starting with server and instance name followed by a date time code.

_x0000_i1040

Click the Clear button to start a new sequence

 

Appendix

Index Usage columns

No.ColumnNote
0dbidDatabase id
1SchemaSchema
2ObjectObject name
3IndexIndex name
4object_idObject id
5index_idIndex id
6otypeObject type
7ReservedReserved space in KB
8Used
9Leaf
10row_count
11user_seeks
12user_scans
13user_lookups
14user_updates
15last_user_seek
16last_user_scan
17last_user_lookup
18last_user_update
19s_seeksSystem seeks
20s_scans
21s_lookups
22s_updates
23ls_seekLast System seek
24ls_scan
25ls_lookup
26ls_update
27itype
28dsid
29Partitions
30is_unique
31ignore_dup_key
32is_primary_key
33is_unique_constraint
34is_disabled
35is_hypothetical
36allow_row_locks
37allow_page_locks
38SeeksThe number of execution plans that reference this index in seek operations
39SeekPlanRefThe execution plan(s) that reference this index in seek operations
40Scans
41ScanPlanRef
42InsertUpdsThe number of execution plans that reference this index in insert or update operations
43InsUpdPlanRef
44Deletes
45DeletePlanRef
46IxCodeCl-Clustered, N-nonclustered, PK-PrimaryKey, UC-Unique Contraint, U-Unique
47IndexColumns
48
49
50

 

Exec Stats

The Exec Stats grid is populated by the query, with additional calculations.

The top row is the summary for all dm_exec_query_stats rows. The second row is the summary for only the top X rows shown below.

 

No.ColumnDescription
0Seqsequential number, may have later uses for cross-referencing ExecStats results for previous samples
1SQLSQL text
2CntExecution count
3CPU-sCPU or worker time in seconds.
4Dur-sDuration or elapsed time in seconds.
5AvCPU-usAverage CPU in micro-seconds.
6AvDur-usAverage Duration in micro-seconds.
7MxCmsMax CPU in milli-seconds.
8MxDmsMax Duration in milli-seconds.
9PhyRdsPhysical Reads
10LogWrLogical Writes
11PGenNPlan Generation Number
12CompileTime
13CompileCPU
14EstRowsEstimate Rows
15PlanCost
16NStmtsNumber of statements. Always 1 for stmt mode, can be more for batch mode.
17PlanStepsNumber of Plan Steps. Each RelOp node in the XML plan is a plan step
18ParallelParallel Plan: In the summary rows (1 & 2), this is the number of execution plans that have parallel operations.
For the individual query plans (rows 3 and up) this is the number of plan steps that are parallel.
19MissingIx
20Impact
21Warnings
22ScansThe number of plans that involve scan operations (summary) or the number of scan operations in the given query plan. (May only count scans of > 10MB?)
23sCPUPlan Estimate CPU cost of scan operations
24sIOPlan Estimate IO cost of scan operations, 1 = 1350 pages or 10.5MB
25sRowsEstimate rows from scan operations.
26RBOpsSummary: the number of plans that contain Rebind operations.
Query Plan: the number of plan steps that have rebind operations.
27RebindsTotal number of rows involved for all rebind ops in the plan. In SQL Server 2000, this was the number of executions?
28Rewinds
29PredicatesIndicates that an index seek or table scan operation had a predicate (search condition) which was not involved in the index seek.
30InsertsSummary: the number of execution plans that have insert operations.
Query Plan: The number of insert operations in the query plan.
31Updates
32Deletes
33IRowsEstimates number of rows for all insert operations
34URowsUpdate rows
35DRowsDelete rows
36HashJHash Joins
37HashAHash Aggregates
38MergeMerge joins
39MTMMany-to-many merge joins
40dbiddatabase id
41CreationPlan Creation date-time
42LastExecLast Execution date
43OptLevOptimization Level
44EARSQL optimization Early Abort Reason
45StmtTypeStatement Type
46ParamParameters

Show Plan

The intent of this screen is to somewhat replicate the SHOWPLAN_ALL output that was available even in SQL Server 2000. The SHOWPLAN_ALL columns are: StmtText, StmtId, NodeId, Parent, PhysicalOp, LogicalOp, Argument, DefinedValues, EstimateRows, EstimateIO, EstimateCPU, AvgRowSize, TotalSubstreeCost, OutputList, Warnings, Type, Parallel, EstimateExecutions

The top row is the summary for all dm_exec_query_stats rows. The second row is the summary for only the top X rows shown below.

 

No.ColumnDescription
0PlanCorresponds to the Seq column from the ExecStats page.
1StmtStatementId
2CpIdStatementCompId, not sure what this means
3LineThere is one line for each RelOp in the XML plan
4DatabaseFor top level, Line 0, this is the StatementText field. For higher Line numbers, if a object is referenced, this is the Database
5Schema
6Table
7Index
8NodeNodeId
9Parent
10PhysicalOp
11Logical Op
12EstRows
13EstIO
14EstCPU
15AvgRowSz
16EstTotSubTr
17OperatorOperator cost. The Total Subtree cost is not always equal to the sum of the local operation CPU and IO plus the subtrees that feed into it. Certain operations like Key Lookup and Nested Loops inner source are executed multiple times.
18Parallel
19Rebinds
20Rewinds
21TableCardTable Cardinality is a SQL Server 2008 value
22Alias
23TableRef
24SeekPred
25Predicate
26IndexSc
27Join1Should be join outer source
28Join2should be join inner source
29Join3additional join details
30Warnings
31MiscTable Cardinality is a SQL Server 2008 value
32Output

Distribution Stats

The Exec Stats grid is populated by the query, with additional calculations.

The top row is the summary for all dm_exec_query_stats rows. The second row is the summary for only the top X rows shown below.

 

No.ColumnDescription
0dbid
1schema
2Object
3Stat
4object_id
5stats_id
6Columns
7oType
8cType
9auto_created
10user_created
11no_recompute
12Updated
13Rows
14Sampled
15Steps
16Density
17AvgKeyLen
18StringIndex
19Density1
20Density2