Meta Data Cardinality Explored

1. Meta DataCardinality Explored CSSQLUG User Group - June 2009

2. Outline • Define Cardinality • Experience Cardinality • Relevance of Cardinality • Other related meta data • Questions

3. About Me • Love to Golf… Why are we here? • Founder of Data Realized • Consultant • Boss • Employee • Grunt • Occasional speaker • Dev Connections Fall 08 • SSWUG vConference Spring 09

4. Cardinality - Defined • Wikipedia • In mathematics, the cardinality of a set is a measure of the “number of elements of the set”. There are two approaches to cardinality – one which compares sets directly using bijections and injections, and another which uses cardinal numbers. • BOL • Troubleshooting Poor Query Performance: Cardinality Estimation • The first factor, cardinality, is used as an input parameter of the second factor, the cost model. Therefore, improved cardinality leads to better estimated costs and, in turn, faster execution plans.

5. Cardinality Defined • Optimizer • Have you ever wondered why the “estimated” quantity of rows is not what you expected in an execution plan? • Cost based • Will choose the lowest estimated processing cost • Total number of rows processed at each level of a query plan. • Cost model of the algorithm dictated by the operators used in the query. • SQL Server estimates cardinalities • Sources – • Histograms when indexes are created • Constraint information • Logical rewrites of queries

6. Cardinality Defined • Magic? • Not quite • Cases where SQL Server cannot accurately calculate cardinalities. • Queries with predicates that use comparison operators between different columns of the same table. i.e. table.col1 > table.col2 • Queries with predicates that use operators, and any one of the following are true: • There are no statistics on the columns involved on either side of the operators. • The distribution of values in the statistics is not uniform, but the query seeks a highly selective value set. This situation can be especially true if the operator is anything other than the equality (=) operator. • The predicate uses the not equal to (!=) comparison operator or the NOT logical operator. • Queries that use any of the SQL Server built-in functions or a scalar-valued, user-defined function whose argument is not a constant value. • Queries that involve joining columns through arithmetic or string concatenation operators. • Queries that compare variables whose values are not known when the query is compiled and optimized.

7. Cardinality Defined • Enough • I’m sold! • What’s my cardinality, dude? • Demo query

8. Cardinality Experienced • Results • Takeaway • Great for profiling your data without “looking” at it all. • Great for ensuring that your testing sets are indicative of reality. • Great for determining which columns would be potential candidates for indexing, partitioning (date) and which wouldn’t. • That give me an idea… let’s look at adding more data to the result set! • Demo

9. Cardinality Experienced • Results • Takeaway • Great for profiling your data AND indexing without “looking” at it all. • Great for performance environments • Is your cardinality what you expect? • Are the indexes behaving the way that you expect? • Great for determining which columns would be potential candidates for indexing (clustered and non-clustered, partitioning (date) and which wouldn’t.

10. Cardinality Audience • QA – Performance Testing • Environment • Greenfield • Developer unit tests • Existing or mature environment • Prod  QA • Prod  Performance environment • Critical to accurate benchmarks • Prod  Development • Critical to writing t-sql that’s appropriate for your data, schema and cardinality

11. Cardinality Audience • Developers • Parameter Sniffing • Demo • The main points that you need to remember are: • If you find that the optimizer is picking different plans over time that have varying performance characteristics, consider using a parameter hint with a representative “average” value to get a good, common query plan that will work reasonably for all values. • If you are running really complex queries and need the plan choice to be exact in each case, you can consider using the “recompile” hint – just know that it will compile each time it runs, so this is likely more appropriate for longer-running queries to justify that compilation cost. • Moving a query into a stored procedure can put it into a separate procedural context and can be a good way to get that value visible to the optimizer (Note: this works in SQL 2000 as well) • Source: http://blogs.msdn.com/queryoptteam/Default.aspx?p=2

12. Cardinality Audience • Developers • Cases where SQL Server cannot accurately calculate cardinalities. • Queries with predicates that use comparison operators between different columns of the same table. • Potential resolutions include computed columns or another table • Queries with predicates that use operators, and any one of the following are true: • There are no statistics on the columns involved on either side of the operators. • Create statistics – ensure that “Auto Create Statistics” = True • The distribution of values in the statistics is not uniform, but the query seeks a highly selective value set. This situation can be especially true if the operator is anything other than the equality (=) operator. • Avoid the use of local variables. Instead, use param’s, literals or expressions. • The predicate uses the not equal to (!=) comparison operator or the NOT logical operator. • Always known as a performance killer… now you know why! • Queries that use any of the SQL Server built-in functions or a scalar-valued, user-defined function whose argument is not a constant value. • Queries that involve joining columns through arithmetic or string concatenation operators. • Queries that compare variables whose values are not known when the query is compiled and optimized. • BOL has additional descriptions

13. Cardinality Audience • Developers -- Continued • Constant Folding and Expression Evaluation During Cardinality Estimation • Foldable Expressions and Nonfoldable expressions • Foldable • Arithmetic expressions, such as 1+1, 5/3*2, that contain only constants. • Logical expressions, such as 1=1 and 1>2 AND 3>4, that contain only constants. • Built-in functions that are considered foldable by SQL Server, including CAST and CONVERT. • Nondeterministic functions are not foldable. Deterministic built-in functions are foldable, with some exceptions. • Exceptions are large data types (text, image, varchar(max), varbinary) • SQL Server does not fold expressions that involve user-defined functions, even if they are deterministic. • Non-Foldable • Nonconstant expressions such as an expression whose result depends on the value of a column. • Expressions whose results depend on a local variable or parameter, such as @x. • Nondeterministic functions. • User-defined functions (both Transact-SQL and CLR). • Expressions whose results depend on language settings. • Expressions whose results depend on SET options. • Expressions whose results depend on server configuration options. • BOL has additional descriptions

14. Cardinality Audience • Developers -- Continued • Constant Folding and Expression Evaluation During Cardinality Estimation • Expression Evaluation – At compile time • The following operators are also evaluated at compile time if all their inputs are known: • Arithmetic operators: +, -, *, /, unary -, • Logical Operators: AND, OR, NOT • Comparison operators: <, >, <=, >=, <>, LIKE, IS NULL, IS NOT NULL • Quick Demo

15. Cardinality – Query Processor • Optimizer – Things to keep in mind • Cardinality will be represented (often at 30%) by optimizer • Avoid the use of local variables in queries. Instead, use parameters, literals, or expressions in the query. • Limit the use of operators and functions embedded in a query that contains a parameter to those listed under Compile-Time Expression Evaluation for Cardinality Estimation. • Make sure that constant-only expressions in the condition of your query are either constant-foldable, or can be evaluated at compilation time. • If you have to use a local variable to evaluate an expression to be used in a query, consider evaluating it in a different scope than the query. For example, it may be helpful to perform one of the following: • Pass the value of the variable to a stored procedure that contains the query you want to evaluate, and have the query use the procedure parameter instead of a local variable. • Construct a string that contains a query based in part on the value of the local variable, and then execute the string by using dynamic SQL (EXEC or sp_executesql). • Parameterize the query and execute it by using sp_executesql, and pass the value of the variable as a parameter to the query.

16. Cardinality – Query Processor • Optimizer – Things to keep in mind • Trace flag 2301 • Available in 2005 SP1 • Enhanced modeling ability for optimizer • Increases compile times • Increases memory usage (sometimes severely) • Source: • http://blogs.msdn.com/ianjo/archive/2006/04/24/582219.aspx

17. Meta Data – Cardinality Explored • Questions? • Thank You! • Contact Information • Jeremy@DataRealized.com • http://wordpress.datarealized.com • http://datarealized.com • Twitter: DataRealized • LinkedIN: http://www.linkedin.com/in/jeremylowell • References • http://msdn.microsoft.com/en-us/library/ms175933(SQL.90).aspx • http://msdn.microsoft.com/en-us/library/ms181034(SQL.90).aspx