Navigation

Showing posts with label sql. Show all posts
Showing posts with label sql. Show all posts

Thursday, November 18, 2010

How to update a large SQL tables without killing the server!

Often I have the need to update a very large (>1 million rows) and a simple sql statement like this takes a long time;

update mytable set myfield = getdate() where myfield is null

This statement provides better performance (if you have triggers on the table disabling them if you can provides better performance also);

DISABLE trigger [dbo].[myTrigger] ON  [dbo].[myTable];
go
SET ROWCOUNT 10000

WHILE 1=1
BEGIN
UPDATE myTable
SET myField = GETDATE()
WHERE myField is null
IF @@ROWCOUNT = 0 BREAK

END
go
enable trigger [dbo].[myTrigger] ON  [dbo].[myTable];
go

By changing the “Set RowCount” value you can tune the statement as needed.

Thursday, November 11, 2010

How to determine if a table has changed in SQL

Often you need to know if a sql tables data has changed.  The change can be a new row, deleted row or updated row.  To determine if a row has changed you can use the following command that will return a checksum identifying a change to the table.
SELECT CHECKSUM_AGG(BINARY_CHECKSUM(*)) FROM <table name> WITH (NOLOCK);
You can also use a where clause.  For example get the checksum on just part of the table.

 

CHECKSUM

Returns the checksum value computed over a row of a table, or over a list of expressions. CHECKSUM is intended for use in building hash indices.
Syntax
CHECKSUM ( * | expression [ ,...n ] )
Arguments
*
Specifies that computation is over all the columns of the table. CHECKSUM returns an error if any column is of noncomparable data type. Noncomparable data types are text, ntext, image, and cursor, as well as sql_variant with any of the above types as its base type.
expression
Is an expression of any type except a noncomparable data type.
Return Types
int
Remarks
CHECKSUM computes a hash value, called the checksum, over its list of arguments. The hash value is intended for use in building hash indices. If the arguments to CHECKSUM are columns, and an index is built over the computed CHECKSUM value, the result is a hash index, which can be used for equality searches over the columns.
CHECKSUM satisfies the properties of a hash function: CHECKSUM applied over any two lists of expressions returns the same value if the corresponding elements of the two lists have the same type and are equal when compared using the equals (=) operator. For the purpose of this definition, NULL values of a given type are considered to compare as equal. If one of the values in the expression list changes, the checksum of the list also usually changes. However, there is a small chance that the checksum will not change.
BINARY_CHECKSUM and CHECKSUM are similar functions: they can be used to compute a checksum value on a list of expressions, and the order of expressions affects the resultant value. The order of columns used in the case of CHECKSUM(*) is the order of columns specified in the table or view definition, including computed columns.
CHECKSUM and BINARY_CHECKSUM return different values for the string data types, where locale can cause strings with different representation to compare equal. The string data types are char, varchar, nchar, nvarchar, or sql_variant (if its base type is a string data type). For example, the BINARY_CHECKSUM values for the strings "McCavity" and "Mccavity" are different. In contrast, in a case-insensitive server, CHECKSUM returns the same checksum values for those strings. CHECKSUM values should not be compared against BINARY_CHECKSUM values.
Examples
Using CHECKSUM to build hash indices
The CHECKSUM function may be used to build hash indices. The hash index is built by adding a computed checksum column to the table being indexed, then building an index on the checksum column.
-- Create a checksum index.
SET ARITHABORT ON
USE Northwind
GO
ALTER TABLE Products 
ADD cs_Pname AS checksum(ProductName)
CREATE INDEX Pname_index ON Products (cs_Pname)


The checksum index can be used as a hash index, particularly to improve indexing speed when the column to be indexed is a long character column. The checksum index can be used for equality searches.


/*Use the index in a SELECT query. Add a second search 
condition to catch stray cases where checksums match, 
but the values are not identical.*/
SELECT * 
FROM Products
WHERE checksum(N'Vegie-spread') = cs_Pname
AND ProductName = N'Vegie-spread'


Creating the index on the computed column materializes the checksum column, and any changes to the ProductName value will be propagated to the checksum column. Alternatively, an index could be built directly on the column indexed. However, if the key values are long, a regular index is not likely to perform as well as a checksum index.


See Also


BINARY_CHECKSUM


CHECKSUM_AGG








Reference; http://msdn.microsoft.com/en-us/library/aa258245%28SQL.80%29.aspx

Microsoft SQL Server Denali CPT1 Released!

The SQL server team at MS just released the first CPT of the next version of SQL Server code named Denali.
A few of the areas that the team focused on were;
· Enhanced mission-critical platform: An enhanced highly available and scalable platform through the new SQL Server AlwaysOn for greater flexibility in achieving increased availability and data protection and new Column-Based Query Accelerator for huge performance gains in data warehousing.
· Developer and IT Productivity: A new unified development experience for data developers called SQL Server Developer Tools code-named “Juneau”, new beyond relational enhancements including FileTable for file storage within the SQL Server database, easier of use delivered via SQL Server AlwaysOn, data integration tools and features such as security & supportability.
· Pervasive Insight: Expand the reach of BI to business users via Project code-name “Crescent”, a highly interactive, web-based data exploration and visualization tool built on the breakthrough analytical performance of the VertiPaq technology. Meanwhile, holistic data integration and management tools through enhanced Master Data Services and new Data Quality Services will help ensure organizations can deliver the right data to the right users at the right time.
· Select capabilities of the new integrated high availability solution, SQL Server code-named “Denali” AlwaysOn, including availability groups, single active secondary for offloading read-only workloads and multi-site clustering
· Flexible server roles to allow administrators to create custom roles for ease separation of duties
· Simplified development and standardized deployment, configuration and management for SQL Server Integration Services
· Column-Based Query Accelerator will help dramatically increase query performance ~10x and reduce performance tuning through interactive experiences with data for near instant response times and streamlined setup which removes the need to build summary aggregates.
You can download the CPT at; http://www.microsoft.com/downloads/en/details.aspx?FamilyID=6a04f16f-f6be-4f92-9c92-f7e5677d91f9&displaylang=en
Reference
http://www.microsoft.com/sqlserver/en/us/product-info/future-editions.aspx
http://blogs.technet.com/b/dataplatforminsider/archive/2010/11/10/microsoft-dives-deeper-into-sql-server-code-named-denali-the-next-version-of-sql-server.aspx

Thursday, November 04, 2010

SQL Server Compact Toolbox

SQL Server Compact Toolbox add-in for Visual Studio 2010. Supports both version 3.5 and 4.0 of SQL Server Compact.
Adds several features to help your SQL Server Compact development efforts:
- Script tables, including data
- Script entire schema, optionally with data, both of SQL Server Compact and SQL Server 2005 or later databases
- Import to SQL Server Compact from a SQL Server 2005/2008 database or a CSV file
- Migrate from SQL Server Compact to SQL Server and SQL Azure
- Basic, free form query execution
- Parse SQL scripts
- Display graphical estimated execution plan
- Generate detailed DGML files for visualizing table columns and relationships (requires VS 2010 Premium or higher to view)
- Create and manage Merge Replication subscriptions

For support, full source code, a standalone version for 4.0 and feedback, go to:http://sqlcetoolbox.codeplex.com

For users not having Visual Studio 2010 Pro or higher installed, an add-in for 3.5 with similar functionality for the free SQL Server 2008 Management Studio Express and also command line versions  are available  here:http://exportsqlce.codeplex.com

New in version 1.6:
- Export to SQL Azure
- Launch the Toolbox directly from Server Explorer!
- Beta testing of Db Diff (scripting of database differences)
- other bug fixes, see http://sqlcetoolbox.codeplex.com/SourceControl/list/changesets

alt

alt

alt

alt

Tuesday, July 27, 2010

SQL I/O Performance Measures

The following query will present you with the top 25 most i/o intensive queries your system has run.

SELECT TOP 25

q.[text],
(total_logical_reads/execution_count) AS avg_logical_reads,
(total_logical_writes/execution_count) AS avg_logical_writes,
(total_physical_reads/execution_count) AS avg_phys_reads,
Execution_count
FROM sys.dm_exec_query_stats
cross apply sys.dm_exec_sql_text(plan_handle) AS q
ORDER BY
(total_logical_reads + total_logical_writes) DESC


Monday, June 21, 2010

How to get index fragmentation of all indexes

SELECT ps.database_id, ps.OBJECT_ID,


ps.index_id, b.name,

ps.avg_fragmentation_in_percent

FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) AS ps

INNER JOIN sys.indexes AS b ON ps.OBJECT_ID = b.OBJECT_ID

AND ps.index_id = b.index_id

WHERE ps.database_id = DB_ID()

ORDER BY ps.OBJECT_ID

GO

Friday, May 14, 2010

How to disable or enable all triggers in a database

I am a big fan of database triggers especially for audit records.  The problem is sometimes we need to import data into a database and don’t want the triggers firing during the migration.

These two scripts allow you to disable and enable all the triggers in a database.

GO
DISABLE Trigger ALL ON ALL SERVER;
GO

GO
ENABLE Trigger ALL ON ALL SERVER;
GO

This post shows a script for displaying all the triggers in a database;

http://justinpdavis.blogspot.com/2010/04/get-all-triggers-in-sql-database.html

How to get row counts for all tables in a database

I often need to clean out sample data from a database but need to leave static reference data.  I find this script very useful to ensure that I have removed table data that should not remain.

SELECT '[' + SCHEMA_NAME(t.schema_id) + '].[' + t.name + ']' AS fulltable_name, SCHEMA_NAME(t.schema_id) AS schema_name, t.name AS table_name,
i.rows
FROM sys.tables AS t INNER JOIN
sys.sysindexes AS i ON t.object_id = i.id AND i.indid < 2 order by rows desc

Thursday, May 13, 2010

How to reorganize all indexes in a database

The DBCC DBIndex command works on one table at a time.  If you need to run DBCC DBIndex on all the tables in a database you can use the script below.

USE myDatabase
DECLARE @table varchar(200)
DECLARE Cursor1 CURSOR FOR
SELECT table_name FROM information_schema.tables
WHERE table_type = 'base table'
OPEN Cursor1
FETCH NEXT FROM Cursor1 INTO @table
WHILE @@FETCH_STATUS = 0
BEGIN
DBCC DBREINDEX(@table,' ',90)
FETCH NEXT FROM Cursor1 INTO @table
END
CLOSE Cursor1
DEALLOCATE Cursor1

While this script will work the Alter Index Reorganize|Rebuild command is the recommended method moving forward.

Tuesday, April 27, 2010

Get all triggers in a SQL Database

Ever need to get all the triggers in a SQL database.  This script will return them;

 

SELECT S2.[name] TableName, S1.[name] TriggerName,
CASE
WHEN S2.deltrig = s1.id  THEN 'Delete'
WHEN S2.instrig = s1.id THEN 'Insert'
WHEN S2.updtrig = s1.id THEN 'Update'
END 'TriggerType' , 'S1',s1.*,'S2',s2.*
FROM sysobjects S1 JOIN sysobjects S2 ON S1.parent_obj = S2.[id] WHERE S1.xtype='TR' order by TableName

Wednesday, April 21, 2010

Logging to a database with nLog

Nlog is a great tool for logging with in VB.Net applications(http://nlog-project.org/).  This post will discuss setting up nLog to write to both a local log file and a a SQL database.

First, we need to configure the log file;

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="
http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" internalLogFile="Nlog.log">
    <targets>
        <target name="file" xsi:type="File"   layout="${longdate}|${level}|${callsite}|${logger}|${threadid}|${windows-identity:domain=false}--${message} ${exception:format=message,stacktrace:separator=*"  fileName="c:\psnet\myapplication.log" />
    <target name="database" type="Database">
      <connectionString>
        Data Source=databaseservername;Initial Catalog=databasename;User Id=username;Password=password;
      </connectionString>
      <commandText>
        insert into system_logging(log_date,log_level,log_logger,log_message,log_machine_name, log_user_name, log_call_site, log_thread, log_exception, log_stacktrace) values(@time_stamp, @level, @logger, @message,@machinename, @user_name, @call_site, @threadid, @log_exception, @stacktrace);
      </commandText>
      <parameter name="@time_stamp" layout="${longdate}"/>
      <parameter name="@level" layout="${level}"/>
      <parameter name="@logger" layout="${logger}"/>
      <parameter name="@message" layout="${message}"/>
      <parameter name="@machinename" layout="${machinename}"/>
      <parameter name="@user_name" layout="${windows-identity:domain=true}"/>
      <parameter name="@call_site" layout="${callsite:filename=true}"/>

      <parameter name="@threadid" layout="${threadid}"/>
      <parameter name="@log_exception" layout="${exception}"/>
      <parameter name="@stacktrace" layout="${stacktrace}"/>

    </target>
    </targets>
    <rules>
        <logger name="*" minlevel="Info" writeTo="file"/>
    <logger name="*" minlevel="Info" appendTo="database"/>
    </rules>
</nlog>

There are a couple fields that need to be configured (in bold).  One adjustment for testing is setting the (internalLogFile="Nlog.log").  This will assist us in debugging any database issues we have as this is nLog’s own log file.

Next, we need to created a matching database table;

/****** Object:  Table [dbo].[system_logging]    Script Date: 04/21/2010 17:05:06 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[system_logging](
    [system_logging_guid] [uniqueidentifier] ROWGUIDCOL  NOT NULL,
    [entered_date] [datetime] NULL,
    [log_application] [varchar](200) NULL,
    [log_date] [varchar](100) NULL,
    [log_level] [varchar](100) NULL,
    [log_logger] [varchar](8000) NULL,
    [log_message] [varchar](8000) NULL,
    [log_machine_name] [varchar](8000) NULL,
    [log_user_name] [varchar](8000) NULL,
    [log_call_site] [varchar](8000) NULL,
    [log_thread] [varchar](100) NULL,
    [log_exception] [varchar](8000) NULL,
    [log_stacktrace] [varchar](8000) NULL,
CONSTRAINT [PK_system_logging] PRIMARY KEY CLUSTERED
(
    [system_logging_guid] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

ALTER TABLE [dbo].[system_logging] ADD  CONSTRAINT [DF_system_logging_system_logging_guid]  DEFAULT (newid()) FOR [system_logging_guid]
GO

ALTER TABLE [dbo].[system_logging] ADD  CONSTRAINT [DF_system_logging_entered_date]  DEFAULT (getdate()) FOR [entered_date]
GO

You must ensure that you give access to this table to the username you provide in the nLog configuration file.

In order to get emailed with error entries simple add this trigger to the system_logging table;

/****** Object:  Trigger [dbo].[LogEmail]    Script Date: 04/21/2010 17:06:42 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
-- =============================================
-- Author:        Justin Davis
-- Create date: 4/1/2008
-- Description:    Send email of log message
-- =============================================
ALTER TRIGGER [dbo].[LogEmail]
   ON  [dbo].[system_logging]
   AFTER insert
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;
    Declare @ToEmail varchar(100)
    Declare @Title varchar(100)
    Declare @logmessage varchar(8000)
    declare    @loglevel as varchar(100)   
    set @ToEmail = 'youremailaddress'
    set @Title = 'Error'
    set @loglevel = (select log_level from inserted)
    set @logmessage = (select
    'User Date:' + char(9) + char(9) + log_date + char(13) + char(10) +
    'Computer:'+ char(9) + log_machine_name + char(13) + char(10) + 
    'User:' + char(9) + char(9) + log_user_name + char(13) + char(10) + 
    'Level:' + char(9)+ log_level + char(13) + char(10) + 
    'Logger:' + char(9)+ log_logger + char(13) + char(10) +
    'Thread:'+ char(9) + log_thread + char(13) + char(10) +   
    'StackTrace:'+ char(9) + log_stacktrace + char(13) + char(10) + 
    'CallSite:'+ char(9) + log_call_site + char(13) + char(10) +
    'Message:' + char(9) + log_message + char(13) + char(10) + 
    'Exception:'+ char(9) + log_exception as 'emailmessage'   
    from inserted)
    if @loglevel <>'Info'
    EXEC msdb.dbo.sp_send_dbmail @recipients=@ToEmail, @body= @logmessage,  @subject = @Title, @profile_name = 'default'

END

Make sure that you configure SQL mail first.  You might get an exception in the nlog.log file like this;

2010-04-21 13:53:02.0729 Error Target exception: System.Data.SqlClient.SqlException: The EXECUTE permission was denied on the object 'sp_send_dbmail', database 'msdb', schema 'dbo'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at NLog.Targets.DatabaseTarget.DoAppend(LogEventInfo logEvent)
   at NLog.Targets.DatabaseTarget.Write(LogEventInfo logEvent)
   at NLog.LoggerImpl.Write(Type loggerType, TargetWithFilterChain targets, LogEventInfo logEvent, LogFactory factory)

If so you it is because the username in the nlog.config file does not have access rights to the send mail SP.  Run this script to fix.

EXEC msdb.dbo.sp_addrolemember @rolename = 'DatabaseMailUserRole'
    ,@membername = 'logger';
GO

 

References

http://nlog-project.org/