Showing posts with label SQL Server 2005. Show all posts
Showing posts with label SQL Server 2005. Show all posts

Thursday, June 17, 2010

Using SQL Transactions in Powershell

It occurred to me that I have not seen a write up on System.Transactions.TransactionScope usage in powershell. I have tried this on Powershell v2, not sure if this works in Powershell 1. So here it goes...

The example below just shows the usage, I leave it as an exercise to the reader to add in the database query code.

"Start. " + [DateTime]::Now().ToString();

try {
 $transScope = New-Object System.Transactions.TransactionScope;

 ## Insert Queries

 "Complete";
 $transScope.Complete();
}
catch [Exception] {
    #$_ | fl * -Force
 Write-Host $_.Exception.ToString();
}
finally {
 if ($transScope) {
  $transScope.Dispose();
 }
}

"Done. " + [DateTime]::Now().ToString();

A more complex example just waiting for SQL statements. It shows some nested transactions.
"Start. " + [DateTime]::Now().ToString();

try {
 $transScope = New-Object System.Transactions.TransactionScope;
 
 ## Insert Queries
 
 try {
  ## Performs queries that are not included in a transaction.
  $transScope2 = New-Object System.Transactions.TransactionScope([System.Transactions.TransactionScopeOption]::Suppress);
  
  ## Insert Queries
  
  $transScope2.Complete();
 }
 finally {
  if ($transScope2) {
   $transScope2.Dispose();
  }
 }
 
 ## Insert Queries

 try {
  ## Perform actions that should be included in a side transaction outside of the enclosing transaction
  $transScope3 = New-Object System.Transactions.TransactionScope([System.Transactions.TransactionScopeOption]::RequiresNew);
  
  ## Insert Queries
  
  $transScope3.Complete();
 }
 finally {
  if ($transScope3) {
   $transScope3.Dispose();
  }
 }
 
 ## Insert Queries
 
 "Complete";
 $transScope.Complete();
}
catch [Exception] {
    #$_ | fl * -Force
 Write-Host $_.Exception.ToString();
}
finally {
 if ($transScope) {
  $transScope.Dispose();
 }
}

"Done. " + [DateTime]::Now().ToString();

Friday, May 21, 2010

Use XQuery to convert a timestamp value to a Base64 string

SQL is a very powerful language, even with its limitations. SQL doesn't have the ability to convert Timestamp values to their base64 representations. However, you can use XQuery to do the conversion for you.

In the example below "Version" is of type Timestamp. I didn't notice any performance degradation during my testing.

SELECT  t1.Version
      , cast(N'' as xml).value('sql:column("t1.Version")', 'varchar(20)') 
FROM Table1 t1

This example opens up a new way to solve problems in SQL. It is intriguing to me to think about the other uses for leveraging XQuery to pick up SQL's slack.

Thursday, August 27, 2009

XQuery Rename/Replace Xml Node and Update K2 Workflow Process Instance Xml Fields

I recently ran across a case where a property in our application was re-termed due to a change in requirements. Updating the code was very easy, however it meant that all of the Xml documents needed to be updated and that was not going to be as easy. It would involve updating the serialized object xml documents in our workflow data repository and K2.

Out of curiosity I ran the process using the old Xml to see what would happen. Deserializing the old Xml documents resulted in everything after that renamed property being omitted from the deserialization and it didn't throw and error. That was unexpected, but good information to know.

As it turns out, updating the workflow data repository was easy. K2 has many issues which resulted in the K2 update sql. The code below shows how I accomplished this:
CREATE PROCEDURE [dbo].[usp_brock_RenameXmlNode]
AS
BEGIN
 -- UPDATE Workflow Instances
 UPDATE [WorkflowInstances] SET InstanceData.modify('
  insert element NewTag {/ObjectXML/OldTag/text()} after (/ObjectXML/OldTag)[1]
 ')
 WHERE  WorkflowID='7B4DEB62-D462-4FE8-A8E2-2057B5F31B19'
   AND InstanceData.exist('/ObjectXML/OldTag') = 1
 UPDATE [WorkflowInstances] SET InstanceData.modify('
  delete (//OldTag)[1]
 ')
 WHERE  WorkflowID='7B4DEB62-D462-4FE8-A8E2-2057B5F31B19'
   AND InstanceData.exist('/ObjectXML/OldTag') = 1
 
 -- UPDATE K2 Workflows
 -- Create Table Var and Populate
 DECLARE @K2ProcXml2XmlType table
 (
  ProcInstID INT PRIMARY KEY,
  WorkflowName VARCHAR(128),
  WorkflowID UNIQUEIDENTIFIER,
  InstanceData XML,
  IsModified BIT
 )
 PRINT 'POPULATE TABLE VAR';
 INSERT INTO @K2ProcXml2XmlType 
  SELECT pin.ID
     ,ps.[Name] WorkflowName
     , wi.WorkflowID
     ,CONVERT(XML,REPLACE(CAST(f.string as nvarchar(max)),'<?xml version="1.0" encoding="utf-8"?>','')) InstanceData
     , 0 --IsModified
    FROM [K2].[dbo]._FieldOnDemand f INNER JOIN [K2].[dbo]._ProcInst pin ON f.ProcInstID=pin.ID
    INNER JOIN [DemoDatabase].dbo.WorkflowInstances wi ON wi.ProcessID=pin.ID
    INNER JOIN [K2].[dbo]._Proc p ON pin.ProcID = p.ID
    INNER JOIN [K2].[dbo]._ProcSet ps ON p.ProcSetID = ps.ID
  WHERE  (ps.[Name] in ('WorkflowX'))

 UPDATE @K2ProcXml2XmlType
 SET InstanceData.modify('insert element NewTag {/ObjectXML/OldTag/text()} after (/ObjectXML/OldTag)[1]'), IsModified = 1
 WHERE (InstanceData IS NOT NULL) 
  AND (InstanceData.exist('/ObjectXML/OldTag') = 1)

 UPDATE @K2ProcXml2XmlType
 SET InstanceData.modify('delete (//OldTag)[1]'), IsModified = 1
 WHERE (InstanceData IS NOT NULL) 
  AND (InstanceData.exist('/ObjectXML/OldTag') = 1)


 -- End Result of all updates
 --PRINT 'QUERY RESULT:';
 --SELECT
 --        [ProcInstID]
 --   , WorkflowName
 --   , InstanceData
 --  FROM @K2ProcXml2XmlType src
 --WHERE src.IsModified = 1;

 -- Populate Original Table and Query
 PRINT 'POPULATE TABLE FROM VAR:';
 UPDATE dst
 SET dst.String = CONVERT(ntext,'<?xml version="1.0" encoding="utf-8"?>'+CAST(src.InstanceData as nvarchar(max)))
 FROM @K2ProcXml2XmlType src INNER JOIN [K2].[dbo].[_FieldOnDemand] dst ON src.ProcInstID = dst.ProcInstID
 WHERE src.IsModified = 1;

        -- END K2 Update
END;
GO

exec dbo.usp_brock_RenameXmlNode

DROP PROCEDURE dbo.usp_brock_RenameXmlNode
It runs quite fast.

There are a couple caveats, you can't create the XPath location dynamically (unless you are using dynamic sql which I try to avoid at all costs). This means that you can't pass old and new name parameters into the stored procedure and update the records. Disappointing as it is, any future renames should require an easy find and replace.

I tried using a cursor to do this type of updating, but the resulting SQL in execution times in more than more than 90 minutes (I cancelled it at that point) and verified with the DBA's that I was not doing something wrong, I abandoned that path and went back to the set based updates.

Wednesday, August 19, 2009

SQL Server 2005 Extract Bitwise Flags

UPDATE: 8/20/2009 - This post has been updated several times, so read through for all of the information. I ran into a scenario where I had a numeric field which was composed of flags and I needed to extract them. I was told that you couldn't do bitwise (AND/OR) operations, but some undocumented bitwise system functions existed (like fn_IsBitSetInBitmask). Trying to dig into the system function was not very easy and having no real documentation outside of the "exec sp_helptext 'fn_IsBitSetInBitmask'" query, I wasn't very successful.

I started to have memories from my Digital Logic class where we had to do something similar using modulus with a constrained feedback loop. I pseudo-coded an example function, but found that SQL doesn't support arrays (thinking now, I probably could have used a table variable, but that would have needlessly complicated the situation). Regardless of SQL's deficiencies, using the modulus/feedback method does work, but wasn't going to fit my vision of the solution.

function BitFlags(int val) {
 bool[] arr = new bool[8];
 int modulus;
 for(int i=7; i >= 0; i--) {
  int pwr = (2^i);
  if (val == pwr) {
   arr[i] = true;
  } else if (val > pwr ){
   modulus = val % (2 ^ i);
   if (modulus != 0) {
    arr[i] = true;
    val = modulus;
   }
  }
 }
 return arr;
}

For those who haven't used their math skills in a while, here is an example. My Digital Logic skills are a bit rusty, so I may be incorrect, feel free to correct me if I am wrong. We want to find the bit flags for the number 76; using the modulus/feedback method, we get something like:

// Background
2^7 = 128
2^6 = 64
2^5 = 32
2^4 = 16
2^3 = 8
2^2 = 4
2^1 = 2
2^0 = 1

// Now the fun part, cycle through the range 0-7 for 2^i:
// Calc         Output    Description
76 % 128 = 0    [0]    // Feedback 76
76 % 64  = 12   [1]    // Feedback 12
12 % 32  = --   [0]    // 12 is smaller than 32, Feedback 12
12 % 16  = --   [0]    // 12 is smaller than 16, Feedback 12
12 % 8   = 4    [1]    // Feedback 4
4  % 4   = --   [1]    // 4 == 4, so the we get a 1 for the output and there is nothing to feedback, so everything past that point is 0
0  % 2   = --   [0]    // 0 is smaller than 2, Feedback 0
0  % 2   = --   [0]    // 0 is smaller than 2, Feedback 0

Something just did not settle well with me. You just had to be able to do bitwise operations; it seems like elementary funcitonality. I took my C skills and tried to them out in SQL. It threw me for a loop when after evaluating the @val & <number> expression that I would get back the <number> where I was expecting a 1, but 0's everywhere else. I new I was on the write path and was almost there. I did some googling and found:

http://www.sqlusa.com/articles2005/binarypattern/

Which was doing something similar, but didn't exactly fit the situation. It gave me a DUH! moment when I realized that I should just divide by the <number> and POOF, I had the elegant solution I was expecting.

Try it out for your self.
DECLARE @val as tinyint
SET @val = 128+64+16+1

SELECT  (@val&128)/128
      , (@val&64)/64
      , (@val&32)/32
      , (@val&16)/16
      , (@val&8)/8
      , (@val&4)/4
      , (@val&2)/2
      , (@val&1)/1

--RESULT:
--11010001

UPDATE: My good friend Christopher Lauer used my code and showed me how he used it. I did not find anything like this when I googled and thinking that it would be useful for other people he gave me permission to include it.

USE MSDB;

if not exists (select * from sys.schemas where name = 'my_utility')
 exec('create schema my_utility')
go

/*****************************************************************************************
 DESCRIPTION: The MSDB.dbo.sysSchedules has a column named freq_interval, this 
 column may contain a packed bit that needs unpacked to understand what days
 the schedule is set to run on.  See books on line for information on this
 table and column.  This function will unpack the bit and return a string 
 of weekday names that the bit respresents.

 Base code provided by Brock Moeller
 --SELECT [Monday] = (@bitWise&2)/2, [Tuesday] = (@bitWise&4)/4, [Wednesday]= (@bitWise&8)/8, [Thursday] = (@bitWise&16)/16, [Friday] = (@bitWise&32)/32, [Saturday] = (@bitWise&64)/64, [Sunday] = (@bitWise&128)/128

 MAINTENANCE: 
 [date:name]      [description of the change/maintenance being done]
 ==============================  ==================================================
 8/19/2009: Christopher M. Lauer  Created function.
******************************************************************************************/
CREATE FUNCTION [my_utility].[udf_getDayNameFromBitWise](@bitWise AS INTEGER)
RETURNS Varchar(62) AS
BEGIN
 --SET NOCOUNT ON;
 /*
 DECLARE @bitWise INT;
 --SET @bitWise = 63; --Monday, Tuesday, Wednesday, Thursday, Friday
 */

 /***** LOCALS ******/
 DECLARE @ErrorCode INT, @RowsAffected INT;
 SELECT @ErrorCode = 0, @RowsAffected = 0; -- default
 /***** END LOCALS ******/

 DECLARE @WeekTable TABLE(dName VARCHAR(10), isUsed bit, dNumber tinyint, bitWise int)
 INSERT INTO @WeekTable(dName, isUsed, dNumber, bitWise)
 SELECT 'Monday', (@bitWise&2)/2, 1, 2
 UNION ALL 
 SELECT 'Tuesday', (@bitWise&4)/4, 2, 4
 UNION ALL 
 SELECT 'Wednesday', (@bitWise&8)/8, 3, 8
 UNION ALL 
 SELECT 'Thursday', (@bitWise&16)/16, 4, 16
 UNION ALL 
 SELECT 'Friday', (@bitWise&32)/32, 5, 32
 UNION ALL 
 SELECT 'Saturday', (@bitWise&64)/64, 6, 64
 UNION ALL 
 SELECT 'Sunday', (@bitWise&128)/128, 7, 128
 --SELECT * FROM @WeekTable

 DECLARE @ReturnValue as VARCHAR(50)
 SELECT @ReturnValue = COALESCE(@ReturnValue, '') + dName + '; ' 
 FROM @WeekTable
 WHERE isUsed = 1;
 
 IF(LEN(@ReturnValue) > 1) 
    begin
   SET @ReturnValue = substring(@ReturnValue,1,len(@ReturnValue) -1)
    end
 RETURN @ReturnValue
END
UPDATE: 8/20/2009 - I had a little free time and discovered an odd occurrence with bitwise operations.
DECLARE @val as tinyint
SET @val = 128+64+16+1
SELECT (@val&10000001)
--RESULT 129
SELECT (@val&01000001)
--RESULT 65

SELECT (@val&00100000) -- @val&32
--RESULT 128  ---- Incorrect!  Should be '0'.  
It appears that SQLServer 2005 implicitly interprets the number as binary. I can't find any documentation on this issue. The results are accurate for scenarios where what you are looking for matches. However, if you test on something that doesn't match the left value, it doesn't return the correct number. I suppose an answer will have to wait.

On a side note, hexadecimal numbers work as expected like their decimal counterparts.
DECLARE @val as tinyint
SET @val = 128+64+16+1
SELECT (@val&0x40)
--RESULT 64

SELECT (@val&0x20)
--RESULT 0
It still appears that sticking with base 10 is easiest, but if you are good with hex you may find that usefull.

Saturday, February 28, 2009

SQL Server 2005 Table Row Counts... One Select Statement, No Table Scan

A big change that occurred with SQL Server 2005 (from 2000) is that the system table information is always in sync with the actual tables.

When the system tables get out of sync in SQL Server 2000 the following SQL statement must be run to synchronize them.
dbcc updateusage(0)
GO 
According to the MSDN site on DBCC UPDATEUSAGE (Transact-SQL) contains this little bit of information:
Reports and corrects pages and row count inaccuracies in the catalog views. These inaccuracies may cause incorrect space usage reports returned by the sp_spaceused system stored procedure. In SQL Server 2005, these values are always maintained correctly. Databases created on SQL Server 2005 should never experience incorrect counts, however, databases upgraded to SQL Server 2005 may contain invalid counts. We recommend running DBCC UPDATEUSAGE after upgrading to SQL Server 2005 to correct any invalid counts.
Googling for information on getting table counts without a row scan yields several posts on the issue, but they involve creating functions and needlessly complicating a simple SQL statement.

There are 2 possible solutions. 1) Common Table Expressions and 2) Use a join. The first option looks like this:
WITH CTETableRowCount
AS
(
 SELECT 
   rows rowCnt
 , ID ObjectID 
 --, si.*
        FROM sysindexes si
        WHERE indid < 2
)
SELECT 
  so.[name]
, cte.[rowCnt] as [Row Count]
--, so.*
FROM sysobjects so LEFT JOIN CTETableRowCount cte ON so.id=cte.ObjectID
WHERE type='U' and so.name != 'dtproperties'
ORDER BY so.[name]
This works and is reliable, but a bit overkill. There is nothing in the Common Table Expression that really requires it to be like that and in fact just adds lines to the query with no benefit The above query can be simplified to the following:
SELECT 
   so.[name]
 , si.[rows] as [Row Count]
 --, so.*
FROM sysobjects so LEFT JOIN sysindexes si ON so.id=si.ID
WHERE (type='U') and (so.name != 'dtproperties') and (si.indid < 2)
ORDER BY so.[name]
This is probably the best solution to the issue.

Tuesday, February 24, 2009

Duplicate Entries in a Mapping Table with All Guid/Uniqueidentifier Columns

I recently ran into a case where a custom .net membership provider was incorrectly adding a record to a map table every time a user was updated. This wouldn't normally be a problem, except SharePoint's user security has a couple hang-ups when the number of groups doesn't match the distinct group list.

Needless to say, there was a LOT of duplicate records in the table; more than I was willing to weed through by hand. After performing the obligated web search, I couldn't find anything that fit my situation. Common Table Expressions looked promising, but had several limitations that I wasn't able to work around.

The problem could be prevented from the beginning with the correct keys/constraints, but that was not the case and I needed to clean up the table before making the correction.

The first bit of information that tells me how I am going to solve the issue is the DBMS. In this case it is an SQL Server 2005 instance.

Lets setup an example, you have a table with 3 "uniqueidentifier" columns: "ID", "FromMap", "ToMap".
CREATE TABLE [dbo].[DuplicateGuidMap](
 [ID] [uniqueidentifier] NOT NULL,
 [FromMap] [uniqueidentifier] NOT NULL,
 [ToMap] [uniqueidentifier] NOT NULL
) ON [PRIMARY]
Now lets populate the table. You can skip this part, nothing unusual here.
INSERT INTO [Test].[dbo].[DuplicateGuidMap]
           ([ID]
           ,[FromMap]
           ,[ToMap])
SELECT '0f6f2339-4597-4933-b8b0-de04804f4c04','e0bb5dad-f30e-4bcb-a4ec-12363af0e6cd','8a15fde9-f9f3-4ecb-acd4-9c78274f9d95' UNION ALL SELECT
'1a0d0991-6a12-4621-901d-0ca58a4fbf5a','88c3d3aa-663c-41e9-bdcf-f9022604b5fc','d4757f79-3452-4a3e-a4d8-1f2ea3a26c2a' UNION ALL SELECT
'14bc9408-9537-4d2a-b917-6e1b44099799','778cffba-a14b-447f-8648-96d640ae1b93','dd20127d-33ab-4d91-9202-8196a2ca90d6' UNION ALL SELECT
'3801a162-cdef-4ffc-b4b5-928f345fe68f','77121133-8c4f-4e28-b9fa-048707c41a7a','faae6e93-e91d-4729-8b69-367401130a42' UNION ALL SELECT
'e8386333-0e89-4ec0-8b2a-e16ae9544290','d385ea64-68f9-421c-bda0-3c0d9fd785e8','442fd8b3-a5d3-4074-9886-e9104d0f2421' UNION ALL SELECT
'a4b5dacb-3f21-43ea-b516-e7055c3831d1','10b7ec11-0cb8-4128-a146-2ca2ef0a9594','c09aa0e7-96fd-41fa-8fce-105a4ce908ee' UNION ALL SELECT
'2c340b29-cd5d-4036-92af-7ae24bdc47d9','dde6379c-6626-46c0-ac70-b7ddd0377e62','5ac17542-fc1d-4581-a616-5dd602e5f34f' UNION ALL SELECT
'd6d56501-98e0-4662-a72c-116b6e2a2e4c','08d0200d-375a-4022-af9e-bdc5ff4a57fe','4e001ae6-d8a0-45b5-a30a-1ab95b66dae4' UNION ALL SELECT
'ee392830-d077-48c0-a817-5a2169dc8bb6','abd1ed2d-3f4d-4f8d-9240-7aebc46ff021','3fdc2214-e11d-411d-b524-7cdd57037f6c' UNION ALL SELECT
'a37b664d-1735-444b-a326-2d23f8bf0b55','c0a1a8a0-efdd-4740-807a-f5f0aaba90ed','f95a288d-3e6c-43a4-9b59-54d382825e99' UNION ALL SELECT
'1d8a2558-0f50-4765-98a9-dd2cf2127bcf','e0bb5dad-f30e-4bcb-a4ec-12363af0e6cd','8a15fde9-f9f3-4ecb-acd4-9c78274f9d95' UNION ALL SELECT
'577a05de-4bbc-4b8d-93dc-22f9dd838e43','e446232a-02bb-47a0-b9a1-e5fab5c0aba3','faae6e93-e91d-4729-8b69-367401130a42'
We will assume that the relationship of "FromMap" to "ToMap" is 1 to Many, meaning that each "ToMap" should be unique. A good place to start would be to find out how many "ToMap" duplicates there are.
SELECT DISTINCT ToMap
  FROM DuplicateGuidMap
GROUP BY ToMap
HAVING Count(ToMap) > 1
There are 2 potential sets of duplicate records in the table now. ToMap:
  • FAAE6E93-E91D-4729-8B69-367401130A42
  • 8A15FDE9-F9F3-4ECB-ACD4-9C78274F9D95
Getting the list of all of the duplicated records is a trivial join, but I want a query that would return only the duplicate rows (leaving one of the duplicated records). Many DBA's who have worked for years would use cursors and possibly create functions. This is all well and good, but they are rather slow and consume a lot of CPU time. The idea of one SQL statement to rule them all doesn't come to mind. I tend to think in sets. I haven't found too many things I can't do in one Select statement. Knowing this, I banked heavily on my understanding of how SQL works and created the following query.
SELECT 
        dgm.ToMap
   , dgm.FromMap
   , dgm.ID
  FROM DuplicateGuidMap dgm
WHERE ToMap IN (
 SELECT DISTINCT ToMap
   FROM DuplicateGuidMap
 GROUP BY ToMap
 HAVING Count(ToMap) > 1
) AND ID NOT IN (
 SELECT TOP 1 ID
   FROM DuplicateGuidMap
 WHERE ToMap = dgm.ToMap
)
ORDER BY ToMap
The meat of the query lies in the Where clause.
ID NOT IN (
 SELECT TOP 1 ID
   FROM DuplicateGuidMap
 WHERE ToMap = dgm.ToMap
)
What's happening here? Since we are currently in the Where clause, we are banking on the fact that it is evaluated for every row that passes the first condition. So in the subquery, we are selecting the record we want to keep. The Where clause "ToMap = dgm.ToMap" grabs the "ToMap" in the main query for use in the subquery. This functionality works almost the same way as a SQL function without having to load and unload the function from memory every time. Having this included in the statement makes the query run faster. In a table with ~10,000 records, ~79 unique duplicates, and ~196 duplicates to remove the query took "00:00" (min:sec).

There is a couple points to make before we proceed. The Guid 'FAAE6E93-E91D-4729-8B69-367401130A42' case is not a valid dupicate and never occurred in the scenario. I added this case to dive farther into controlling which duplicates are actually valid. The only valid duplicate is '8A15FDE9-F9F3-4ECB-ACD4-9C78274F9D95' because the "FromMap"'s are the same. This is the solution I developed for originally.

We need to look at the "ID" Where clause. We can refine it to be: "FromMap = dgm.FromMap AND ToMap = dgm.ToMap".

The final query looks like the following. It returns a list of the duplicate records ready to delete leaving one of the duplicate records.
SELECT 
        dgm.ToMap
   , dgm.FromMap
   , dgm.ID
  FROM DuplicateGuidMap dgm
WHERE ToMap IN (
 SELECT DISTINCT ToMap
   FROM DuplicateGuidMap
 GROUP BY ToMap
 HAVING Count(ToMap) > 1
) AND ID NOT IN (
 SELECT TOP 1 ID
   FROM DuplicateGuidMap
 WHERE FromMap = dgm.FromMap AND ToMap = dgm.ToMap
)
ORDER BY ToMap
Never stop thinking outside the box. Most of the time you can find a simple answer with a little extra thought.