Thursday, November 8, 2012

Extract Oracle Blob Image to Physical Files on Disk using SSIS

Are you looking for solution to extract Oracle Blob Images to Physical files on Disk using SSIS ?

If yes, then here is the solution.

You can use OLEDB connector to connect to Oracle system , but to gain performance Attunity has came up with connector plugin which is freely distributed on microsoft site ( no license required).

3 steps  in data flow are  :
1)  use Attunity Oracle Connector as source connecting to Oracle DB . It can be downloaded from below link :

2) Derived column which maps base file path ( ex c:\) + File name of file ( we can pull from table or generate new id ) + file mime type ( we can pull from source table  ex: jpg, png etc.)

3) Destination Export column.

Link to Download Attunity Connector


Hope it helps...







Saturday, August 11, 2012

WEBAPI Mesage Handler

I have read an article on WebAPI Message Handler which is very cool. I am not giving any details of it , thinking of duplicate effort but here is the link for it :
Click Here


Thanks to Aliostad for presenting wonderfull article.

Monday, July 30, 2012

SQL 2012 : Column Store Index

Column store Index is one of the new feature introduced by Microsoft in sql server 2012 edition.
This is a new kind of indexing which provides significant performance for queries & ssas for aggregations etc.
In a regular indexing , each row is stored on a page where as in the column store index each column is stored per page which makes it much more efficient for queries such a aggregations etc.. which will have specific columns in it. They are highly compressed , so easy to store large volume of data in  memory resulting in very low IO cost.

How ever there are some limitations , few are mentioned below :
1) they are read only, once created  inserts/ updates are not possible, this can be used in warehousing applications where data is loaded periodically & before data load index can be dropped or disabled & then created once the data load is finished.
2) there can be only one column store index per table
3) it cant be clustered index.
4) cant be pk  or fk
5)cant be used with tables that has cdc or file stream

Reading material :

http://social.technet.microsoft.com/wiki/search/searchresults.aspx?q=SQL+Server+Columnstore
http://blog.sqlauthority.com/2011/10/29/sql-server-fundamentals-of-columnstore-index/
http://blog.sqlauthority.com/2011/11/07/sql-server-video-performance-improvement-in-columnstore-index/
http://blog.sqlauthority.com/2011/11/06/sql-server-updating-data-in-a-columnstore-index/
http://blog.sqlauthority.com/2011/10/30/sql-server-how-to-ignore-columnstore-index-usage-in-query/

SQL Merge + SCD ( slowly changing dimensions)

Sql server Merge is a powerful t-sql statement which can be used to write simple code to do insert, update, delete in one statement instead of writing 3 different statements one for update, one for insert & one for delete.  Using Merge is cleaner & much more efficient when it executes.

Syntax looks like this :


merge in to  dbo.destination as d
using dbo.source as s
on (s.pk= d.pk)
when matched
            then update set d.col1= s.col1
when not matched
         then insert....

How ever Merge can do much more than what is seen here, There is an excellent presentation by Kimball group about how Merge can be used in SCD scenario :

http://www.kimballgroup.com/html/08dt/KU107_UsingSQL_MERGESlowlyChangingDimension.pdf


Thursday, July 26, 2012

Real Time Data Push to SSAS Cube

Real time data push in to SSAS cube is very much possible.

There are different ways of doing it, How ever you have explore & see which one best fits your needs

The below are some of the options :

1) Push the data directly from the WH in to SSAS cubes using SSIS Package.
ex: Lets say there is a product dimension & Orders Fact table , in the SSIS package which loads the data in to these tables we can extend them to push the data to Product dimension & Orders Fact  of OLAP cube.

2) Proactive Caching based on policy based management. The Policy settings  - SilenceInterval, SilenceOverrideInterval, ForceRebuildInterval & latency play very important role.

But how do we get the data changed notifications from the data source, Here are the options :
 1) use event trace entries
 2)  application can notify through webservice call
3) by polling the data source using sql queries to see if data has changed using a timestamp or any other column.


Umbraco + MVC

After the announcement from Umbraco team that V5 was ripped off, we were left with version 4.7 which is latest version we could have at the point of this post is published.
we were not left with much options . so we went ahead with what we have & utilised MVCBridge package installer which gives a way to create a page in Umbraco & give the controller , action names to the call. with this we were able to integrate our existing MVC webapp in to Umbraco ..

We thought issue was resolved , how ever we boggled down with Ajax Calls, then we  went ahead and modified the source code of MVCBridge to make ajax calls with just Controller , Action as we don't have access to HTMLHelper classes & used the Umbraco /base ajax calls with parameters controller & action which will then make a call to routine call from MVCBridge. Even though we don't have helper class to pass parameters still we can pass parameters to ajax  /base class as they will be available in Context object for MVC call.

Hope this helps ..

MOLAP ROLAP HOLAP

MOLAP, ROLAP, HOLAP are terms you will come across in SSAS .
Keeping it simple what they mean are :

MOLAP is Multidimensional OLAP : what ever aggregations, group by etc you can imagine on the query to generate the counts are pre-aggregated & calculated before hand  in cube , so that the execution is fast & don't have to calculate when a request is made.

ROLAP is Relational OLAP : when ever a query is issued , they will be executed against the relational databases to return the resultset. you can imagine this as a stored procedure call.

HOLAP is Hybrid OLAP : It is a Hybrid of above two, HOLAP is used where aggregations, group by etc are used to return the result set & MOLAP is used for regular queries. Sharepoint deployment of reports built on SSAS cubes & SSRS reports mostly use HOLAP.  MOLAP for main repots & HOLAP for drill down reports.


Sunday, January 8, 2012

Message[264] An attempt was made to send an email when no email session has been established

When we configure the database email, we can successfully send the test email.
How ever when we configure to set the alerts to send out email it will fail with the error
Message[264] An attempt was made to send an email when no email session has been established
which is found @ sql server agent error logs.

Fix :

select sql server agent -> right click -> properties -> alert system ->  check enable mail profile
-->  select mail system as Database Mail. --> Mail profile. --> ok  --> restart sql server agent




Error : The specified @server_name does not exist

When we rename the machine name by  default it will just rename the machine name ,where as the sql server still holds the reference to old machine name.When you try to add monitoring jobs or any thing else which needs to take the machine name in to consideration then it will try to look for the old machine name which is unable to access. This error is thrown at that instance.

Fix :

select @@servername , if it is different from machine name
then drop the old & add the new one.


sp_dropserver 'old machine name'
sp_addserver 'new machine name', local
restart the sql server

Saturday, January 7, 2012

SQL Server Configuration Manager Error: Connection to target machine could not be made in a timely fasion

I Faced this error when i was playing  with db mirroring & screwed up the database mirroring settings.
I was not able to open the sql server configuration manager .

Fix is :

go to start -> run -> type service.msc--> right click Windows Management Instrumentation & restart.



Monday, January 2, 2012

Sql Server Log Shipping

Sql server has log shipping as one of the disaster recovery solution. Log shipping provides the automated way of taking the log back ups on one server & restore them on another server.
Log shipping in general will have one primary server with Primary Database , Multiple secondary servers where the log will be shipped to & monitor server used to monitor the activities & raise the alerts when some thing goes wrong. We can use either primary server (or) secondary server to serve the purpose of Monitoring , How ever if some thing goes wrong on the server  then there could be chance to loose the monitoring as well, that's why it is suggested to have a seperate dedicated server for monitoring.

When logshipping is configured ,   4 jobs will be created on the primary , secondary & monitor servers.

1) Back up  job is created on  primary server -  purpose of this job  is to take a database back up

2) Copy job is created on secondary server - purpose of this job  is to copy the back up database to secondary server location

3) Restore job is created on secondary server - purpose of this job is to restore the back up log file that is copied over to secondary server in step 2

4) Alert Job is created on either secondary server (or) monitor server based on where the monitoring is configured , purpose of this job is to alert if some thing goes wrong in the process.

Sql server agent which runs these jobs should have read/ write permission to the location where the back ups will be taken  , copied to.

In this presentation i will show the implementation of logshipping by taking  one default instance & other named instance on the same machine.

1) create a database on primary server.

 CREATE DATABASE [testdb] ON  PRIMARY
( NAME = N'testdb', FILENAME = N'E:\SQLDATA\testdb.mdf' , SIZE = 2048KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB )
 LOG ON
( NAME = N'testdb_log', FILENAME = N'G:\SQLLOG\testdb_log.ldf' , SIZE = 1024KB , MAXSIZE = 2048GB , FILEGROWTH = 10%)
GO

2)  take back up of the database

 BACKUP DATABASE [testdb] TO  DISK = N'E:\SQLBACKUPS\testdb.bak' WITH NOFORMAT, NOINIT, 
 NAME = N'testdb-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO

3) restore the database on second server 
RESTORE DATABASE [testdb] FROM  DISK = N'E:\SQLBACKUPS\testdb.bak' WITH  FILE = 1,
  MOVE N'testdb' TO N'E:\SQLDATA\INST1\testdb.mdf',  
  MOVE N'testdb_log' TO N'G:\SQLLOG\INST1\testdb.ldf',
    STANDBY = N'E:\SQLBACKUPS\INST1\ROLLBACK_UNDO_testdb.BAK',  NOUNLOAD,  STATS = 10
GO

4) configuring the primary database : right click the database -> properties -> transaction logshipping
  set the network path where the backups has to be taken. Edit job to schedule the time how frequent the back ups has to be taken.
 5)  Add the secondary server instance , here we can add as many secondary servers as we want .
 6) Restore Transaction Log can be set to  No recovery mode (or) stand by mode . Advantage with stand by mode is the database can be used in read only mode . This is useful for reporting purposes.

7)  Configure the Monitor server , here i am using the named instance as both secondary & monitor server.

















8) Now we will have 4 sql jobs one on primary & rest on secondary servers.

































9) Thats all,  Logshipping is configured.

Monitoring Logshipping :
  Logshipping can be monitor 3 ways

  1) GUI :  Right click on the server instance -> reports -> transaction logshipping
  2) system stored procedures.

      sp_help_log_shipping_monitor_primary 'satishbudati-pc', 'testdb'
     sp_help_log_shipping_monitor_secondary 'satishbudati-pc\inst1', 'testdb'

 3) msdb tables :
 
  select * from msdb..log_shipping_monitor_primary
  select * from msdb..log_shipping_monitor_secondary

Additional Reading Material : http://sqlcat.com/sqlcat/b/technicalnotes/archive/2009/02/24/scheduling-sub-minute-log-shipping-in-sql-server-2008.aspx

Tuesday, December 27, 2011

Sql Server Moving System Databases


Today i am going to discuss  how can we move the system database files ( data file - mdf  & log file  ndf ) to a different location with in the same machine. There will be some exceptions which will be discussed at the bottom

Different scenarios where we need to move the files are :

1) Planned relocation - ex:  you installed it on c drive & realized that those files should not be there on OS drive :)
2) Relocation for scheduled disk maintainance
3) Failure recovery - hardware recovery
  could be many more ..

Steps :

1) Identify the files that are used by a  database & you want to move  :









2) Create the files in the new location  using command for each file :
   use master
go
alter database tempdb modify file (name='tempdev', filename='c:\tempDB.MDF', size = 1mb)
go

3) Stop the sql server engine either from sql server management studio (or)  from sql server management console ( sqlservermanager10.msc   in run command  for sql server 2008 )

4)  copy the files from old location to new location.

5) restart the sql server .  From the restart time the sql server will use the files from the new location.

6) To verify if sql server  is using the files from  new location , you can run the query to find the location :
    SELECT name, physical_name AS CurrentLocation, state_desc
FROM sys.master_files
WHERE database_id = DB_ID(N'tempdb');

Exceptions :
1)  For Tempdb , we need not restore the files to new location as for every sql server restart tempdb files will be initialized . Even if we don't restore the files, still sql server will create new files in the new location.
2)  Resource database can't be moved
 3) For master database , we have to change the start up parameter to point to new location
    sql server configuration manager --> sql server services --> right click --> properties --> advanced --> startup
 This has to be done before we restart the service.














4) for msdb database, make sure you check the service broker is enabled , if database mail is configured.
Database Mail uses msdb database.

SELECT is_broker_enabled
FROM sys.databases
WHERE name = N'msdb';

If there is a Hardware failure , then moving files follows a seperate logic :
1) Stop the sql server instance
2) Start the instance in master only recovery mode by typing the cmd from command prompt :
      
NET START MSSQLSERVER /f /T3608   ( for default instance )     
    NET START MSSQL$instancename /f /T3608  ( for named instance )
 
3) for each file, run the alter database command as mentioned above .
4) Stop the instance  (  NET STOP MSSQLSERVER ) 
5) move the files to new location 
6) start the instance  ( NET START MSSQLSERVER )
7) check if service is using new location or not as mentioned above.
Happy Reading.. Hope this helps.. 

Sunday, December 25, 2011

Sql Server Isolation Levels

Isolation level is one of the property of ACID . Sql server has different levels of Isolations implemented using locks .
The Term Isolation defines Isolating the transaction & doesn't having impact on other transactions happening in a different user session.
Sql Server 2008 defines different isolation types to cater the needs from High Concurrent executions to  the accuracy of the data.

What is the default Isolation Level of Sql Server ?
Well .. it is Read Committed.  So by default if you do not specify any isolation level for a query then it will take default isolation level of Read Committed.

to find the default Isolation Level execute the query :
dbcc useroptions




















Before going in to the isolation levels  , let us see some of the locks that sql server uses during transactions :

Row Level Locks :

 shared locks (S) , Update locks (U) , Exclusive locks (X)

By default every time you connect to database, shared locks are applied. these locks are for more concurrency.
Once a row need to be updated , Update lock is applied & then exclusive lock is applied . so that only that transaction can exclusively have access to the row.

Update additionally have 3 other kinds of locks
1) table level exclusive lock
2) page level exclusive lock
3) row level exclusive lock

Update Locks :

1) update locks provide high concurrancy
2) they isolate rows before the update happens
3) they does not prevent the select statements , that means readable statements can still read the data
4) they upgrade to exclusive lock to update a row.

Page & Table Level locks will be discussed in a seperate thread..

Let us go in to the isolation levels now :

1) ReadUncommitted :
    In this isolation level, transaction T1 can have simultaneous access to data that is being modified by another   transaction T2 running in a different session. It means T1 will have dirty reads as T2 can either commit the data or roll back the data . Even if T2 rollbacks the changes still they are displayed in T1 . How ever this isolation level provides more concurrancy.
  Pros :  No read locks required, no read / write blocking
  Cons : High inconsistancy in data reads, as other transactions can roll back the changes
  Usage: These are used in transactions where data consistancy is not of  high priority & can have stale data.

NOTE :  Read Uncommitted & No Lock  - both does the same thing.

 ex:
Create table























Begin the transaction & update the records , do not commit (or) rollback




















Query the records in a seperate Window  with readuncommitted mode. It will return the result set as it doesn;'t hold read locks & gets the stale ( dirty ) data.




















T2 can roll back the updates resulting in stale data shown in Transaction T1.

2) Read Committed :
    In this mode, Transaction T1 can read only committed data. Each time a row is accessed it will be apply a shared lock & releases the shared lock once the row is read. If another transaction T2 modifies the data , then T1 can't read the data which is modified in T2 till it is committed or rolled back.

ex:  if T2 modifies row2  & T1 query need to return the row1 & row2 ,  then T1 Reads row1 how ever it will wait for row2 to be released by T2. This creates some blocking.

Pros : high consistancy of data
Cons : Can have blocking caused till data is committed or rolled back
Usage : This is sql server default isolation mode.

In the above example if updates are done & not committed or rolled back , then select query will not return the result set as they will be blocked.

ex :




















The Data is not returned till data is either committed or rolled back, causing some kind of blockage but data will be highly consistent.






















If you take the spid for above transaction & query the locks for the spid, then we will find that Shared lock is required & waiting for a shared lock .














3) Repeatable Read :
    In this isolation level , transaction T1 can only access committed data & additional guarantee of not modifying the already read data till  the transaction is completed. This is achieved by applying a shared lock on rows.How ever it doesn't protect from new inserts .

 Pros:  High data consistency
 Cons: Locking & Blocking, Rows are locked till the transaction is completed, How ever doesn't protect against Phantom rows.
Usage : Rare
  The below example illustrates the situation :
 Create table, insert some records in to it


























Begin Transaction & Update 3rd row. Don't commit



















As Transaction is not yet committed , select statement would be blocked on 3rd row.
it already reads 1st , 2nd rows & waiting on 3rd row.























The above query is looking for condition technology  = sql server , it already read row 1 as it satisfied the condition. Now update the rows making 2rd row as sql server instead of row 1 & commit the data.




















The commit will release the locks & select query will be able to read row3 , how ever data retrieved says row 1 as sql server consultant where as actual record that matches now is  row 2
This incosistency is caused because select query has read row 1, row2  & found row 1 is satisfying the query.
It is blocked at Row3  as different transaction has updated the row 3 & didn't commit it.
When the row2, row1 are updated & committed, lock is released & able to read row 3
How ever it will not look again for Row 1 & row 2 for consistency of data.

































To avoid this inconsistency of data during a transaction , repeatable read can be used.

Repeatable Read will lock each row that it transverses & doesn't release the lock till the complete transaction is done.   In the above example , if we use repeatable read isolation then  First update query will lock 3rd row. Select will lock 1st & 2nd row  , waiting for 3rd row.  Second Update will be waiting for 1st & 2rd rows which were locked by select statement resulting in dead lock.
 Dead lock is fine  in some cases, where data consistency is a must till the transaction is completed.













Let us try a different scenario :

Drop the table , recreate it again & insert some records in to it.
Set the isolation level to repeatable read & run the select query  with begin transaction :
The transaction will apply a  shared lock on the row & will not release a lock till the transaction is completed.
















Run the update query in another window. Update mode will be blocked as the row is already locked by above select statement.






















Now go back to the select window & commit the transaction  , it will release the lock & update statement can  acquire the lock .




















Update statement will be executed :





















How ever Repeatable Reads doesn't stop the phantom reads . phantom reads are read that read the newly inserted data,
Since repeatable reads just lock the rows that are scanned through , it  just blocks the existing rows . It will not lock the table from inserting any new rows , new rows can be inserted & can be read next time.

ex:   Run the select query , with out committing.

















Now the select query has locked the first row , how ever it doesn't block any new inserts from happening.
Now run the insert query in a seperate window & it inserts the record.










4)  Serializable : This isolation level provides highest data consistency including excluding phantom rows by applying range level locks (or) table level locks, resulting in low concurrency. The serializable isolation  takes the transactions one at a time in a serializable fashion.  Let us say 4 transactions runs  viz.  T1, T2, T3, T4 .
They run one after the another transaction is completed.
 
 Pros : High data consistency
Cons : Severe blockage, low concurrency.
Usage:: Used in High Data Consistent environments.

Ex : Use the same example from above repeatable read.
Create table , insert records,  Run select query in  serializable isolation.
Select query will apply a range lock blocking any other transaction to access the range.
Insert statements will be blocked.


























Now insert statement will be blocked , as select statement has applied range level lock.
Insert statement will wait till select statement commits or roll backs.

























5) Snapshot Isolation :
  This isolation level provides the data consistency as much as read committed does & provides high performance as read uncommitted .  This isolation level doesn't apply locks on the rows.
How ever it doesn't read the modified data till it is committed. Means no dirty reads. It reads data from a  snapshot.
Let us take the same sample example.
 Update a row in the table , it locks the row .


















Now set the isolation level to snapshot & run the select query :

You may see the error that snapshot isolation is not defined on database:




Alter database to set the snapshot isolation level for a database :



























NOw run the select statement in Snapshot isolation , it will not apply locks on the rows & returns the committed data only.

Sunday, December 18, 2011

Full Text Search Internals - part 3

Overview :

Full text search provides the ability to search the character based data
1) character data
2) varbinary data -->  when html documents are used for full text search  ms-locale meta tag is used to define the language for the full text search
3) xml data  --> xmllang is used to define the language for search
4) file stream data

Full text search functionality includes
1) simple search  using freetext
2) prefix search using contain
3)inflectional forms of same words( like run, ran etc ) using contains & form of inflectional
4) words near other words using contains & nearby
5) ranking values & weighted values using containstable & isabout

Components:

1) supports close to 50 languages .
   SELECT        lcid, name  FROM            sys.fulltext_languages
2) for each language there would be a word breaker & stemmer.
   word breaker breaks the document sentences in to word & decide on what constitues the words where as stemmers look for the inflectional forms of words. Third party word breakers can be purchased & supported,
stemmers are not invoked when full text search index is populated , they are invoked when full text search is done.
3) per instance thesauraus file.  they are the xml files on the hard disk located @
   C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\FTData












they are used for expansion sets & replacement sets ( if the search spells wrong then thesarus looks for the right words from xmls, replaces them & returns the search results on right words.
4) stop lists called as noice words, they are moved to internal tables in sql server 2008 . we can add the stop list per full text index.

Ifilters :

Filters are provided for each document type one for .doc, .docx, .pdf etc. Full text search looks for the registry and loads the ifilters for newly added ifilters. Filters implement Ifilter interface , third party filters can be supported.
Ifilters are used to parse the documents during the creation of fulltext index population not during the querying of data. We can provide the type of document for each row in the table & based on the type relevant ifilter is loaded , so same table can support multiple document types .













New Ifilters (or) third party Ifilters can be loaded using command :

exec sp_fulltext_service 'load_os_resources' ,1

Service :

Sql server 2005 used the external windows service for word breaker & stemmer. In  sql server 2008 those service are brought in to sql server. FDHost launch service should be up & running to support the full text search.  search runs with in the sql server engine.

Programming :

Full text search provides 4 programming functions,
1) Freetext 2) Contains  are predicate based
3) Freetexttable 4) containstable are table valued functions.

1) Free text is to search the scentence , it does the word breaking, stemmer (inflectional words ) , thesaurus ,stop words & finds documents.
2) contains - all of the above are not done automatically, need to explicitly define thesarus, forms of inflectional etc  based on what we are looking for.

---------

We can find the words that are indexed in the full text search per table with command :
select * from sys.dm_fts_index_keywords( DB_ID(), OBJECT_ID('HumanResources.Employee'))













fts parser gives the exact matches , stop word & numbers for a word given in the search.
Numbers will have nn in it.











sys.internal_tables  gives more information on full text .





















DOCID is the  unique id which refers to the rows in the table. when the table has unique int primary key ,then that key will act as the DOC ID.

Thesaurus files can be modified &  loaded  as shown in diagram,













SQL Server Denali Improvements :

1) Indexing is made multi threading, in 2008 it is single threaded.
2) predicative performance is improved. ex: search for many things like contains( tree) and contains (road ) , they modified to convert ( tree and road )
3) faster time response using streamed table valued functions, means we don't have to wait till all rows are returned. they stream the rows.
4)More granular locks & min time locking.
5) property based search for the documents, like  document author name, date created etc..
 properties are per index & when ever properties are updated , full text index must be repopulated.
Ifilters must support the extraction of properties .

-----------

Friday, December 16, 2011