table of Contents

Introduction

I'm developing a large data warehouse. We have data from multiple transactional systems with different database environments and architectures of. COBOL flat files, the Oracle , SQL Server is the main data source. Nightly process to import from all data sources daily transaction data (incremental change), and combined to form a single view of the customer database.

By comparing the Surname , Firstname , DOB , Sex , the first character of the first two address lines, the ETL process all transactions in the system can be found in Common client - based on the condition field created HASHBYTE checksum value, and Checksum on the field use Gruop BYE . Therefore, ETL process of creating a data delete duplicate Customer table.

Then use the latest update new contact details (telephone, mobile phone, email, address, etc.) Staging the Customer table. In addition, the Customer table after some data validation and standardization. For example, all the phone numbers to the international format, verify your email address, delete bad data.

Each unique client ( the Customer ) will receive a customer ( the Customer ) ID ( CID ). A lower ETL process, the CID may be combined or incorporated as part of a change of the lift on the transaction system. I only described herein Customer table, but the ETL process is based on CID created a number of other tables (for example, PRODUCT ).

Finally, I have to Staging Customer table and production of real-time customer ( Live the Customer ) table synchronization, because the Web and Mobile API all in real time, so they will have synchronized during the day, even at midnight should be synchronized. As the Customer table with millions of records and CID possibility of merging, so from staging Customer table is updated Live Customer table is not easy, and the cost is not low. This process takes a long time, and cause other applications to run out. To update the form in real time, follow these steps:

Targeting customers ( the Customer ) table
In ETL load any new customer is created in the process of real-time table
Truncated real-time (live) Customer table
From Staging Customer table load data
Rebuild all indexes
Unlock Customer ( the Customer ) table

The question now is above steps take almost 5-10 minutes, while other Web and mobile applications will be affected because of a timeout. I have been using temporary tables ( DROP and RENAME ) to solve the problem. The system worked, only a second to lock customers table!

step:

In Production create a temporary database Customer table.
From the staging area to load data into the temporary table.
Create the required index on a temporary table (in my case is a full-text index)
In the ETL any new customer is created on the active list process will be loaded into a temporary table.
DROP real-time (live) Customer table
RENAME temporary table survive the customer table

Using the code

Code is created on the production database tmp_Customer and from the temporary ( staging ) to load the data table. Next to create all necessary indexes. Collect any missing customer details from the customer table in real time - in ETL , in the course of Production database created in the Customers . Finally DROP real-time (live) customer table and RENAME temporary table live. Code shows only the Customer how table in Production refresh. You can repeat all the code for the other tables. Backup necessary tables before continuing.

-- Microsoft SQL Server 2014 (SP2-CU16) (KB4482967) - 12.0.5626.1 (X64)

-- Author: Prasannakumaran Sarasijanayanan

IF OBJECT_ID('dbo.tmp_Customer', 'U') IS NOT NULL DROP TABLE dbo.tmp_Customer;

　

CREATE TABLE [dbo].[tmp_Customer](

[CID] [bigint] NOT NULL,

[FirstName] [varchar](50) NULL,

[SurName] [varchar](50) NULL,

[Address1] [varchar](50) NULL,

[Address2] [varchar](50) NULL,

[Address3] [varchar](50) NULL,

[Address4] [varchar](50) NULL,

[Address5] [varchar](50) NULL,

[Title] [varchar](10) NULL,

[Sex] [varchar](1) NULL,

[DOB] [varchar](12) NULL,

[Fulladdress] [varchar](200) NULL,

[email] [varchar](100) NULL,

[Mobile] [varchar](20) NULL,

[Fullname] [varchar](100) NULL

-- Other columns are purposely removed

PRIMARY KEY CLUSTERED

(

[CID] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF,

ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 90) ON [PRIMARY]

) ON [PRIMARY]

　

--Load data from ETL staging area

insert into tmp_customer

( CID, FirstName, SurName, Address1, Address2, Address3, Address4, Title, Sex, DOB, Fulladdress, email, Mobile, Fullname)

select CID, FirstName, SurName, Address1, Address2, Address3, Address4, Title, Sex, DOB, Fulladdress, email, Mobile, Fullname from CCF

--========================= Creating Indices ==================================

--Mobile

CREATE NONCLUSTERED INDEX [NonClusteredIndex-Mobile] ON [dbo].[tmp_Customer]

(

[Mobile] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

　

--CREATE FULLTEXT CATALOG Customer_FullText WITH ACCENT_SENSITIVITY = OFF


declare @sql varchar(max)

Declare @CONSTRAINT_NAME varchar (256)

SELECT @CONSTRAINT_NAME=KU.CONSTRAINT_NAME

FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS TC

INNER JOIN

INFORMATION_SCHEMA.KEY_COLUMN_USAGE AS KU

ON TC.CONSTRAINT_TYPE = 'PRIMARY KEY' AND

TC.CONSTRAINT_NAME = KU.CONSTRAINT_NAME AND

KU.table_name='tmp_customer'

ORDER BY KU.TABLE_NAME, KU.ORDINAL_POSITION

--BUILD Dynamic SQL

set @sql=

'CREATE FULLTEXT INDEX ON tmp_customer

(

Email

Language 1033,

Fullname

Language 1033,

Fulladdress

Language 1033

)

KEY INDEX ' + @CONSTRAINT_NAME + '

ON Customer_FullText

WITH STOPLIST = OFF

-- SEARCH PROPERTY LIST = OFF

'

EXEC (@sql)

--insert Customer data created during ETL

INSERT INTO tmp_Customer

( CID, FirstName, SurName, Address1, Address2, Address3, Address4, Address5, Title, Sex, DOB, Fulladdress, email, Mobile, Fullname)

SELECT CID, FirstName, SurName, Address1, Address2, Address3, Address4, Address5, Title, Sex, DOB, Fulladdress, email, Mobile, Fullname

FROM Customer P WHERE NOT EXISTS (SELECT NULL FROM tmp_Customer F WHERE P.CID = F.CID)

--Finally Drop and Rename table

IF OBJECT_ID('dbo.Customer', 'U') IS NOT NULL DROP TABLE dbo.Customer;

EXEC sp_rename 'tmp_customer', 'Customer'

Original Address: https://www.codeproject.com/Tips/5062305/Busy-Production-Database-update-without-timeout

Busy production database update, no Timeout

Introduction

Using the code

Guess you like