Showing posts with label SQL. Show all posts
Showing posts with label SQL. Show all posts

Saturday, August 18, 2012

Oracle MERGE statement - Consistent READ

MERGE Statement & constraint violations
Oracle database ALWAYS provides Consistent-Read. This is a very powerful feature in Oracle and there are hundereds of articles on-line about this.
Simple description of Consistent Read is
When a (reading) query starts, it is important that the values of the rows selected are the same as when the query started, even if another session has changed those rows. This is refered to as read consistency. Read consistency is achieved through the SCN (system change number). (REF: http://www.adp-gmbh.ch/ora/concepts/consistent_read.html)
A somewhat bit more technical details can be found at http://dioncho.wordpress.com/2009/04/18/simple-and-stupid-test-on-consistent-read/
However, what's Consistent Read got to do with MERGE statement that this article is about?
MERGE statement allows a developer to take a source result set (result of a query) and MERGE that with a target result set. Syntax is documented at http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_9016.htm.
Key points from the doumantation (above) are
  1. Use the MERGE statement to select rows from one or more sources for update or insertion into a table or view. You can specify conditions to determine whether to update or insert into the target table or view.
  2. This statement is a convenient way to combine multiple operations. It lets you avoid multiple INSERT, UPDATE, and DELETE DML statements.
  3. MERGE is a deterministic statement. That is, you cannot update the same row of the target table multiple times in the same MERGE statement.
One of our developer developed a very complex MERGE statement. The program worked fine for most of the time, however at seeming random instances (for spcific input) the statement failed with Unique Key constraint Violation
It took a deeper understanding of consistent read to not only resolve the problem, but also leverage MERGE statement better in future. I hope our experience help others who are learning about Oracle.
I'll explain this experience with simplified example.
Version
 
1select * from v$version where rownum < 2;
2
3Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
Setup
We'll set up two small tables src (source) and tgt (target with PK).
 
01drop table tgt;
02
03drop table src;
04
05create table tgt as
06 select 1 c, 'A' str from dual ;
07
08alter table tgt add primary key (c);
09
10create table src as
11 select 1 c, 'B' str from dual union
12 select 2 c, 'B' str from dual union
13 select 2 c, 'C' str from dual
14 ;
Note: The tgt has a primary key and row with value 1 already exists.
If the following MERGE state is executed, what will be the result?
 
1merge into tgt
2 using src
3 on (tgt.c = src.c)
4 when matched then
5 update set tgt.str = src.str
6 when not matched then
7 insert (tgt.c,tgt.str)
8 values (src.c, src.str)
9;
This will result in ERROR: SQL Error: ORA-00001: unique constraint (HR.SYS_C0021834) violated.
Developer was very flustered with this error and called it a bug in Oracle. I am not sure if other databases behave this way.
The expectation was based "row-by-row" thought process.
It was expected that when a matching row is encountered the row will be updated, but when "first non-matching row" is encountered it will be treated as an INSERT and subsequent rows with same key will be treated as UPDATE.
This was misplaced expectation.
Oracle ALWAY performs in read-consistent manner.For Merge statement both the source and target result sets are determined at the time when execution of the statement begins.
Therefore, in the example above both rows from source (src) with key value of 2 are treated as INSERT. Not the first one as INSERT and second one as UPDATE.
With this understanding, resolution of the problem is simple, but with misplaced expectation the struggle continues.
Hope this will save others from some undue struggle.

Multi-Table Insert in Oracle

It is important to understand the fundamentals of the tool before using it, especially in production. We learnt this, costly lesson through struggle. In this BLOG, I'll document my experience, so it may help others.

Couple of years back, one of the developer learnt about multi-table insert ability in Oracle. We were running Oracle version 10g and in our application many primary/foriegn keys were generated using sequences.

The new code worked fine. Performance improvement (using multitable insert) in batch programs, instead of multiple insert statement, were GREAT. Significant (but not effective testing) was performed. Everything passed and code went to production.
However, after the code was in production for over 1 year, the batch programs (many of them) occassionally resulted in ORA-02291: integrity constraint (nnnnnnnn) violated - parent key not.

And then the frequencey of failure increased to almost multiple time a day. The processes run many times a day.

Since the error was not "reproducible" consistently in QA or development environment, the struggle continued. Developers blaming DBA and DBA blaming developers; Every one blaming Everyonelse. Furstration grew.

Now, a quick search for Note 265826.1 (in google, or MOS) might have resolved everything, however no one even thought of looking under the cover or search for this unknown MOS note.
Let us take a look at some sample code:
Setup:
For evaluating the behavior of Multi table insert using Oracle sequence as keys, with Parent-child (Master-Detail) relationships between tables, we'll try a simple example. Here is the setup code.
 
01drop table tb;
02drop table ta;
03
04create table ta (a number not null primary key ,
05 da varchar2(30) not null);
06
07create table tb (b number not null references ta(a) ,
08 db varchar2(30) not null);
09
10drop sequence ts;
11create sequence ts;
When following SQL is executed, everything works fine.
 
1insert all
2 into ta (a, da) values (ts.nextval, xx)
3 into tb (b, db) values (ts.currval, xx)
4select 'XXXXX' xx from dual connect by level < 500;
However, when you increase the the number of records to be inserted as shown below, we ran into ORA-02291.
For eg.
 
1insert all
2 into ta (a, da) values (ts.nextval, xx)
3 into tb (b, db) values (ts.currval, xx)
4SELECT 'XXXXX' xx from dual connect by level < 500000;
This is documented BUG of Multi-Table Insert: Note 265826.1. This was intitated in 2004. The workaround is described however the BUG is not corrected.
Developer tried adding ORDER BY clause and spent four days, struggling to do various alternatives.
Hopefully this will be addressed (if addressable) in some release. It still exists in 11.2.0.2 release.

MOS note excerpt:
This issue is an open bug:
MULTI TABLE INSERT (INSERT ALL) FAILS WITH ORA-2291
Based on the above:
"The order of the tables into which Oracle inserts data is not determinate. Therefore, before
issuing a multitable insert statement, you should defer any constraints and disable any triggers
that depend on a particular table order for the multitable insert operation."

FIX

WORKAROUND:
1. Disable the foreign key when run such MultiPath Inserts.
2. Use DEFERRED CONSTRAINTS so the checkout happens only at Commit time.
About Deferred Constraints check Metalink "Deferred Constraints Example"


References:
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:6915127515933
MOS document: Note 265826.1

Hope this helps others.

Working with LOB in Oracle (series 01)

One of my customers wanted to manage, store and retrieve BLOB in Oracle 11g database. They enquired me about the available options and evaluate effective feasibility.
Though Oracle DB supported BLOB/CLOB for a long time (since 8i), there is still apparent some mystery in achieving this.
Before we go to 11g new features, I wanted to demonstrate the “traditional” BLOB management in Oracle DB.
In this BLOG, I’ll demonstrate how to load a file as BLOB in Oracle 11g.
Environment setup
I am using Oracle 11g (11.1.0.7) instance hosted in AWS instance. We’ll use a table created under HR schema. In this posting I’ll use only SQL and PL/SQL constructs to store and retrieve BLOB objects in the table.
The first example is with BLOB column as follows…
SQL> select * from v$version where rownum < 3;

BANNER
-----------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
PL/SQL Release 11.1.0.7.0 - Production

SQL> select user, sysdate from dual;

USER                           SYSDATE
------------------------------ ---------
HR                             17-AUG-12

SQL>
A BLOB can be stored into column that is defined as BFILE, which will store the file outside the database. We’ll demonstrate this in next posting.
Create a table with BLOB object as one of the column.
SQL> create table tblob (cid int, bcol blob, ts timestamp);

Table created.

Now we are ready to load BLOB in the database table and retrieve using SQL, PL/SQL. DBMS_LOB is Oracle supplied package that can be used to handle LOB (BLOBs & CLOBs).
I have a text file on the server under /tmp directory that I’ll load into the table and retrieve. In my example the text file should ideally be loaded into a CLOB column.
Here is the file that will be loaded into BLOB column.
$ ll
total 4
-rw-r--r-- 1 oracle oinstall 232 Aug 17 18:58 testblob.txt
oracle@domU-12-31-39-00-7C-94:[/tmp/blob]
Since, SQL/PL-SQL run within the database on the server, the file (BLOB) and the location must be on the server and accessible to database server. This can be accomplished by creating a directory in database and granting required privilege to the user. This can be accomplished by using a directory object in database.
Create directory:
I’ll create a directory in database and grant read privilege to the HR user.
$ sqlplus  / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Aug 18 08:01:25 2012

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> create or replace directory myblobdir as'/tmp/blob';

Directory created.

SQL> grant read on directory myblobdir to hr;

Grant succeeded.

SQL>
SQL> connect hr
Enter password:
Connected.
SQL> select * from all_directories;

OWNER    DIRECTORY_NAME    DIRECTORY_PATH
-------- ----------------- --------------------------------------------
SYS      MYBLOBDIR         /tmp/blob

SQL>
I’ll create a PL/SQL procedure to load the file into BLOB.
CREATE OR REPLACE PROCEDURE load_file is
  src_lob  BFILE := BFILENAME('MYBLOBDIR', 'testblob.txt');
  dest_lob BLOB;
BEGIN
  INSERT INTO tblob VALUES(2, EMPTY_BLOB(),systimestamp)
     RETURNING bcol INTO dest_lob;

  DBMS_LOB.OPEN(src_lob, DBMS_LOB.LOB_READONLY);
  DBMS_LOB.LoadFromFile( DEST_LOB => dest_lob,
                         SRC_LOB  => src_lob,
                         AMOUNT   => DBMS_LOB.GETLENGTH(src_lob) );
  DBMS_LOB.CLOSE(src_lob);

  COMMIT;
END;
We can use SQLPlus to access the LOB from tables.
SQL> desc tblob;
 Name                                Null?    Type
 ----------------------------------- -------- ----------------------
 CID                                          NUMBER(38)
 BCOL                                         BLOB
 TS                                           TIMESTAMP(6)

SQL> select count(*) from tblob;
  COUNT(*)
----------
         0

SQL> select * from tblob;
no rows selected

SQL> select dbms_lob.getlength(bcol) from tblob;
no rows selected

SQL>
Execute the procedure to load the file.
SQL> exec load_file;

PL/SQL procedure successfully completed.

SQL> select count(*) from tblob;

  COUNT(*)
----------
         1

SQL> select dbms_lob.getlength(bcol) from tblob;

DBMS_LOB.GETLENGTH(BCOL)
------------------------
                     232

SQL>
Note The size (length) is matching indicating the file has been loaded.
SQL> !ls -ltr;
total 8
-rw-r--r-- 1 oracle oinstall 232 Aug 17 18:58 testblob.txt
-rw-r--r-- 1 oracle oinstall 471 Aug 18 08:35 load_file.sql

SQL>
Of course, since we loaded text file as BLOB, SQLplus retrieval displays the HEX representation of the data. The data text can be retrieved using DBMS_LOB conversion function.
SQL> select * from tblob;

       CID BCOL                                                                                                                                                             TS
---------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------- ----------------------------
         2 54686973206973206120544558742066696C652063726561746564207573696E6720564920656469746F7220756E646572206C696E7578204F532E0A4F66636F75727365207468697320636F756C6420 18-AUG-12 08.43.48.247162 AM

SQL>
SQL> select UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(bcol,4000,1)) t from tblob;

T
--------------------------------------------------------------------------------
This is a TEXt file created using VI editor under linux OS.

SQL>
In the next posting I’ll demonstrate the use of CLOB, conversion between CLOB and BLOB, and explote additional functionality of DBMS_LOB package.
Till then Bye…
References
Oracle 11g (11.2) documentation managing BLOB using various programming languages and environment can be found here…
·         Working with LOB and BFILEs in Oracle (using JDBC)
·         PHP example for loading image and displaying the image:
·         There is a supplied package DBMS_LOB that provides subprograms to operate on BLOBs, CLOBs, NCLOBs, BFILEs, and temporary LOBs.
·         ORAFAQ has a very simple introduction for loading BLOB in Oracle using DBMS_LOB package.
·         Check out Arup Nanda’s post about new features on SecureFiles LOB management in 11g here…