Being the DBA here, I felt it was my responsibility to create a 10046 trace file and summarize where all the time was going.  Thankfully this was 100% reproducible in our staging environment.  What I found was very interesting.

Generally when there’s something running in the database that I running slower that expected, the first thing I do is figure out how to best get an accurate trace of the problem.  That involves initiating the trace immediately before the problem actions and terminating the trace immediately after those actions are complete.  That keeps dooda from creeping into your trace file.  There are a whole bunch of ways to that these days and the internet is plastered with info on how to do that.   I then just run tkprof on the resulting file with sys=no and everything else default.  That gives me the sql in order of execution  and I just follow the execution order and make note of SQL with the highest elapsed time.  I know Millsap would be disappointed but that’s usually enough technical info to figure out what to start trying to make happen faster.  Not rocket science but it usually does the trick.

In this case, what was extremely interesting was the non-recursive and recursive summary output found at the bottom of the tkprof file.  When I think recursive SQL I think of all the stuff Oracle has to do in the background that I can’t do much about. and user defined functions embedded in SQL somwhere.  If you embed user defined functions in SQL to hide complexity you are usually handicapping the optimizer and vastly increasing the chance that you will get some overly resource intensive activity.  Here’s what that summary looked like:

OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTScall

count       cpu    elapsed       disk      query    current        rows

——- ——  ——– ———- ———- ———- ———-  ———-

Parse        2      0.01       0.01          0          7          0           0

Execute      2      0.88       5.98          0      12145      74414           2

Fetch        0      0.00       0.00          0          0          0           0

——- ——  ——– ———- ———- ———- ———-  ———-

total        4      0.89       5.99          0      12152      74414           2

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows

——- ——  ——– ———- ———- ———- ———-  ———-

Parse     2141     17.89      17.89          0          0          0           0

Execute 13589173    492.64     489.04          5        180          0           0

Fetch   13589083   4007.37    4035.89     778773  293428221        462    13589118

——- ——  ——– ———- ———- ———- ———-  ———-

total   27180397   4517.91    4542.83     778778  293428401        462    13589118

13.5 million executions of recursive SQL statements.  Returning 13.5 million rows.  After I picked my jaw up off the floor, I gave myself a pep talk and started hunting through the trace file.  Now, if I keep this writing thing up, I’m will eventually start sounding like a broken record but our homegrown applications seem to be heavy on the views on top of views on top of views methodology for saving the developers some time.  What we end up with are tons of rabbit holes to crawl down when performance problems arise.  Anyways…

So, I went backup up the trace file and found the following SQL was executed nearly 13.5 million times.

SELECT PERIODID

FROM

PERIOD WHERE TRUNC(STARTDATE) <= :B1 AND TRUNC(ENDDATE) >= :B1

Odd.  I looked through the PL/SQL package that implemented the job, searching for this SQL.  Nope.  Eventually I found the culprit.  It was in a view and sure enough, it was a user defined function used to filter results for the current period.  This will be short on details but I spent a day and a half working this view into submission. I got the LIO’s down from ~ 17 million down to 18 thousand.  There is no doubt more that could be done (isn’t there always) but I feel like I plucked the lowest hanging fruit and after re-tracing the original job, the recursive calls fell from 13.5 million down to 4996.  More importantly the job went from lasting > 2.5 hours to completing in < 5 minutes.

My biggest problem now is stepping away from the computer before I start obsessing about all that can be done with the code base here.

I few things that really helped me here:

1. Get rid of the user defined functions unless they really make sense to keep around.
2. If you use /*+ gather_plan_statistics */ + dbms_xplan + allstats to dig into what is going on and notice that the a-rows is way different from the e-rows from a row source that you see could explicitly have some more filters applied to it because of transitivity, don’t be scared to do that.
3. If you are having trouble reading the SQL don’t be scared to start breaking it up with the ‘WITH’ clause so you can focus on the real work.
4. If you are still having problems, start breaking out individual pieces of the SQL  and do the /*+ gather_plan_statistics */ + dbms_xplan + allstats thing on that and work your way back.

Why didn’t I catch this with my AWR monitoring?  Well, it only ran once a month and it wasn’t a single SQL taking a ridiculous amount of time to run.  I do have a report that shows the top 5 mostly frequently executed SQL each day but you’d be surprised at how much internal stuff shows up on that.  (hmmm…I should probably filter that out).

Sorry for the lack of details but it’s beer:30 and I have an abundance of Hopslam waiting for me at home.

Later that afternoon I get an email stating that my suggestion resulted in an ORA-02436.  Oops.  It turns out that check constraints cannot contain calls to functions that are not deterministic.  Ugh.  I should have known that. So much for this blog advertising what a knowledgeable Oracle guy I am!

Anyways, I responded with a list of three options.

1. This was a very modestly sized APEX app.  Validating the data through APEX validations and calling it a day seemed perfectly reasonable.  The other two options provide no additional information to the optimizer that check constraints potentially do.  Sure you want data rules as close to the data as possible but this is so trivial that I don’t see it mattering much.
2. Define a trigger that raises an exception if the :new value violates the rule.  I generally despise triggers.  I’ve wasted far too much of my life trying to track down the source of some unexplained event only to eventually discover it has something to do with triggers.  They complicate things far more often than they simplify them.  However, I think it is commonly acknowledged that a valid use for triggers is data validation that cannot be done declaratively.  In fact I think I’ll be hearing a presentation given by Toons Koppelaars making that very same claim at Hotsos this year.
3. We just upgraded to 11G so you can play games with virtual columns but its really a kludge and I could imagine a day when the database kernel wises up to the trick in order to prevent the problems the restriction is meant to address.  I discouraged this.

In the end, I think the developer is going to go with both 1 and 2.  That way you prevent unecessary submits back to the database and protect yourself against bad data getting in through some route other than the application.

In case anyone wants to implement an non-deterministic function into a constraint here’s an example that you can tailor to your needs (I lifted this idea from here :

create or replace function year_in_past( in_year in table.year_of_birth%type) return varchar2 deterministic is
begin
return case when in_year >= extract(year from sysdate) then ‘N’ else ‘Y’ end;
end year_in_past;
/
alter table table1   add (year_in_past_ind as (cast(year_in_past(year_in_past) as varchar2(1))));

alter table table1 add constraint yip_ck1 check (year_in_past_ind = ‘Y’);

This wouldn’t work if you wanted to ensure a year was set to some future value since, as we all know, time marches on and what once was the future always becomes the past.

Ok, I will endeavor to post something a little farther from DBA 101 next time.

Chris

Surprisingly, no one knew the answer.  My guess was that a check constraint such as “column_name in (‘val1′,’val2′)” would imply the column could not contain null values and so the ‘not null’ piece of the table ddl was redundant. I was a little embarrassed that I didn’t know the answer to such a seemingly basic question. Perhaps more surprisingly, the question was dropped and no one showed any further interest. You would think that a group of developers that make their living working with Oracle databases would want to know the answer. Some folks just don’t give a rip!

SQL> CREATE TABLE TST_CHK (COL VARCHAR2(1), COL2 VARCHAR2(1));

SQL> ALTER TABLE TST_CHK ADD CONSTRAINT TST_CHK1 CHECK (COL2 IN (‘A’, ‘B’)) ENABLE;

SQL> Insert into tst_chk(col, col2) values (‘T’,NULL);

1 row created.

SQL> Insert into tst_chk(col, col2) values (‘T’,'C’);

Insert into tst_chk(col, col2) values (‘T’,'C’);

ERROR at line 1:

ORA-02290: check constraint (DB_ADMIN.TST_CHK1) violated

SQL>

So, check constraints are only enforced when the column is actually populated with a value.

So much for impressing the world with an earth-shattering first post about my job!  More positively, I feel I did my part for the BAAG this morning!

Anyways, I’ve always felt like writing would greatly enhance my ability to recall some fraction of what I ‘learn’.  I’ve made several attempts to start writing about my experiences with Oracle and the technologies I use to manage the environment I’m responsible for.  ….here I am trying to jump start that effort again.