I wrote last year about Graphing I/O data using gnuplot and Oracle V$SYSSTAT, using a script from Kevin Closson in his article How To Produce Raw, Spreadsheet-Ready Physical I/O Data With PL/SQL. Good For Exadata, Good For Traditional Storage. Here I’ve got a simple comparison of the data recorded through this script (in essence, Oracle’s V$SYSSTAT), and directly on the OS through HP’s MeasureWare. It’s graphed out with my new favourite tool, rrdtool:

I’ve added two new toys to my geek arsenal today. First is one with which I’ve dabbled before, but struggled to master. The second is a revelation to me and which I discovered courtesy of twitter.

rrdtool

rrdtool is a data collection and graphing tool which I’ve been aware of for a while. I wanted to use it when I wrote about Collecting OBIEE systems management data with JMX, but couldn’t get it to work. I’ll not lie to you - it is a bitch to work with at first. Or put a more polite way, it has a steep learning curve. But when you reach the top of the curve and realise its potential…wow. You’ll soon understand why it is so widely used. I plan to write this up soon, but it let me draw nice graphs like this:

Introduction

This is the third part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

This final article describes how to bolt on to OAS a simple web page hosting the graphs that you created in part 2, plotting data from OBIEE collected in part 1.

The webpage

This is just an old-school basic HTML page, with a meta-refresh tag (which note that Chrome doesn’t work with) and img tags:

Introduction

This is the second part of three detailed articles making up a mini-series about OBIEE monitoring. It demonstrates how to capture OBIEE performance information, and optionally graph it out and serve it through an auto-updating webpage.

This article takes data that part one showed you how to collect into a tab-separated file that looks something like this:

[sourcecode] 2010-11-29-14:48:18 1 0 11 0 3 2 1 676 340 0 53 1 0 41 0 3 0 2010-11-29-14:49:18 1 0 11 0 3 2 1 676 0 0 0 1 0 0 0 3 0 2010-11-29-14:50:18 2 0 16 1 4 3 1 679 0 0 0 1 0 0 0 4 0 2010-11-29-14:51:18 2 2 19 1 4 3 1 679 32 0 53 1 0 58 0 4 0 2010-11-29-14:52:18 2 1 19 1 4 3 4 682 0 0 0 1 0 0 0 4 0 2010-11-29-14:53:18 2 1 19 1 4 3 4 682 0 0 0 1 0 0 0 4 0 2010-11-29-14:54:18 2 0 19 1 4 3 1 682 0 0 0 1 0 0 0 4 0 [/sourcecode]

They say about travelling that it’s the journey and not the destination, and the same is true with this problem we hit during a deployment to Production.

We were deploying a new OBIEE 10g implementation, with authentication provided by Microsoft Active Directory (AD) through the LDAP functionality in OBIEE. As a side note, it’s a rather nice way to do authentication, although maybe I’m biased coming from our previous implementation which used EBS integrated authentication and was a bugger to set up and work with.

Continuing in the beard-scratching theme of Unix related posts (previously - awk), here’s a way to graph out the I/O profile of your Oracle database via the Oracle metrics in gv$sysstat, and gnuplot. This is only the system I/O as observed by Oracle, so for belts & braces (or to placate a cynical sysadmin ;-)) you may want to cross-reference it with something like sar.

First, a pretty picture of what you can get:

Introduction

We’re in the process of implemention Resource Manager (RM) on our Oracle 11gR1 Data Warehouse. We’ve currently got one DW application live, but have several more imminent. We identified RM as a suitable way of - as the name would suggest - managing the resources on the server.

In the first instance we’re looking at simply protecting CPU for, and from, future applications. At some point it would be interesting to use some of the more granular and precise functions to demote long-running queries, have nighttime/daytime plans, etc. I’d also like to explore the management of IO but for us the pain is in bandwidth that a query consumes, and it looks like RM can only work with total session MB, or IOPS. Reading about Exadata it sounds like the Exadata I/O Resource Management might do this ([…]It allows intra and inter-database I/O bandwidth to be defined and managed[..]). But for that I’ll have to write to Santa and promise to be a good boy this year.

Here are a couple of little unix scripts that I wrote whilst developing my performance testing OBIEE method.

They’re nothing particularly special, but may save you the couple of minutes it’d take to write them :)

Note that some of this data is available from Usage Tracking and where it is I’d recommend getting it from there, databases generally being easier to reliably and repeatably query than a transient log file.

This error caught me out today. I was building a Linux VM to do some work on, and for the life of me couldn’t get the OBIEE Admin Tool to connect to the BI Server on the VM.

The error I got when trying to define a DSN on the Windows box was:

[nQSError: 12008] Unable to connect to port 9703 on machine 10.3.105.132 [nQSError: 12010] Communication error connecting to remote end point: address = 10.3.105.132; port = 9703. [nQSError: 12002] Socket communication error at call=: (Number=-1) Unknown

A very interesting new PDF from Sun on deploying OBIEE has been published, with discussions on architecture, performance and best practice.

This Sun BluePrints article describes an enterprise deployment architecture for Oracle Business Intelligence Enterprise Edition using Sun servers running the Solaris Operating System and Sun Storage 7000 Unified Storage systems. Designed to empower employees in organizations in any industry—from customer service, shipping, and finance to manufacturing, human resources, and more—to become potential decision makers, the architecture brings fault tolerance, security, resiliency, and performance to enterprise deployments. Taking advantage of key virtualization technologies, the architecture can be used to consolidate multiple tiers onto a single system to help reduce cost and complexity. A short discussion of the performance characteristics of the architecture using a realistic workload also is included.

Introduction

Whilst the BI Cluster Controller takes care nicely of clustering and failover for BI Server (nqsserver), we have to do more to ensure further resilience of the stack.

A diagram I come back to again and again when working out configuration or connectivity problems is the one on P16 of the Deployment Guide. With this you can work out most issues for yourself through simple reasoning. Print it out, pin it to your wall, and read it!

Production cluster is 2x BI Server and 2x Presentation Services, with a BIG-IP F5 load balancer on the front.

Symptoms

Users started reporting slow login times to BI. Our monitoring tool (Openview) reported that “BIServer01 may be down. Failed to contact it using ping.”. BIServer01 cannot be reached by ping or ssh from Windows network.

Diagnostics

nqsserver and nqsclustercontroller on BIServer01 was logging these repeated errors:

[nQSError: 12002] Socket communication error at call=send: (Number=9) Bad file number

Introduction

In this article I plan to get samplesales and paint repositories hosted on a single server, using one BI Server instance and two Presentation Services instances. This is on both Unix (OEL 4) and Windows, and both OC4J (OBIEE’s “basic installation” option) and OAS (“Advanced Installation”).

Both samplesales and paint are shipped with 10.1.3.4 of OBIEE, you’ll find them in $OracleBI/OracleBI/server/Sample. This article assumes you’ve got the RPD of each into $OracleBI/OracleBI/server/Repository and unpacked the web cats for each into $OracleBIdata/web/catalog. It also assumes that you know your way around the architecture of BI and are familiar with NQSConfig.ini and instanceconfig.xml - if neither of those files mean anything to you then you will find some background reading useful.

A question that pops up on the OBIEE OTN forum quite often is how to use non-Oracle databases like MS SQL Server when the OBIEE server is running on a non-Windows OS such as Linux.

The answer in a nutshell is that since version 10.1.3.3.1 OBIEE has been bundled with ODBC drivers for unix/linux from a company called DataDirect. See the release notes here for more information and installation instructions (as well as a list of support databases).

Here’s a set of scripts that I use on our servers as a quick way to check if the various BI components are up and running.

Because we split the stack across servers, there are different scripts called in combination. On our dev boxes we have everything and so the script calls all three sub-scripts, whereas on Production each server will run one of:

1. BI Server
2. Presentation Server & OAS
3. Informatica & DAC

The scripts source another script called process_check.sh which I based on the common.sh script that comes with OBIEE.

As a kid I loved the idea of lego where you can disassemble and reassemble something from the ground up. As soon as I got my hands on a computer it was the same. You can have your Acorn Archimedes with its games, where do I find the sprites and sound files behind it? Likewise Microsoft Word, let me at the VBA underneath to hack it around and see what else it can do.

[edit] See this post too [/edit] On an OBIEE server run [sourcecode language=“bash”] nohup obiee/systemsmanagement/runagent.sh & [/sourcecode] and then run jconsole (make sure you’ve set the DISPLAY first if you’re running it from UNIX). NB: if you don’t have jconsole in your path you can search for it: [sourcecode language=“bash”] $whereis jconsole jconsole: /opt/java1.5/bin/jconsole /opt/java6/bin/jconsole</span> [/sourcecode] You should find it under your java/bin directory

You should get this kind of connection dialog: Click connect, and the console will launch. From here click on the MBeans tab, where you’ve got access to performance and configuration data

(See here and here for history)

I edited the shell script which is eventually called by run-saw.sh to start the sawserver, (OracleBI)/setup/sawserver.sh, to use trus:
Comment out the final line:

$SASAWSERVER

and insert as a new line:

tusc -fepan -o /tmp/sawserver_tusc.out $SASAWSERVER

The output of trus ended with this:

open("/app/oracle/product/10.2.0/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) ……… ERR#2 ENOENT
open("/app/oracle/product/obiee/server/Bin64/libstd_v2.so.1", O_RDONLY|0x800, 0) . ERR#2 ENOENT
open("/app/oracle/product/obiee/web/bin64/libstd_v2.so.1", O_RDONLY|0x800, 0) …. ERR#2 ENOENT
open("/app/oracle/product/obiee/odbc/lib64/libstd_v2.so.1", O_RDONLY|0x800, 0) … ERR#2 ENOENT
open("/usr/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) …………………………. ERR#2 ENOENT
open("/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) …………………………….. ERR#2 ENOENT
open("/app/oracle/product/10.2.0/lib/libstd_v2.so.1", O_RDONLY|0x800, 0) ……… ERR#2 ENOENT
open("/opt/aCC/lib/hpux64/libstd_v2.so.1", O_RDONLY|0x800, 0) ……………….. ERR#2 ENOENT

In investigating the problems with sawserver I was pointed towards a tool called tusc (which appears to be an HP version of truss).

You can use it to invoke a program, and get out a bunch of debug information including system calls.

You run it like this:

$tusc -fep /app/oracle/product/obiee/web/bin64/sawserver64

As a beginner when it comes to hardcore *nix I can only look at this and take pot shots at what’s going on, but with Google by my side I’m interested in the last lines of the output:

We’re getting this error in the Presentation Services plug-in [analytics].
Log file: /j2ee/home/application-deployments/analytics/home_default_group_1/application.log

09/03/30 13:16:38.75 analytics: Servlet error
java.net.ConnectException: Connection refused (errno:239)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:517)
at java.net.Socket.connect(Socket.java:467)
at java.net.Socket.(Socket.java:364)
at java.net.Socket.(Socket.java:178)
at com.siebel.analytics.web.sawconnect.ConnectionPoolSocketFactoryImpl.createSocket(ConnectionPoolSocketFactoryImpl.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.siebel.analytics.web.sawconnect.ConnectionPoolSocketFactoryImpl.createSocket(ConnectionPoolSocketFactoryImpl.java:70)
at com.siebel.analytics.web.sawconnect.ConnectionPool.createNewConnection(ConnectionPool.java:314)
at com.siebel.analytics.web.sawconnect.ConnectionPool.getConnection(ConnectionPool.java:133)
at com.siebel.analytics.web.SAWBridge.processRequest(SAWBridge.java:299)
at com.siebel.analytics.web.SAWBridge.doGet(SAWBridge.java:325)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:713)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:3
70)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:302)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:190)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:239)
at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:34)
at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:880)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.3.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.
java:303)
at java.lang.Thread.run(Thread.java:595)