This is just to mark the autumn conference of the Bulgarian Oracle User Group (BGOUG). The conference will be held in Spa hotel Hissar in Hisarya, between 18th and 20th of November, 2011. As always it would be great experience to have a chat with friends, listen to interesting presentations and enjoy the party at the official dinner.

For more information visit the BGOUG homepage:

http://www.bgoug.org/en/events/details/86.html

See you there!

Regards,
Sve

Well, this is quick notice in case someone else got into this error. I’m having Audit Vault server, patched up to 10.2.3.2.5 and its repository database to 10.2.0.7. The problem is that I’m able to connect as av_admin into the console, but not as av_auditor. When I try to login as av_auditor I’ve got redirected to wrong URL, like this one:

http://192.168.1.100:0/av/console/f?p=7700:100:::::

It’s obvious that’s wrong, port 0 does not exist and I’m getting error Unable to connect in the browser.

Just to make sure whether this is the problem, check to see if the lsnrctl status is having line like this one:

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hostname)(PORT=5707))(Presentation=HTTP)(Session=RAW))

Also use dbms_xdb.gethttpport to get the port on which the console is listening at:

SELECT DBMS_XDB.gethttpport() from dual;

DBMS_XDB.GETHTTPPORT()
----------------------
0

These tips are described at the documentation Oracle® Audit Vault Administrator’s Guide, in particular A.3.6.1 Oracle Audit Vault Reports Not Displaying

The correct port of Oracle Audit Vault Reports HTTP is 5707 and running the above query should return exactly this port.  If this is the case and you get port 0, then login as sysdba and set the correct port:

SQL> EXEC DBMS_XDB.SETHTTPPORT(5707);

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

Make sure the changes are applied:

SQL> SELECT DBMS_XDB.gethttpport() from dual;

DBMS_XDB.GETHTTPPORT()
----------------------
                  5707

And finally register the database:

SQL> ALTER SYSTEM REGISTER;

You’re now happy Audit Vault auditor who can login successfully to the console.

Regards,
Sve

Recently I had an interesting setup for one of our customers. Because they got Oracle Standard Edition and Windows 2008 Server R2 Standard Edition licenses I was asked to create HA database installation. After looking around I found few docs about installing Standard Edition with Clusterware and I had some ideas. Finally I installed Grid Infrastructure on both servers and Oracle Database binaries. Then created single instance database on the second server and replicated the configuration to the first one. Currently the relocation of the database is done manually, but one could create a start/stop/monitor scripts and integrate these with GI. Once the database starts it’s registering at the scan listener so in theory it’s running in HA (just the relocation is manual)

So during the weekend I received mail from my colleagues above error messages they received from the database: connect error, Socket read timed out. It wasn’t a rush as the database is not yet in production, but it’s ahead and this was the first task for the Monday. Next day I looked around and everything was up and running, except that I wasn’t able to login through the listener and I also wasn’t able to stop or relocate it. Looking at the logs I found at some point the following message: TNS-12531: TNS:cannot allocate memory which explains the previous message.

That was weird, the server on which error appeared was the first one and had only GI running and SCAN LISTENER. This really looked like a memory leak, it’s a Windows so maybe that was obvious. I decided to look around the processes using the Resource Monitor when I found a lot of many cmd.exe processes. To confirm the problem I used Process Explorer which is a very nice tool for Windows. As could be seen below I’ve got plenty of cmd processes which were spawned, but not (obviously) closed after completion:

It turned out that this is a bug for 11.2.0.2 and Windows (64 bit). The Oracle CVU resource (ora.cvu), which by default is started on the first node in the cluster (this makes sense now) it’s doing checks on every six hours (CHECK_INTERVAL=21600) and leaves process open. Because of this the heap memory is exhausted and that’s the reason why the SCAN LISTENER is failing and giving the error message TNS-12531: TNS:cannot allocate memory

The following errors could be seen in Windows Eventlog, once the patch is applied the errors disappeared:
Faulting application lsnrctl.exe, version 11.2.0.2, time stamp 0x4cea8f55, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb73957, exception code 0xc0000142, fault offset 0x00000000000b1b48, process id 0x1eac, application start time 0x01cc6ab588f992c0.

Faulting application cmd.exe, version 6.0.6001.18000, time stamp 0x47918bde, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb733e1, exception code 0xc0000142, fault offset 0x0006f1e7, process id 0×1004, application start time 0x01cc6af0fa982500.

Faulting application sclsspawn.exe, version 0.0.0.0, time stamp 0x4ce622a7, faulting module kernel32.dll, version 6.0.6001.18538, time stamp 0x4cb73957, exception code 0xc0000142, fault offset 0x00000000000b1b48, process id 0x1ca0, application start time 0x01cc6c0e5efd5380.

This is the bug at MOS:
Bug 12529945: CVU HEALTH CHECKS EXHAUST WINDOWS HEAP MEMORY

The bug should have been fixed in BP8, but I applied the latest one BP10:
Patch 12849789: ORACLE 11G 11.2.0.2 PATCH 10 BUG FOR WINDOWS (64-BIT AMD64 AND INTEL EM64)

Regards,
Sve

More than a week after the SIOUG (Slovenian Oracle User Group) I wanted to say thanks to Joze Senegacnik for inviting me and for being very hospitable.

I’ve been many times in Slovenia, but It’s my first time being at the SIOUG and I really liked it. There were a lot of people, the hotel was at the sea side and Croatia was just on the other side of the sea, just a few kilometres away. Speaking about Croatia I met Vladimir and Davor from HROUG.

I also met Doug Burns and Debra Lilley and they are really cool people.

Oracle Open World starts in two days. Although I won’t be able to attend it this year I’m pretty excited about the new products and announcements.

Regards,
Sve

p.s. I met Stanley in a cave where The Human Fish lives, wonder what was he doing there.

Yesterday at Open World, Oracle announced the release of Enterprise Manager Cloud Control 12c release 1 (version 12.1.0.1), where c stand for cloud. EM12c is a management solution providing centralised monitoring, administration, and lifecycle management functionality for the complete IT infrastructure.

A lot of organisations deploy private cloud environments to offer great flexibility to business users and meet SLAs and security requirements. The Oracle response to these demands is Oracle Enterprise Manager Cloud Control 12c which offers complete life cycle cloud management – setup, deliver and manage clouds.

Once deployed IT must first setup the cloud infrastructure by defining share storage pools and compute resources. Administrators could also design a catalogue of virtual machines, assemblies, databases and applications and publish it.

It allows administrators to monitor usage, setup metering and chargeback for users, manage deployed applications, manage assets, link applications to MOS. It also allows quick to identify, diagnose and resolve incidents from a single console.

 Keynotes:

• The press release, Oracle Unveils Oracle Enterprise Manager 12c.
• Short flash presentation of Cloud Management with Oracle Enterprise Manager.
• Oracle Enterprise Manager Cloud Control 12c Video Series are available here.
• Meanwhile Oracle Learning is adding more videos to YouTube and they could be found here.
• The full installer (OMS, Agent and Repository) is available for Linux x86-64 (64-bit) and could be download from here.
• The Oracle Management Agent software can be downloaded using the Self Update feature within the Enterprise Manager Cloud Control console or use Self Update feature in offline mode then manually download the sar files available here.
• Documentation is available here, thanks to Gokhan Atil for the link.
• Installation requires Oracle Weblogic Server 10.3.5, which is already included in the package and installed by default.
• Installation requires certified  Oracle Database on which Oracle Management Repository will be configured. The certified versions of Oracle Database are from 10.2.0.5 onwards. The EM12c itself could manage databases from 9.2.0.8 onwards.
• You could upgrade your EM 10g GC Release 5 (10.2.0.5.0) or EM 11g GC Release 1 (11.1.0.1.0) to Enterprise Manager Cloud Control 12c (12.1.0.1).
• I’m still unable to find any MOS notes regarding EM12c – few notes already appeared at MOS, refer to last update for more information.

UPDATE 1:

• What’s New in Enterprise Manager 12c Install, here.
• Documentation library is now available here.

 UPDATE 2:

• Enterprise Manager Grid Control and Database Control Certification with 11g R2 Database [ID 1266977.1]
• If repository database is less than 11.2.0.2, additional packages has to be applied. Refer at MOS Certification for more information.

UPDATE 3:

• How to Install Enterprise Manager Cloud Control 12.1.0.1 (12c) [ID 1359176.1]
• EM12c: How to install Enterprise Manager Cloud Control 12c Agent [ID 1360183.1]
• Master Index for Cloud Control Agent Installation, Upgrade and Patching [ID 1363767.1]
• FAQ: Enterprise Manager Cloud Control 12c Install / Upgrade Frequently Asked Questions [ID 1363863.1]
• Master Index for Cloud Control Oracle Management Service (OMS) and Repository Installation, Upgrade and Patching [ID 1363769.1]

Regards,
Sve

Master Index for Cloud Control Oracle Management Service (OMS) and Repository Installation, Upgrade and Patching [ID 1363769.1]

Few days ago Oracle announced the release of Oracle Enterprise Manager Cloud Control 12c. I tried to summarize most of the information in my post so I’ll not discuss any details here, but I’ll go only with few details regarding EM12c installation.

For the purpose I have setup a VMWare virtual machine with 2 CPUs, 4GB RAM and 32 GB HDD, one network interface. Installed Oracle Enterprise Linux 6.1 (64 bit) with following parameters:

• Perform custom disk layout, I dedicated 4GB for swap and the rest for the root (/) file system and formatted it with ext4.
• Perform default installation, needed packages will be installed later.
• Set hostname, timezone and root password.
• After installation disable the firewall and most of some of the services, like IPV6.

After installation the network adapter won’t be available that why you have to install several packages and then compile the VMWare tools. Insert the installation DVD/ISO and install the following packages:

mount /dev/cdrom /mnt
rpm -ivh -ivh gcc-4.4.5-6.el6.x86_64.rpm cloog-ppl-0.15.7-1.2.el6.x86_64.rpm cpp-4.4.5-6.el6.x86_64.rpm glibc-devel-2.12-1.25.el6.x86_64.rpm
glibc-headers-2.12-1.25.el6.x86_64.rpm kernel-uek-headers-2.6.32-100.34.1.el6uek.x86_64.rpm ppl-0.10.2-11.el6.x86_64.rpm
mpfr-2.4.1-6.el6.x86_64.rpm kernel-uek-devel-2.6.32-100.34.1.el6uek.x86_64.rpm
umount /dev/cdrom

Then disconnect the drive and from the console go to VM->Guest->Install/Upgrade VMWare Tools, then install the guest additions:

cp /mnt/VMwareTools-8.3.2-257589.tar.gz /tmp
umount /mnt
cd /tmp
tar xfz VMwareTools-8.3.2-257589.tar.gz
cd vmware-tools-distrib
./vmware-install.pl

At this point you should be able to configure the network interfaces.

Before starting the installation, download the packages from OTN and transfer them to the server. The installation consist of two zip packages, which are 5.5GB total, but this includes Oracle Weblogic Server 10.3.5, which is installed by default from the wizard.

Oracle Enterprise Manager Cloud Control 12c installation prerequisites

For the installation of Enterprise Manager Cloud Control I’m following the documentation:
Oracle® Enterprise Manager Cloud Control Basic Installation Guide 12c Release 1 (12.1.0.1)

1. From Oracle Database 10.2.0.5 onwards, all versions are certified for Management Repository. The last two releases 11.2.0.2 and 11.2.0.3 do not need additional patches for it to be configured successfully. For the rest of the version additional patches are needed, refer to MOS for more information.

Except the support for Management Repository few more parameters are needed to be set. They could be set before or after the installation. For setting database initialization parameters refer to Table-6 or Table-7 from Appendix A at the documentation.

Once you are ready you could run the EM Prerequisite Kit which is run by the wizard during the installation.

2. According to the Oracle documentation for small environment, you need following servers parameters:
For the OMS: 2 CPUs, 4 GB RAM and 7 GB space excluding the installation which is 5.5 GB.
For the Management Repository: 2 CPUs, 2 GB RAM and 50 GB space.

3. Packages and kernel parameters required for OMS:
The following packages should be installer, either from ISO or from public yum server:

yum install make.x86_64 binutils.x86_64 libaio.x86_64 glibc-common.x86_64 libstdc++.x86_64 sysstat.x86_64 glibc-devel.i686 glibc-devel.x86_64

The shmmax kernel parameter should be set to value bigger than 4GB. In OEL 6.1 this parameter is far beyond and it’s set to 64GB. Its current value could be retrieved by following command:

cat /proc/sys/kernel/shmmax

4. Create group and user for the installation of Enterprise Manager 12c
The installation could not be done by root and oracle user has to be created. I’m using the same group id and user id as they would be created by oracle-validated package (which is not yet available for OEL 6.x).

groupadd -g 54321 oinstall
useradd -u 54321 -g oinstall -s /bin/bash -d /home/oracle -m oracle
passwd oracle

5. Configure limits.conf file:
The following two parameter has to be set in /etc/security/limits.conf file
oracle soft nofile 4096
oracle hard nofile 4096

Oracle Enterprise Manager Cloud Control 12c installation

Proceed with default installation and following the installation wizard.

If the wizard gives you warning at “Checking whether required GLIBC installed on system” although you have installed all the prerequisites you could ignore the warning. The installer is checking whether the package glibc-devel.i386 is installed, but you have already installed glibc-devel.i686.

Supply the repository details and please be sure to check whether the database control doesn’t exists. Otherwise you’ll get an error after supplying the database credentials to drop the database control of the repository database:
$ORACLE_HOME/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD <sys_password> -SYSMAN_PWD <sysman_password>

Once the installation is complete you’ll get a screen with installation summary and details how to access the console.

Then you could login and select default home page. This is how the console look like:

Meanwhile I just saw two useful installation guides:

• Installing Oracle Enterprise Manager 12c on OL 5.7 by Martin Bach
• How to Install Oracle Enterprise Manager Cloud Control 12c by Gokhan Atil

Regards,
Sve

Few years ago when I was Linux system administrator I already knew PHP and I used it to create monitoring scripts for the servers and services that I was responsible for. They were really simple application which I kept open in a browser window and integrated javascript to alert me in case some service goes down. The web design wasn’t my strong side, I guess people are born with it, but for my applications these were enough, although they were not as fancy and beautiful as I wanted to be. Then I start coding ASNI C (as a SA this is a must), created some client-server applications, loading data in MySQL and displaying it with PHP. Both PHP/C are much similar and programming in one of them clearly sets your mind of thinking in a way that it’s very easy to start the other. As system administrator I really liked using Bash, PHP, C for my daily duties. Writing applications, automating few process and protectively monitoring the systems, these kept my brain busy, made me think for new ideas and approaches and especially learned new things, I felt really positive and useful.

Now I’m an Oracle DBA, although I still do some SA and apart from my daily duties, I wanted to feel the same way. To create something that will be useful and make me satisfied, to learn new things. During the six years within this company, I saw a lot of things and found the company is missing a centralised tool for managing company assets, car park, people contacts details, certificates and more. I felt that this could be a chance to be useful and create let’s say corporate portal, integrate it with AD and have people use it.  I thought that Oracle APEX is the perfect fit for this case and perfect for me to learn APEX.

I’m pretty sure there are a lot of brilliant developers out there and I do not tend to call myself developer, but personally I feel like APEX is the development tool for me as an Oracle DBA. The very first moment I created and run my first application in APEX, I felt the same was as I was feeling back when I was writing PHP/C applications, I wanted to learn more. I got few books, read forums and online docs, but I think that the main progress I made with APEX was because I had an idea to follow.

Now, few months later I feel prepared to start developing corporate portal for the company needs. I’ve created SVN repo for this project, integrated SQL Data Modeler with SVN, created backup script with APEXExport and SVN, integrated APEX with Microsoft AD (both auth and load users from it), integrated it with Jasper for PDF reports, put some fancy jQuery for pulling data and at after all APEX is using Oracle Database. As I said, I do not tend to call myself developer, but as an Oracle DBA, I feel great and thankful that I can use APEX to create and support company applications.

That’s why I like Oracle APEX.

Regards,
Sve

As I mentioned in my previous post I’m using Subversion to keep my SQL design, APEX application and all application serving files. Concurrent Versions System are not something new and they are must for big projects. Although I’m single person, I’m finding SVN very useful for many reasons:

• All my files are kept at one place. At any time I could checkout the whole application and deploy it on different server.
• I have a history (revisions) of my changes. If I know something worked yesterday I could just checkout the file from yesterday and get it working.
• Using APEXExport I’m doing daily backups of my applications.
• Oracle SQL Developer and SQL Data Modeler are easy to integrate with SVN.
• When the time comes, I’ll create a branch, stable version of my application, by doing so I’ll have at any time a working and stable copy of my application.
• Having branches I could still develop and improve the application itself, if a bug appears I’ll be fixing it in the branch not in the main version (trunk).
• At any time more people could join the project and this would not stop the development process.

Just saw today, that Eddie Award retweeted Subversion Best Practices: Repository Structure, so I recommend you to get familiar with SVN first, before start using it.

Usually I’m using Debian for CVS systems, but in this case I’m installing Subversion on OEL 5.6, the procedure is the same for OEL 6.x.

First of all you need to install web server and svn packages. Assuming you have a configured repository, this is how to installed the packages:
yum install httpd mod_dav_svn subversion

If you want to change specific parameters you could edit Apache configuration file /etc/httpd/conf.d/httpd.conf, otherwise it’s not necessary.

Next you configure the SVN repository and authentication, the repository itself will be created next.

Edit file vi /etc/httpd/conf.d/subversion.conf and paste following:
<Location /svn>
DAV svn
SVNParentPath /var/www/svn
SVNListParentPath on
SVNIndexXSLT “/repos-web/view/repos.xsl”
AuthType Basic
AuthName “Oracle Repository authentication required”
AuthUserFile /etc/httpd/conf.d/svn.users
Require valid-user
</Location>

Create users who can access the repository:
htpasswd -cm /etc/httpd/conf.d/svn.users oracle
New password:
Re-type new password:
Adding password for user oracle
[root@db ~]# cat /etc/httpd/conf.d/svn.users
oracle:$apr1$9t19J…$hCF2GJTlizZfnPjKyk9rk/

Create the SVN repository:
mkdir /var/www/svn/
cd /var/www/svn/
svnadmin create oracle
chown -R apache:apache oracle

Finally restart apache and make sure it starts after reboot:
/etc/init.d/httpd restart
chkconfig httpd on

Troubleshooting:
In case you see the following error then most probably you’ve omitted the SVNListParentPath parameter:
Could not fetch resource information.  [403, #0]
(2)No such file or directory: The URI does not contain the name of a repository.  [403, #190001]

These is also an issue if you use SVNListParentPath and AuthzSVNAccessFile, bug description and workaround could be found here.

Conclusion:
Now you are ready to create you’re first project and start using SVN to maintain the source code of your applications. Given the example, the access URL would be http://hostname/svn/oracle or locally svn info file:///var/www/svn/oracle/. For maintaining the code you could use TortoiseSVN, which is excellent client for Windows.

The parameter I specified early SVNIndexXSLT will define the repository style once opened in a web browser. In my case I used a package called repo-style, which could be found here.

This is how my repository looks like:

plus the repository history looks really cool

It’s much better than the default one:

Regards,
Sve

This happened to be when I tryed to apply Bundle Patch 10 of Oracle Database 11.2.0.2 on Windows 2008, but I guess it could happen to any 11.x database version. I decided to apply this patch after I stepped the bug in which the heap memory is exhausted because of an CVU health checks (I described it here).

After running opatch apply I got that the following files are still active:
d:\app\11.2.0\grid\bin\oraclient11.dll
d:\app\11.2.0\grid\bin\orageneric11.dll
d:\app\11.2.0\grid\bin\orapls11.dll
d:\app\11.2.0\grid\bin\oracommon11.dll
d:\app\11.2.0\grid\bin\oci.dll
d:\app\11.2.0\grid\bin\orahasgen11.dll
d:\app\11.2.0\grid\bin\oraocr11.dll
d:\app\11.2.0\grid\bin\oraocrb11.dll
d:\app\11.2.0\grid\bin\oraocrutl11.dll
d:\app\11.2.0\grid\bin\mDNSResponder.exe
d:\app\11.2.0\grid\bin\ocssd.exe
d:\app\11.2.0\grid\bin\cssdagent.exe
d:\app\11.2.0\grid\bin\cssdmonitor.exe
d:\app\11.2.0\grid\bin\evmd.exe
d:\app\11.2.0\grid\bin\evmlogger.exe
d:\app\11.2.0\grid\bin\gipcd.exe
d:\app\11.2.0\grid\bin\gpnpd.exe
d:\app\11.2.0\grid\bin\octssd.exe

It’s unlikely to have something running, because I have stopped all GI processes. Again to find out which is the process holding the dll’s I’ve used ProcessExplorer. It seemed that process WmiPrvSE.exe had the dlls open:

Description of WMI:
The wmiprvse.exe file is otherwise known as Windows Management Instrumentation. It is a Microsoft Windows-based component that provides control and information about management in an enterprise environment. Developers use the wmiprvse.exe file in order to develop applications used for monitoring purposes.

For some reason WMI is holding the CRS dlls. Stop the WMI service or kill the process and this should release the lock on the drivers and allow the opatch to proceed.

Regards,
Sve

It’s not necessary to work on a big project to use SVN. Keeping your design into SVN repository is very useful and good practice. By doing so you could easily see what changes you’ve made to the design and in case of accident you could restore any older version and correct the problem.

After you successfully setup Subversion (here) it’s time to start integrating tools with SVN. Integration of Oracle SQL Data Modeler is really easy and it’s done in few simple steps:

1. Integrated Oracle SQL Data Modeler with SVN:
Start Oracle SQL Data Modeler
Go to View -> Team -> Versioning Navigator
At the Versioning Navigator window, right click on Subversion and then New Repository Connection.
Populate the fileds at new window Create Subversion Connection with repository URL, username, password and name for the repository.

2. Import files to the repository:
Once configured you have to import all the files on which you are working, including the database design to the repository. For this purpose you could use Data Modeler itself, but I prefer TortoiseSVN to create the repository structure and then import all the files in it. As I mention in preventions post, the best practice is to create three directories trunk, branch and tags and then import your working directories to trunk.

When these files are imported you should delete them from your hard drive and checkout the same files from the repository so you have a working copy.

3. Start using Oracle SQL Data Modeler and SVN:
Checking out from the repository could be done by TortoiseSVN or by Oracle SQL Data Modeler -> Versioning menu -> Checkout. Once you have a working copy you could open the design and start working on it.

Next time you make a change and save the design you’ll be able to commit these changes to the repository, again by going to Versioning menu and then Commit. Using comments is also good idea so you could easily track when and what changes you’ve made to the design.

Regards,
Sve

In case you need to run PHP with Oracle support this is how to do it with Oracle Enterprise Linux 5.x. I assume that Apache and PHP are already installed.

First you need to download the source of latest version of oci8 driver, here. Also you need to download and install the latest version of Oracle Instant Client Packages – Basic and SDK, here.

The next step is to install both packages on the system:

rpm -ivh oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64.rpm
rpm -ivh oracle-instantclient11.2-devel-11.2.0.3.0-1.x86_64.rpm

After installing the instant client, you have to build the oci8:

tar xfzv oci8-1.4.6.tgz
cd oci8-1.4.6
phpize
./configure -with-oci8=shared,instantclient,/usr/lib/oracle/11.2/client64/lib
make install

Module will be installed in PHP modules directory, which is /usr/lib64/php/modules/.

Edit php.init and add the following line to make oci8 driver to be loaded by PHP:
extension=oci8.so

After restarting the web server,  you now have Oracle support enabled in PHP. Oracle support could be tested from command line without restarting the web server, with this simple script:

cat > /tmp/oracle.php

<?php
$username = "scott";
$passwd = "tiger";
$db="(DESCRIPTION=
           (ADDRESS_LIST=
             (ADDRESS=(PROTOCOL=TCP)
               (HOST=192.168.8.36)(PORT=1521)
             )
           )
             (CONNECT_DATA=(SERVICE_NAME=orcl))
      )";
$conn = OCILogon($username,$passwd,$db);
if (!$conn)
{
    echo "Connection failed";
    echo "Error Message: [" . OCIError($conn) . "]";
    exit;
}
else
{
    echo "Connected!";
}

Running the script shows that it is connecting successfully to the database:

[root@app tmp]# php oracle.php
Connected!

There is also manual at PHP for installation and configuration of OCI8, which could be found here.

Regards,
Sve

It’s less than a day, tomorrow starts the Bulgarian Oracle User Group (BGOUG) conference. I’ve been looking for so long for this conference as I missed OOW this year. The conference will took place at Hisarya (near Plovdiv) for three days where there are session only at Friday and Saturday. We are proud that the conference will host seven Oracle ACE Directors and two Oracle ACEs.

I’m looking forward to meet and have a chat with these great people, of course with a lot of friends and have a nice and relaxing weekend.

This is the link to the conference event.

See you there

I was doing some installation at customer site when they asked if there anything specific to run GI 11.2 on HP-UX as this was their first interaction with 11g. Of course I replied that there is nothing specific, just to make sure the ownership of the raw disk is correct and had a correct ASM discovery string. They said that this is all done as it’s written in the documentation, but disks could not be discovered. This made me curious and asked them to log me in the system so I could have a look.

The system was running latest HP-UX 11.31 and we were going to install Oracle GI 11.2.0.2, the LUN was presented from HP EVA storage.

I couldn’t believe what they are saying and wanted them to show me what exactly they are doing. Unfortunately they were correct, after installing GI 11.2.0.2 software only, we tried to create an asm instance with asmca, but no disks were discovered although everything looked correct.

While I was looking around I remembered that the disk owner patch in HP-UX is a mandatory and it should be installed as the installation guide says this explicitly. I asked the customer and he said that all the required patches are installed, but when I checked the patch wasn’t  installed. The patch number as per installation guide is PHCO_41479, but the latest version is PHCO_41903. Also running kfed against disk on system on which the patch is not installed shows following:

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

I installed the patch and double checked everything and thought that this could be the reason why we are not seeing the disk, so I try to discover the disk, but again without success. The disk couldn’t be seen at ASM so I had to go deeper and see what asmca was actually doing. For the purpose I had to trace the system calls and for HP-UX the utility capable of doing this was tusc. There is MOS note describing how to trace systems call and what utilities should be used with different unix distributions [ ID 110888.1].

I run asmca and then using tusc got attached to its process, then changed the discovery string, pointing exactly to the disk I would like to use (in my case /dev/rdisk/disk3). So this is the paragraph which makes sense to me:

access("/dev/rdisk/disk3", W_OK|R_OK) ........................................................................... = 0
.......
open("/dev/rdisk/disk3", O_RDONLY|O_NDELAY|0x800, 0) ............................................................ = 7
lseek(7, 8192, SEEK_SET) ........................................................................................ = 8192
read(7, "L V M R E C 0 1 \r/ % aeN e2\va0".., 1024) ............................................................. = 1024
lseek(7, 73728, SEEK_SET) ....................................................................................... = 73728
read(7, "L V M R E C 0 1 \r/ % aeN e2\va0".., 1024) ............................................................. = 1024
close(7) ........................................................................................................ = 0

The disk is first successfully tested for read and write access and it’s opened for read-only in non-blocking mode. Then first 1024 bytes are read from offset 8192 from /dev/rdisk/disk3. This looked like a LVM header, AHA! So it seems that the disk was once used as LVM Physical Volume. Although the disk is not part of any volume group it has a LVM header and that’s why asmca it not showing this disk as CANDIDATE. It turned out that storage admins did not recreate the virtual disk on the storage, but the LUN was once used for LVM on another server.

After doing dd on the disk now the header looks better and disk could be seen as CANDIDATE:

oracle@vm:/$ dd if=/dev/zero of=/dev/rdisk/disk3 bs=1024k count=10

Now tusc output shows that header is filled with zeros:

read(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".., 1024) ........................................................ = 1024

Just for troubleshooting purpose I try to read the disk header with kfed, before and after showed the same error:

KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0]

If you are not sure whether the disk contains valuable information you could import the physical volume and activate the volume group. In my case I was sure that the disk should be deleted and simple dd do the job.

Regards,
Sve

Few days ago I received a call from customer about problem with their EM console and messages about file system full. They run DB 11.2.0.2 on OEL 5.7 and had only binaries installation at that file system and the database itself was using ASM. I quickly logged on to find out the file system was really full and after looking around I figure out that all the free space was eaten by alert and trace diagnostic directories. The trace directory was full of 10MB files and the alertlog file was quick growing with following messages:

 WARNING: failed to read mirror side 1 of virtual extent 2917 logical extent 0 of file 271 in group [1.2242406296] from disk DATA_0000 allocation unit 24394 reason error; if possible,will try another mirror side
Errors in file /oracle/app/oracle/diag/rdbms/baandb/baandb/trace/baandb_ora_17785.trc:
WARNING: Read Failed. group:1 disk:0 AU:24394 offset:1007616 size:8192
WARNING: failed to read mirror side 1 of virtual extent 2917 logical extent 0 of file 271 in group [1.2242406296] from disk DATA_0000 allocation unit 24394 reason error; if possible,will try another mirror side
Errors in file /oracle/app/oracle/diag/rdbms/baandb/baandb/trace/baandb_ora_17785.trc: 

At first I though there is a storage problem, but looking at the ASM views everything seemed to be all right and these seemed to be false messages. I deleted all the trace files, but then few minutes later the file system became again full. It turned out that generated log per minute were more than 60MBor around 7GB for two hours, because of this huge number of messages the machine was already loaded.

Then after quick MOS search I found that this is a Bug 10422126: FAILED TO READ MORROR SIDE 1 and there is a 70KB patch for 11.2.0.2.

The following MOS notes are also useful:
WARNING: ‘Failed To Read Mirror Side 1′ continuously reported in the alert log [ID 1289905.1]
Huge number of alert log entries: ‘WARNING: IO Failed…’ ‘WARNING: failed to read mirror side 1 of virtual extent …’ [ID 1274852.1]

After applying the patch everything became normal and no more false messages appeared in the logs. The bug is fixed in 11.2.0.3.

Regards,
Sve

It was busy month and I’m behind with my blog posts. First of all Happy New Year to all of you and wish you all the best through the new year!

This is a short note for installing Oracle Database with ASM under Oracle Enterprise Linux 6.x, which is not certified YET!

If you are running ASM with ASMLib you may get the following error when try to create ASM disk:

[root@db-app2 ~]# oracleasm createdisk DISK01 /dev/mapper/data01
Writing disk header: done
Instantiating disk: failed
Clearing disk header: done

Writing the header on the disk fails without obvious reason. Well, it turned out that Security Linux was blocking the access to the disk. Because configuration of SELinux is not part of the startup configuration any more I just forgot it.

To disable the SELinux edit file /etc/selinux/config and configure SELINUX=disabled. Then reboot your system and run getenforce to confirm that SELinux is disabled:

[root@db-app2 ~]# /usr/sbin/getenforce
Disabled

The ASM disk is created successfully this time:

[root@db-app2 ~]# oracleasm createdisk DISK01 /dev/mapper/data01
Writing disk header: done
Instantiating disk: done

Regards,
Sve

This is something I would like to share for a long time, but for one or another reason I didn’t. The error is nothing special, but then I run into very interesting case.

This happen when I had to shutdown one server for maintenance and then boot it again. Once the OS started I notice that the database didn’t ran automatically. When I try to run it manually I got the following error:

SQL> startup
 ORA-01078: failure in processing system parameters
 ORA-01565: error in identifying file '+DATA/orcl/spfileorcl.ora'
 ORA-17503: ksfdopn:2 Failed to open file +DATA/orcl/spfileorcl.ora
 ORA-15077: could not locate ASM instance serving a required diskgroup

The reason for this was that the ASM diskgroup wasn’t mounted, again I tried to mount it manually, when I got this error:

SQL> alter diskgroup data mount;
 alter diskgroup data mount
 *
 ERROR at line 1:
 ORA-15032: not all alterations performed
 ORA-15024: discovered duplicately numbered ASM disk 29

So how could this happen as I didn’t change any of the parameter and configuration, the database was started for just three months. Running quick script with kfed:

ls -1 /dev/oracle/*for i in `ls -1 /dev/oracle/*`; do echo /dev/oracle/$i; kfed read /dec/oracle/$i | grep kfdhdb.dskname; done

showed me that I there are really two duplicate disks. ASM does not support disks with the same disk names, even if this is the same disk, but accessed by different path. The duplicate disk was an old disk, which wasn’t removed at the time of the change, only its filename was changed so once ASM discovers disks in the future this one would be excluded.

Looking at the asm_diskstring I found that it has changed to ‘/dev/oracle/*’ and should be ‘/dev/oracle/orcl*’ and that’s why ASM was scanning all the disks. This made me think what could happen for the time database was running and why this happen.

After looking at the logs and made few tests I had a conclusion. At some point when the DBA added new disks to ASM he done it through the dbca and he changed the discovery string for unknown reason. What happens in background is that dbca, implicitly is recreating the ASM parameter file with the new discovery string entered by the DBA. This is why ASM started with wrong discovery string when I rebooted the server and it discovered more disks.

The database ran at 26 March and a month later at 29 April new disks were added to the ASM and its parameter was recreated, almost two months later at 7 July I turned it back with correct values:

-rw-r----- 1 oracle oinstall 1336 Jul 7 18:42 init.ora
-rw-r----- 1 oracle oinstall 1327 Apr 29 11:32 init.ora.662011221038

Regards,
Sve

In continue of my previous posts about APEX and Subversion I made a short script using APEXExport utility to automate and backup APEX to Subversion repository.

I’m using Oracle Express Edition 11.2, which by default doesn’t have the APEXExport utility. That’s why you need first to create the directory holding it and then copy the utility there.

Create destination directory holding the utility and copy APEXExport.class there:

mkdir -p $ORACLE_HOME/utilities/oracle/apex

Create backup directory in the SVN project for holding the backup files:

[oracle@dbsrv ~]$ svn mkdir -m "Making a backup dir." http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup

Committed revision 33.

At this point you’ll be asked for password which must be cached in order the automated backup to work correctly. The password itself is written (I assume you’re logged in with oracle user) in file in this directory: /home/oracle/.subversion/auth/svn.simple/

Finally you need to decide where the backup directory will be checked out. This will be used to store all the APEX backups, which will be committed to the SVN repository:

cd /oracle
svn checkout http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup

This is the script for automating the backup of APEX applications, because the formatting is not correct here is download link for the script:

#!/bin/bash

### Automated APEX applications backup to Subversion (SVN), v1
### 2012-03-07
### Svetoslav Gyurov, http://sve.to

# Environment settings for Oracle database and Java
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export JAVA_HOME=/oracle/ora112ee/jdk

export PATH=$ORACLE_HOME/bin:$PATH
export CLASSPATH=$ORACLE_HOME/jdbc/lib/ojdbc5.jar:$ORACLE_HOME/utilities

# How many days/backups should be kept back
export RETENTION=7
export COMPRESS=0

# SVN directory for Oracle APEX
SVN_DIR=/oracle/Backup

# Get current date and oldest backup
DATE=`date +%Y%m%d`
OLDEST_BACKUP=`date --date="-${RETENTION} days" +%Y%m%d`


# Change to SVN directory and export data
cd $SVN_DIR
$JAVA_HOME/bin/java oracle.apex.APEXExport -db 192.168.8.34:1522:XE -user sve -password secret -applicationid 100

exitValue=$?

# If APEXExport is successfully continue, otherwise fail with message
if [ $exitValue -eq 0 ];
then
svn update

for i in `ls -1 f*.sql`
do
if [ $COMPRESS -eq 1 ];
then
gzip $i
mv $i.gz ${DATE}_$i.gz
svn add ${DATE}_$i.gz

if [ -f ${OLDEST_BACKUP}_$i.gz ];
then
rm ${OLDEST_BACKUP}_$i.gz
svn remove ${OLDEST_BACKUP}_$i.gz
fi
else
mv $i ${DATE}_$i
svn add ${DATE}_$i

if [ -f ${OLDEST_BACKUP}_$i ];
then
rm ${OLDEST_BACKUP}_$i
svn remove ${OLDEST_BACKUP}_$i
fi
fi
done

svn commit -m "Daily APEX backup completed and committed."

echo Daily APEX backup completed.

else
echo "APEX applications could not be exported"
fi

Of course there are few parameters which need to be set. These are ORACLE_HOME, JAVA_HOME, RETENTION period defines how many backups to be kept back and COMPRESS defines whether the exported applications be compressed with gzip. Also don’t forget to change the application id, you could also put the whole workspace to be backed up.

Finally put the script at the crontab:

31 1 * * * /home/oracle/apex_backup.sh > /dev/null 2>&1

To check whether the backup was successful run svn info or svn log within the backup directory or just browse the repository:

cd /oracle/Backup
[oracle@dbsvn Backup]$ svn info
Path: .
URL: http://192.168.8.34/svn/oracle/Corporate_Portal/trunk/Apex/Backup
Repository Root: http://192.168.8.34/svn/oracle
Repository UUID: 40fb9d41-50a5-4b10-b803-b133fb623816
Revision: 59
Node Kind: directory
Schedule: normal
Last Changed Author: oracle
Last Changed Rev: 59
Last Changed Date: 2012-03-07 01:31:07 +0200 (Wed, 07 Mar 2012)

[oracle@dbsrv Backup]$ svn log
------------------------------------------------------------------------
r59 | oracle | 2012-03-07 01:31:07 +0200 (Wed, 07 Mar 2012) | 1 line

Daily APEX backup completed and commited.
------------------------------------------------------------------------

Feel free to modify the script for your own needs.

Regards,
Sve

Last month I created a simple APEX application with enabled mobile support and latest version of jQuery, which integrates with HP Service Manager through web services. The purpose was to give option for company engineers to open and update incidents through mobile in few easy steps.

The first step was to create form and report by using the web service. At this point web service request process is created where authentication and input parameter are described. The problem appears if you try to change any parameter and update the web service process. This is the error:

 Error error updating web service parameters
ORA-01403: no data found

To fix this, one option is to apply patch 12934733 on top of APEX 4.1. The other option is to apply latest patch set for APEX version 4.1.1, patch number 13331096.

At the time I got the error patch set wasn’t released yet and I went with the patch only to fix this issue. Later I’ve decided to update the APEX to latest version 4.1.1 and I’ll review the update process at glance.

To upgrade to APEX 4.1.1 make sure first to review the release notes here. The process is really simple and takes few minutes.

Before applying the patch make sure to prevent access to the APEX. In my case I’m using Oracle Database 11g Express Edition and I’m using Embedded PL/SQL gateway. Then apply the patch using apxpatch.sql and update the images directory. Because I’m using Express Edition, my images are stored in the XML DB repository and script apxldimg.sql has to be used to upload the new images within the repository.

Disabling Oracle XML DB HTTP Server:

SQL> SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;

GETHTTPPORT
-----------
 0

SQL> EXEC DBMS_XDB.SETHTTPPORT(0);

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

SQL> SELECT DBMS_XDB.GETHTTPPORT FROM DUAL;

GETHTTPPORT
-----------
 0

Run apxpatch.sql to patch the system:

SQL> @apxpatch.sql

.......

timing for: Complete Patch
Elapsed: 00:06:25.48

Updating the Images Directory When Running the Embedded PL/SQL Gateway:

@apxldimg.sql /tmp/patch

.......

Commit complete.

timing for: Load Images
Elapsed: 00:04:12.56

Directory dropped.

Enabling Oracle XML DB HTTP Server:

SQL> EXEC DBMS_XDB.SETHTTPPORT(8080);

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

APEX is now updated to version 4.1.1

Regards,
Sve

If for any reason you have a two databases running on the same server and one of them is Oracle 11g Express Edition then you will get surprised that listener of XE is not started automatically during boot if there is another listener already running. Debugging the startup script I found that this happens because of a bug in the code, which assumes that XE listener is running if it finds the word LISTENER within the processes list.

In file /etc/init.d/oracle-xe, at line 556 the code is failing:

+ status='oracle 2889 1 0 May13 ? 00:00:05 /oracle/ora112se/bin/tnslsnr LISTENER -inherit'
+ '[' 'oracle 2889 1 0 May13 ? 00:00:05 /oracle/ora112se/bin/tnslsnr LISTENER -inherit' == '' ']'

As you can see I have another database running and because of the XE startup script found the keyword LISTENER it’s supposing that the XE LISTENER is already running.

Simply changing line 556 from:

status=`ps -ef | grep tns | grep oracle`

To:

status=`ps -ef | grep tns | grep oracle | grep xe`

is fixing the error and now XE listener is started automatically during boot.

Regards,
Sve