Friday, July 24, 2015

Passwordless Connection between RAC NOdes

Hi Team,

During the add node or  cluster upgradation,rdbms upgradation  or cluster installation or rdbms installation if you want to check any pre-requisites  using runcluvfy.sh script or cluvfy.sh script  it required password less connection between the RAC nodes for same user.

About SSH Keys

SSH keys provide a more secure way of logging into a virtual private server with SSH than using a password alone. While a password can eventually be cracked with a brute force attack, SSH keys are nearly impossible to decipher by brute force alone. Generating a key pair provides you with two long string of characters: a public and a private key. You can place the public key on any server, and then unlock it by connecting to it with a client that already has the private key. When the two match up, the system unlocks without the need for a password. You can increase security even more by protecting the private key with a passphrase.
Step1:  Passwordless connection between RAC NODES or same OS user across Nodes
[root@localhost ~]# 
[root@localhost ~]# su - grid              --DO for all Nodes
[grid@localhost ~]$ pwd
/home/grid
[grid@localhost ~]$ ls -lrt
total 0
[grid@localhost ~]$ cd .ssh
-bash: cd: .ssh: No such file or directory
[grid@localhost ~]$ mkdir .ssh
[grid@localhost ~]$ ls -lart
-rw-r--r-- 1 grid grid  124 Jul 24 11:01 .bashrc
-rw-r--r-- 1 grid grid  176 Jul 24 11:01 .bash_profile
-rw-r--r-- 1 grid grid   33 Jul 24 11:01 .bash_logout
drwxr-xr-x 4 root root 4096 Jul 24 11:01 ..
-rw------- 1 grid grid   17 Jul 24 11:02 .bash_history
drwxrwxr-x 2 grid grid 4096 Jul 24 11:03 .ssh
drwx------ 5 grid grid 4096 Jul 24 11:03 .
[grid@localhost ~]$ 

Note: If password less connection required for two different OS users across nodes then make sure the file perssion
By default drwxrwxr-x 2 grid grid 4096 Jul 24 11:03 .ssh
[grid@localhost ~]$ chmod 700 .ssh
it should be
drwx______ 2 grid grid 4096 Jul 24 11:03 .ssh

Step2:  Generate the SSH KEY
[grid@localhost .ssh]$ ssh-keygen  -t rsa       --Do for all Nodes
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
da:b0:2e:46:c1:54:87:ba:ec:ae:88:25:bf:17:f4:ca grid@localhost.localdomain

step3:   COpy the Key
grid@linux01:~$ ssh-copy-id -i grid@Node2   --repeat for all nodes , do from all other nodes else it will be one way no password less.
The authenticity of host 'remotecomputer (66.147.244.84)' can't be established.
RSA key fingerprint is   :  :  :  :  :  :  :  :  :  :  :  :  :  :  :  :.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remotecomputer,10.20.30.40' (RSA) to the list of known hosts.
grid@remotecomputer's password:
Now try logging into the machine, with "ssh 'grid@remotecomputer.com'", and check in:
~/.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
grid@linux01:~$

Tuesday, July 21, 2015

ASM password FILE

Pre 11gR2, each node in the cluster had ASM password file named as orapw<SID> .Starting 11gR2, password file in a RAC cluster is named using the format orapw<+’_asmsid’> .  “_asmsid” is an underscore parameter that defines default SID/Name of ASM instance.
If there are two nodes with +ASM1 running on node 1 and +ASM2 running on node2.
Pre 11gR2 --
Password file on Node1: orapw+ASM1
Password file on Node2: orapw+ASM2

11gR2 --
Password file on Node1: orapw+ASM
Password file on Node2: orapw+ASM
So on 11gR2, when password is changed for a privileged user on ASM instance of one node in a cluster, it is immediately reflected on all other nodes in the cluster.  But who does it ?  Here is a simple test:
[oracle@node1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Mon Aug 29 22:25:44 2011

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS
------------------------------ ----- ----- -----
SYS                            TRUE  TRUE  TRUE
ASMSNMP                        TRUE  FALSE FALSE
On Node2, lets check who is accessing the file
[root@node2 ~]# while true; do lsof | grep orapw+ASM; done
-- No Output so far--
-- lsof command means: lists open file descriptors.
-- This script will list processes which are currently accessing the file
orapw+ASM on node 2
Lets change password on ASM instance on Node 1 and check the output of shell script on Node2









SQL> alter user sys identified by ********;

User altered.

-- On Node 2 --
[root@node2 ~]# while true; do lsof | grep orapw+ASM; done
oracle    4863    oracle   22u      REG       8,16      1536     198556
/u02/app/11.2.0/grid/dbs/orapw+ASM

[root@node2 ~]# ps -ef |grep 4863
oracle    4863     1  0 22:22 ?        00:00:00 asm_ckpt_+ASM2
Another check that can be performed is:
On one of the nodes, suspend the ASM CKPT process.  Then try to execute a password change command (on ASM) from any other node in the cluster and it would hang.


ASMSNMP user in ASM 11g (user creation/password change)


Could not validate ASMSNMP password due to following error- “ORA-01031: insufficient privileges”
ORA-01990: error opening password file ‘/u02/app/11.2.0/grid/dbs/orapw’
ORA-15306: ASM password file update failed on at least one node
ORA-01918: user ‘ASMSNMP’ does not exist

Solutions:
Mostly the reason is missing asmsnmp user or missing password file in ASM instances.
Node1:
#su – grid
$. Oraenv
[ORACLE_SID]+ASM1
$sqlplus ‘/as sysasm’
 SQL> select * from v$pwfile_users;
no rows selected
 or
Node1 :
#su  - grid
$. Oraenv    +ASM1
$asmcmd
ASMCMD> lspwusr
Username sysdba sysoper sysasm
 This shows no user are present it means ASM password file is missing.  We have to recreate the password file.
Node1:
#su - grid
$. oraenv   +ASM1
$ cd $ORACLE_HOME/dbs
$ls -lrt
check whether orapw+ASM file exist or not . If not recreate it
$orapwd file=/u01/app/11.2.0.4/grid/dbs/orapw+ASM password=oracle force=y
copy the same file to rest all nodes .
$ scp orapw+ASM <Node2>:/u01/app/11.2.0.4/grid/dbs/
copy the password files to other nodes if you have more than 2 nodes in the RAC.
Now chek
#su - grid
$. oraenv  +ASM1
$sqlplus '/as sysasm'
SQL>select * from v$pwfile_users;
USERNAME              SYSDB                SYSOP             SYSAS
——————————————————————————————————————— —– —– —————————————————————————————————————–
SYS                    TRUE                TRUE                 TRUE
ASMSNMP                TRUE                FALSE                FALSE
 if still ASMSNMP user not exist, you create user under ASM instance
SQL>
create user and give sysdba privileges
SQL>create user asmsnmp identified by <make_asmsnmp_password>;
SQL> grant sysdba to asmsnmp;
SQL> show parameter pass
NAME                                   TYPE                            VALUE
———————————— —————————————————————————————– ——————————————————————————————————
remote_login_passwordfile             string                          EXCLUSIVE

The ASM instance parameter “REMOTE_LOGIN_PASSWORDFILE” has to be set to EXCLUSIVE or you will get an ORA-01999 error.
Also you can check  user detials from asmcmd
ASMCMD> lspwusr
Username        sysdba         sysoper       sysasm
SYS             TRUE           TRUE          FALSE
ASMSNMP         TRUE           FALSE         FALSE
 How to Change password for ASMSNMP
——————————————————–
ASMCMD> lspwusr
Username  sysdba sysoper sysasm
SYS       TRUE   TRUE     TRUE
ASMSNMP   TRUE   FALSE    FALSE
ASMCMD> orapwusr --modify --password ASMSNMP
Enter password: *******
(give new password and press enter)
orapwusr attempts to update passwords on all nodes in a cluster. The command requires the SYSASM privilege to run. A user logged in as SYSDBA cannot change its password using this command.

Test new asmsnmp password >>
————————–
$ sqlplus asmsnmp/asmsnmp as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Feb 6 12:57:30 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

Note:  You can visit for RAC/ASM/PT/RMAN/Golden gate classes in www.amktechs.com





Friday, July 10, 2015

How To Setup ASM & ASMLIB On Native Linux Multipath Mapper disks

How To Setup ASM & ASMLIB On Native Linux Multipath Mapper disks

After installed and configured the Multipath, follow bellow:
 We installed proper oracleasm rpms
2. Check the disks :
[root@linux01 ~]#ls -l /dev/mapper
brw-rw---- 1 root disk 253, 46 Jul 27 17:19 Mpatha01
brw-rw---- 1 root disk 253, 67 Jul 27 17:59 Mpatha02
brw-rw---- 1 root disk 253, 36 Jul 27 17:19 Mpatha03
brw-rw---- 1 root disk 253, 58 Jul 27 18:00 Mpatha04
 Note: above disks are partitioned from Luns.
3. Create the ASMLIB disks on mapper partitions as follow:
[root@linux01 ~]# /etc/init.d/oracleasm createdisk DSKORA1  /dev/mapper/Mpatha01
[root@linux01 ~]# /etc/init.d/oracleasm createdisk DSKORA2  /dev/mapper/Mpatha02
After create the disks, the ASM put a mark on the disks to know which are your own.
 4. If this is a RAC configuration, then from each node execute:
[root@linux01 ~]# /etc/init.d/oracleasm scandisks
[root@linux01 ~]# /etc/init.d/oracleasm listdisks

5. Configure ASMLIB to use multipath  (from each node on RAC environments):
[root@linux01 grid]# /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=" "
ORACLEASM_SCANEXCLUDE=" "
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

Note:  Now we have to update above two bold character parameters else ASM diskgroup creation or next ASM startup will not work properly for Disk Group.


Option1:
Modify in /etc/sysconfig/oracleasm :
ORACLEASM_SCANORDER=”dm”
ORACLEASM_SCANEXCLUDE=”sd”

Restart ASMLIB (from each node on RAC environments):
/etc/init.d/oracleasm stop
/etc/init.d/oracleasm start

Option2:
[root@linux01 grid]# /usr/sbin/oracleasm configure -i -e -u grid -g asmadmin -o "dm"  -x "sd"
[root@linux01 grid]# /usr/sbin/oracleasm exit
[root@linux01 grid]# /usr/sbin/oracleasm init
[root@linux01 grid]# /usr/sbin/oracleasm scandisks
[root@linux01 grid]# /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

6. Verify if the configuration is correct:
6.1 During the disk discovery, ASMLIB uses file /proc/partitions, see :
[root@linux01 grid]#  cat /proc/partitions
   8     0  877264896 sda
   8     1     104391 sda1
   8     2  877157032 sda2
   8    16  209715200 sdb
   8    32  382730240 sdc
   8    48  379596800 sdd
   8    64    2097152 sde
   8    80    2097152 sdf
   8    96 1169776640 sdg
 253     6  209715200 dm-6
 253     7  382730240 dm-7
 253     8  379596800 dm-8
 253     9    2097152 dm-9
 253    10    2097152 dm-10
6.2 The ASMLIB mount disks at /dev/oracleasm/disks, see:
# ls -la /dev/oracleasm/disks
brw-rw---- 1 grid asmadmin 2536 Ago 16 16:33 DSKORA1
brw-rw---- 1 grid asmadmin 2537 Ago 16 16:33 DSKORA2

6.3 Check if major and minor of disks “dm” not “sd”, is the same in /proc/partitions  and /dev/oracleasm/disks , see “253″ and “6″ bellow:
[root@linux01 grid]# cat /proc/partitions
253     6  209715200 dm-6
[root@linux01 grid]# ll /dev/oracleasm/disks
brw-rw---- 1 grid asmadmin 2536 Ago 16 16:33 DSKORA1

You can check using querydisk too :
[root@linux01 grid]#  /etc/init.d/oracleasm querydisk -d DSKORA1
Disk "DSKORA1" is valid ASM disk on device [253, 6]