Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Dishant J. Doriwala ([email protected]) System Software Engineer, PowerHA, RSCT FVT IBM 22 December 2015 (First published 22 December 2015) Pooja Bableshwar ([email protected]) Software Engineer, PowerHA FVT IBM This article provides detailed steps for the configuration of IBM® System Storage® SAN Volume Controller (SVC) with Peer-to-Peer Remote Copy (PPRC) and integrating it with IBM PowerHA®. It briefs about the SVC configuration with IBM PowerHA to provide PPRC technology for faster failover of data. SVC is block storage which does virtualization using Fiber Channel storage area network (SAN). Introduction This article explains the configuration of SVC with PowerHA to provide PPRC technology for faster fail over of data. SVC is block storage which does virtualization using Fiber Channel SAN. At present, based on high availability and disaster recovery requirements in data centers across locations, PPRC technology is implemented for data replication in IBM Power platform. In PPRC technology, a primary storage volume is mirrored to a secondary storage volume (remote site storage). PPRC mirroring technology can be Metro Mirror type or Global Mirror type. This article explains the configuration and integration of PowerHA using Metro Mirror copy. Metro Mirror copy is also referred as synchronous PPRC ,where every write operation to the primary volume is copied to the secondary volume. This I/O operation is complete only when the update to the primary and secondary volumes is complete. SVC PPRC configuration and operation Configuration demonstrated in this article consists of a site-based cluster network with two nodes on each site referred as production site and auxiliary site respectively as shown in Figure 1. IBM AIX® operating system and IBM PowerHA SystemMirror is installed on these nodes. Production site nodes (Node1SiteA and Node2SiteA) are connected to the primary storage system storwize1 (which has a master SVC cluster referred as the primary SVC cluster), whereas auxiliary site nodes ( Node1SiteB and Node2SiteB) are connected to a secondary storage system storwize2, © Copyright IBM Corporation 2015 Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Trademarks Page 1 of 12 developerWorks® ibm.com/developerWorks/ which has an auxiliary SVC cluster. The two IBM Storwize systems are connected with two or more intercluster links. The primary Storwize and the secondary Storwize systems consist of virtual disks. Each virtual disk of the primary Storwize system will be mapped to the virtual disk of the secondary Storwize system to form a relationship, which will be monitored by PowerHA. The PowerHA SystemMirror nodes at the same site access a shared volume group (VG) across cluster, but the volume group is accessed through different virtual disks on each site. From Figure 1, you can notice that the virtual disk on the primary storage is Node1Node2_SiteA and this is shared across the production site nodes and the virtual disk from the auxiliary storage is Node1Node2_SiteB and this is shared across the auxiliary site. PPRC can be applied between these virtual disks to perform mirroring. The SVC PPRC maintains separate identical local copies of the application data on two separate backend storage subsystems. But in the new Storwize system, the disks are internally allocated, and therefore, backend storages are not required. Virtual disks Node1Node2_SiteA and Node1Node2_SiteB are mirrored through the master SVC cluster, and PPRC from the primary site to the secondary site is established. When node or site failure occurs, all highly available applications are restarted at the secondary site using data copy on the secondary volumes. In general , the application is active on a server at the production site, and all updates to the application data are automatically replicated to the backup disk subsystem by the SVC PPRC framework. The mirrored virtual disks are defined as relationships, and each relationship is monitored by the SVC consistency group (which is referred as the SVC replicated resource in PowerHA SystemMirror). Figure 1. SVC PPRC hardware configuration Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 2 of 12 ibm.com/developerWorks/ developerWorks® Before proceeding with the configuration, ensure that the following required PowerHA SVC PPRC filesets are installed. • cluster.es.svcpprc.cmds • cluster.es.svcpprc.rte Test setup includes SVC Storwize V7000. Configuring SVC PPRC mirroring You need to perform the following steps to configure SVC PPRC mirroring. 1. Configure Secure Shell (SSH) password less authentication between IBM AIX® running logical partitions (LPARs) and Storwize systems. For running a command on the SVC command-line interface (CLI), make sure that passwordless authentication to the storages (Storwize systems) is enabled across all the nodes (AIX LPARs) of the cluster. Generate a public key on the AIX LPAR and after creating an administrator user on the Storwize system, add the SSH key of the AIX LPAR through the Storwize graphical user interface (GUI). 2. Clear the physical volume identifier (PVIDs) of all secondary site node (SiteB) disks. These disks are from the secondary storage connected only to storwize2. Initially, the PVIDs of the primary site disk and the secondary site disk will be different. Therefore, ensure that the PVIDs of the disks on the nodes in SiteB are cleared, so that they are assigned the same PVIDs that are assigned to the disks in SiteA. Without identical PVIDs, the import operation will fail on SiteB. Run the following command to clear the PVID. "chdev -l <hdiskno> -a pv=clear ". (0) root @ Node1SiteB: / # chdev -l hdisk16 -a pv=clear hdisk16 changed (0) root @ Node2SiteB: / # chdev -l hdisk16 -a pv=clear hdisk16 changed 3. Create a consistency group using CLI. Consistency group consist of a relationship that is formed by mirroring the virtual disks of the primary storage and the secondary storage. In this article, the storwize1 and storwize2 virtual disks are mapped to each other using the PPRC mirroring technology, and this mapping done between primary virtual disk and secondary virtual disk is called as relationship. Relationship created between the virtual disks is added to the consistency group. When you use a consistency group, you can prepare and start that group instead of the individual mappings. This process ensures that a consistent copy is made of all the source volumes. You can create a consistency group though CLI or GUI.. This article shows the creation of a consistency group using CLI. The general command to create consistency group is svctask mkrcconsistgrp. ssh admin@storwize1 svctask mkrcconsistgrp -name < CG_name> - cluster <remote_storage_cluster name> (0) root @ Node1SiteA: / # ssh admin@Storwize1 svctask mkrcconsistgrp -name DEMOCG -cluster storwize2 RC Consistency Group, id [5], successfully created Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 3 of 12 developerWorks® ibm.com/developerWorks/ Initially, the state of a consistency group, when created, will be empty. You can view detailed information for consistency group using the command, svcinfo lsrcconsistgrp. (1) root @ Node1SiteA: / # ssh admin@Storwize1 svcinfo lsrcconsistgrp DEMOCG id 5 name DEMOCG master_cluster_id 00000200A04109A2 master_cluster_name storwize1 aux_cluster_id 00000200A04109B2 aux_cluster_name storwize2 primary state empty relationship_count 0 freeze_time status sync copy_type empty_group cycling_mode cycle_period_seconds 0 4. Create relationship between the primary and secondary Storwize virtual disks. SVC PPRC relationship is formed between two different disks on the primary and auxiliary storages. At the storage level, they are termed as virtual disks and you can get information about a virtual disk using the svcinfo lshostvdiskmap command. The master virtual disk is the disk that resides at the primary site storage, and the auxiliary virtual disk is the disk at the backup site storage. Creating a relationship also allows to specify the type of mirroring (Global or Metro) to be created. Metro Mirror functions offer a synchronous remote copy option that constantly updates a secondary copy of a volume to match changes made to a source volume. In this article, creation of a relationship with Metro Mirror copy type is demonstrated. Before creating a relationship between the disks, user need to know the host virtual disk mapping. Check for the Universally Unique Identifier (UUID) of the disk that is to be used for creating a relationship using the lspv -u command on the AIX LPAR. Check for the same UUID in the Storwize system connected to the respective site. Take the corresponding virtual disk name for creating the relationship. The virtual disk name that corresponds to the hdisks in the LPARs can be looked up using the svcinfo lshostvdiskmap command. Figure 2. lshostvdiskmap output from storwize1 Figure 3. lshostvdiskmap output from storwize2 Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 4 of 12 ibm.com/developerWorks/ developerWorks® From Figure 2 and Figure 3, we can understand that the Node1Node2_SiteA virtual disk is shared across Site A LPARs connected to storwize1, whereas, the Node1Node2_SiteB virtual disk is shared across SiteB LPARs connected to storwize2. So a Metro Mirror copy type relationship will be formed between hdisk Node1Node2_SiteA and Node1Node2_SiteB , and this relationship will be added to the consistency group, DEMOCG. The svctask mkrcrelationship command is used to create the relationship. ssh admin@storwize1 svctask mkrcrelationship -master <master_vdisk> -aux <aux_vdisk> -name <relationship_name> -cluster <aux_cluster> -consistgrp <CG_name> (0) root @ Node1SiteA: / # ssh admin@Storwize1 svctask mkrcrelationship -master Node1Node2_SiteA -name Demo_Relation -cluster storwize2 -consistgrp DEMOCG -aux Node1Node2_SiteB RC Relationship, id [74], successfully created. Relationship created can also be listed out to view the detailed information. (0) root @ Node1SiteA: / # ssh [email protected] svcinfo lsrcrelationship Demo_Relation id 74 name Demo_Relation master_cluster_id 00000200A04109A2 master_cluster_name storwize1 master_vdisk_id 74 master_vdisk_name Node1Node2_SiteA aux_cluster_id 00000200A04109B2 aux_cluster_name storwize2 aux_vdisk_id 324 aux_vdisk_name Node1Node2_SiteB primary master consistency_group_id 5 consistency_group_name DEMOCG state inconsistent_stopped bg_copy_priority 50 progress freeze_time status online sync copy_type metro 5. Start the consistency group to perform proper mirroring between the relationship created. Run the svcinfo startrcconsistgrp command to start the consistency group. (0) root @ Node1SiteA: / # ssh admin@Storwize1 svctask startrcconsistgrp DEMOCG After the consistency group is started, the state of group changes to consistent_synchronized indicating that the primary volumes are accessible for read and write I/O operations. The secondary volumes are accessible for read-only I/Ooperations. So, when consistency groups are integrated with PowerHA, only SiteA volumes will be able to perform read/write I/O operations and SiteA storwize1 will act as the primary storage. (0) root @ Node1SiteA: / # ssh admin@Storwize1 svcinfo lsrcconsistgrp DEMOCG id 5 name DEMOCG master_cluster_id 00000200A04109A2 master_cluster_name storwize1 aux_cluster_id 00000200A04109B2 aux_cluster_name storwize2 primary master Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 5 of 12 developerWorks® ibm.com/developerWorks/ state consistent_synchronized relationship_count 1 freeze_time status sync copy_type metro cycling_mode cycle_period_seconds 300 RC_rel_id 74 RC_rel_name Demo_Relation When you run the lsrcconsistgrp command from the storwize1 system, primary master is displayed in the output and when you run the same command from storwize2, auxiliary master" is displayed. 6. Creating volume groups, logical volumes, and file systems on the disk where relationship is defined on the primary site. Note: The volume group should be created with a major number that is available across all the nodes of the cluster. The command to check the free major numbers across all nodes of cluster is lvlstmajor. (0) root @ Node1SiteA: / # lvlstmajor 45..49,51... Commands to create a volume group, logical volume, and file system → (0) root @ Node1SiteA: / # mkvg -f -V 60 -y demovg hdisk9 ---- VG with name demovg demovg → (0) root @ Node1SiteA: / # mklv -t jfs2 -y demolv demovg 40 ---- LV with name demolv demolv → (0) root @ Node1SiteA: / # crfs -v jfs2 -d /dev/demolv -m /demofs -A no --- filesystem with name demofs. File system created successfully. 163628 kilobytes total disk space. New File System size is 327680 7. After creating the volume group, create the file system on the primary site at Node1, stop the consistency group, unmount the file system and change the state of the volume group to inactive using the varyoffvg command. Run the svctask stoprcconsistrgrp command to stop the consistency group. (0) root @ Node1SiteA: / # ssh admin@Storwize1 svctask stoprcconsistgrp -access DEMOCG 8. Enable PVID on the secondary site nodes (Node1SiteB and Node2SiteB which are connected to storwize2). Run the following command to enable PVIDs: chdev -l hdiskXX -a pv=yes (0) root @ Node1SiteB: / # chdev -l hdisk16 -a pv=yes hdisk16 changed (0) root @ Node2SiteB: / # chdev -l hdisk16 -a pv=yes hdisk16 changed Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 6 of 12 ibm.com/developerWorks/ developerWorks® After enabling the PVID on the secondary site, PVID of a disk should be the same as that of primary site disk, between which the relationship is formed. 9. Import the volume group on all the nodes of the secondary site after enabling the PVID's . Ensure to use the same major number that was used at the time of creating the volume group. importvg -V <major_no> -y <VG_name> -f hdiskXX (0) root @ Node1SiteB: / # importvg -V 60 -y demovg -f hdisk16 demovg Then get back the volume group to the inactive state if it is in active state. Varyoffvg is required because PowerHA will take control of the volume group after the cluster services are started. 10. Start the consistency group again by providing the direction (if required), or else the consistency group will be the primary master on storwize1 and auxiliary on storwize2. After starting the consistency group, it will be again in the consistent_synchronized state. (0) root @ Node1SiteA: / # ssh [email protected] svctask startrcconsistgrp -primary master DEMOCG Integrating SVC PPRC with IBM PowerHA SystemMirror 1. Adding SVC clusters to IBM PowerHA SVC cluster details are added to IBM PowerHA to monitor mirror copies .Mirror copies are added to resource group, through which data is accessible, where resource group is active. Primary and auxiliary SVC clusters are added to PowerHA through the IBM AIX System Management Interface Tool (SMIT) from any one node of a cluster and change along cluster wide is seen after verification and synchronization. Here, SVC PPRC related resources are added from Node1SiteA. Open the SMIT interface using the smit hacmp command and select the following options to navigate to the screens shown in Figure 4 and Figure 5. Cluster Applications and Resources → Resources → Configure SVC PPRC-Replicated Resources → SVC Clusters Definition to PowerHA SystemMirror → Add an SVC cluster. Figure 4. Adding SVC primary cluster to PowerHA Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 7 of 12 developerWorks® ibm.com/developerWorks/ Figure 5. Adding SVC auxiliary cluster to PowerHA 2. Adding SVC PPRC relationship to PowerHA Relationships that are created at the storage level are added to IBM PowerHA. SVC relationships can be managed through a consistency group. SVC commands will be issued to the consistency group so it will fall over or fall back as a single entity. Ensure that the master and auxiliary SVC cluster name for the relationships listed here matches the master and auxiliary SVC cluster names for the consistency group. Adding relationship from Node1SiteA.Open the SMIT interface using the smit hacmp command and select the following options to navigate to the screen shown in Figure 6. Cluster Applications and Resources → Resources → Configure SVC PPRC-Replicated Resources → SVC PPRC Relationships Definition → Add an SVC PPRC Relationship Figure 6. SMIT menu to add relationship to PowerHA Refer to the following description for the entries listed in Figure 6. • Relationship Name: Name of relationship created at storwize level. • Master VDisk Info: Name of the virtual disk shared at SiteA , connected to the primary SVC storage. • Auxiliary VDisk Info: Name of the virtual disk shared at SiteB, connected to the secondary SVC storage. 3. Adding SVC consistency group to PowerHA PPRC replicated resources (consistency group) created at Storwize is added to PowerHA. This PPRC replicated resource are handled through Resource Group in PowerHA when cluster services are active. PPRC replicated resources are added to PowerHA from one node and the Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 8 of 12 ibm.com/developerWorks/ developerWorks® changes can be seen on all the nodes after verification and synchronization.Open the SMIT interface using the smit hacmp command and select the following options to navigate to the screen shown in Figure 7. Cluster Applications and Resources → Resources → Configure SVC PPRC-Replicated Resources → SVC PPRC-Replicated Resource Configuration → Add SVC PPRC Resource Figure 7. Adding PPRC replicated resource (consistency group) to PowerHA Refer to the following description for the entries listed in Figure 7. SVC PPRC Consistency Group Name: Consistency group name created from storage. Master SVC Cluster Name: Name of the master SVC cluster Auxiliary SVC Cluster Name: Name of the secondary SVC cluster List of Relationships: Names of the relationships created under a particular consistency group • Copy Type: Type of copy (GLOBAL or METRO) created while creating the relationship • PowerHA SystemMirror Recovery Action: Can be either AUTO or MANUAL. User can chose the recovery action to be taken from the PowerHA site in case of resource failover. If set to MANUAL, manual intervention is required to get the consistency group to proper state during failover, whereas, if it set to AUTO, no manual intervention is required. • • • • 4. Adding SVC PPRC replicated resource to resource group A consistency group, also referred as SVC PPRC replicated resource, is added to the resource group to monitor and handle the consistency group when cluster services are active. Whenever there is a failover of the resource group due to hardware failure or software failure, the consistency group under the resource group can be handle the failover as per the policies defined for the resource group. Open the SMIT interface using the smit hacmp command and select the following options to navigate to the screen shown in Figure 8. Cluster Applications and Resources → Resource Groups (select the resource group in which the PPRC replicated resource need to be added) → Change/Show Resources and Attributes for a Resource Group. Then add the consistency group as shown in the following figure. Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 9 of 12 developerWorks® ibm.com/developerWorks/ Figure 8. Adding consistency group to resource group to monitor after cluster services are active 5. Verifying PowerHA SVC PPRC configuration After adding the consistency group DEMOCG to the PowerHA resource group, run the cl_verify_svcpprc_config utility to verify that the configuration is properly added to PowerHA. It will list all the details of the Storwize system connected to each site, consistency groups created, relationship created with virtual disk, and also the consistency groups added to specific resource groups of PowerHA . In the output, (at the end) always verify for Status = 0, which means that SVC PPRC configuration is properly integrated with PowerHA. Figure 9. cl_verify_svcppc_config utility output to verify SVC PPRC configuration After verifying the configuration, perform verification and synchronization from the node where resources are added to PowerHA. This will synchronize cluster and cluster changes will be Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 10 of 12 ibm.com/developerWorks/ developerWorks® propagated on all nodes of cluster. On starting cluster services, resources will be online as per the policies. The storage on site that has resource group online acts as the master storage and the other site storage acts as the auxiliary storage. Summary This article helps to set up SVC PPRC configuration at the Storwize level and to integrate it with PowerHA SystemMirror. References • SVC replication resources • Defining SVC PPRC relationships • Adding SVC PPRC consistency groups to PowerHA SystemMirror resource groups Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 11 of 12 developerWorks® ibm.com/developerWorks/ About the authors Dishant J. Doriwala Dishant is currently working for the IBM PowerHA, RSCT FVT team. Dishant has three and half years of experience in testing PowerHA product for standard and XD clusters. Pooja Bableshwar Pooja is currently working for the IBM PowerHA test team. She has around six months of experience in testing the PowerHA product for standard and XD clusters. © Copyright IBM Corporation 2015 (www.ibm.com/legal/copytrade.shtml) Trademarks (www.ibm.com/developerworks/ibm/trademarks/) Configuration and integration of IBM System Storage SVC PPRC with IBM PowerHA using CLI Page 12 of 12