The views expressed on this blog are my own and do not necessarily reflect the views of Oracle

May 4, 2014

ASM Disk Group Attributes

Disk group attributes were introduced in ASM version 11.1. They are bound to a disk group, rather than the ASM instance. Some attributes can be set only at the time the disk group is created, some only after the disk group is created, and some attributes can be set at any time, by altering the disk group.

This is the follow up on the ASM Attributes Directory post.

ACCESS_CONTROL.ENABLED

This attribute determines whether ASM File Access Control is enabled for a disk group. The value can be TRUE or FALSE (default).

If the attribute is set to TRUE, accessing ASM files is subject to access control. If FALSE, any user can access every file in the disk group. All other operations behave independently of this attribute.

This attribute can only be set when altering a disk group.

ACCESS_CONTROL.UMASK

This attribute determines which permissions are masked out on the creation of an ASM file for the owner, group, and others not in the user group. This attribute applies to all files in a disk group.

The values can be combinations of three digits {0|2|6} {0|2|6} {0|2|6}. The default is 066.

Setting to '0' does not mask anything. Setting to '2' masks out write permission. Setting to '6' masks out both read and write permissions.

Before setting the ACCESS_CONTROL.UMASK disk group attribute, the ACCESS_CONTROL.ENABLED has to be set to TRUE.

This attribute can only be set when altering a disk group.

AU_SIZE

The AU_SIZE attribute controls the allocation unit size and can only be set when creating the disk group.

It is worth spelling out that each disk group can have a different allocation unit size.

CELL.SMART_SCAN_CAPABLE [Exadata]

This attribute is applicable to Exadata when the disk group is created from the grid disks on the storage cells. It enables the smart scan functionality for the objects stored in that disk group.

COMPATIBLE.ASM

The value for the disk group COMPATIBLE.ASM attribute determines the minimum software version for an ASM instance that can use the disk group. This setting also affects the format of the ASM metadata structures.

The default value for the COMPATIBLE.ASM is 10.1, when using the CREATE DISKGROUP statement, the ASMCMD mkdg command and the Enterprise Manager Create Disk Group page.

When creating a disk group with the ASMCA, the default value is 11.2 in ASM version 11gR2 and 12.1 in ASM version 12c.

COMPATIBLE.RDBMS

The value for the COMPATIBLE.RDBMS attribute determines the minimum COMPATIBLE database initialization parameter setting for any database instance that is allowed to use the disk group.

Before advancing the COMPATIBLE.RDBMS attribute, ensure that the values for the COMPATIBLE initialization parameter for all databases that access the disk group are set to at least the value of the new setting for the COMPATIBLE.RDBMS.

COMPATIBLE.ADVM

The value for the COMPATIBLE.ADVM attribute determines whether the disk group can contain the ASM volumes. The value must be set to 11.2 or higher. Before setting this attribute, the COMPATIBLE.ASM value must be 11.2 or higher. Also, the ADVM volume drivers must be loaded in the supported environment.

By default, the value of the COMPATIBLE.ADVM attribute is empty until set.

CONTENT.CHECK [12c]

The CONTENT.CHECK attributes enables or disables the content checking when performing the disk group rebalance. The attribute values can be TRUE or FALSE.

The content checking can include Hardware Assisted Resilient Data (HARD) checks on user data, validation of file types from the file directory against the block contents and the file directory information, and mirror side comparison.

When the attribute is set to TRUE, the logical content checking is enabled for all rebalance operations.

The content checking is also known as the disk scrubbing feature.

CONTENT.TYPE [11.2.0.3, Exadata]

The CONTENT.TYPE attribute identifies the disk group type that can be DATA, RECOVERY or SYSTEM. It determines the distance to the nearest partner disk/failgroup. The default value is DATA which specifies a distance of 1, the value of RECOVERY specifies a distance of 3 and the value of SYSTEM specifies a distance of 5.

The distance of 1 simply means that ASM considers all disks for partnership. The distance of 3 means that every 3rd disk will be considered for partnership and the distance of 5 means that every 5th disk will be considered for partnership.

The attribute can be specified when creating or altering a disk group. If the CONTENT.TYPE attribute is set or changed using the ALTER DISKGROUP, the new configuration does not take effect until a disk group rebalance is explicitly run.

The CONTENT.TYPE attribute is only valid for NORMAL and HIGH redundancy disk groups. The COMPATIBLE.ASM attribute must be set to 11.2.0.3 or higher to enable the CONTENT.TYPE attribute.

DISK_REPAIR_TIME

The value of the DISK_REPAIR_TIME attribute determines the amount of time ASM will keep the disk offline, before dropping it. This is relevant to the fast mirror resync feature for which the COMPATIBLE.ASM attribute must be set to 11.1 or higher.

This attribute can only be set when altering a disk group.

FAILGROUP_REPAIR_TIME [12c]

The FAILGROUP_REPAIR_TIME attribute specifies a default repair time for the failure groups in the disk group. The failure group repair time is used if ASM determines that an entire failure group has failed. The default value is 24 hours. If there is a repair time specified for a disk, such as with the DROP AFTER clause of the ALTER DISKGROUP OFFLINE DISK statement, that disk repair time overrides the failure group repair time.

This attribute can only be set when altering a disk group and is only applicable to NORMAL and HIGH redundancy disk groups.

IDP.BOUNDARY and IDP.TYPE [Exadata]

These attributes are used to configure Exadata storage, and are relevant for the Intelligent Data Placement feature.

PHYS_META_REPLICATED [12c]

The PHYS_META_REPLICATED attribute tracks the replication status of a disk group. When the ASM compatibility of a disk group is advanced to 12.1 or higher, the physical metadata of each disk is replicated. This metadata includes the disk header, free space table blocks and allocation table blocks. The replication is performed online asynchronously. This attribute value is set to true by ASM if the physical metadata of every disk in the disk group has been replicated.

This attribute is only defined in a disk group with the COMPATIBLE.ASM set to 12.1 and higher. The attribute is read-only and is intended for information only - a user cannot set or change its value. The values are either TRUE or FALSE.

SECTOR_SIZE

The SECTOR_SIZE attribute specifies the sector size for disks in a disk group and can only be set when creating a disk group.

The values for the SECTOR_SIZE can be 512, 4096 or 4K (provided the disks support those values). The default value is platform dependent. The COMPATIBLE.ASM and COMPATIBLE.RDBMS attributes must be set to 11.2 or higher to set the sector size to a value other than the default value.

NOTE: ASM Cluster File System (ACFS) does not support 4 KB sector drives.

STORAGE.TYPE

The STORAGE.TYPE attribute specifies the type of the disks in the disk group. The possible values are EXADATA, PILLAR, ZFSAS and OTHER. If the attribute is set to EXADATA|PILLAR|ZFSAS then all disks in the disk group must be of that type. If the attribute is set to OTHER, any types of disks can be in the disk group.

If the STORAGE.TYPE disk group attribute is set to PILLAR or ZFSAS, the Hybrid Columnar Compression (HCC) functionality can be enabled for the objects in the disk group. Exadata already supports HCC.

NOTE: The ZFS storage must be provisioned through Direct NFS (dNFS) and the Pillar Axiom storage must be provisioned via the SCSI or the Fiber Channel interface.

To set the STORAGE.TYPE attribute, the COMPATIBLE.ASM and COMPATIBLE.RDBMS disk group attributes must be set to 11.2.0.3 or higher. For maximum support with ZFS storage, set COMPATIBLE.ASM and COMPATIBLE.RDBMS disk group attributes to 11.2.0.4 or higher.

The STORAGE.TYPE attribute can be set when creating or altering a disk group. The attribute cannot be set when clients are connected to the disk group. For example, the attribute cannot be set when an ADVM volume is enabled on the disk group.

The attribute is not visible in the V$ASM_ATTRIBUTE view or with the ASMCMD lsattr command until the attribute has been set.

THIN_PROVISIONED [12c]

The THIN_PROVISIONED attribute enables or disables the functionality to discard unused storage space after a disk group rebalance is completed. The attribute value can be TRUE or FALSE (default).

Storage vendor products that support thin provisioning have the capability to reuse the discarded storage space for a more efficient overall physical storage utilization.

APPLIANCE.MODE [11.2.0.4, Exadata]

The APPLIANCE.MODE attribute improves the disk rebalance completion time when dropping one or more ASM disks. This means that redundancy is restored faster after a (disk) failure. The attribute is automatically enabled when creating a new disk group in Exadata. Existing disk groups must explicitly set the attribute using the ALTER DISKGROUP command. This feature is also known as fixed partnering.

The attribute can only be enabled on disk groups that meet the following requirements:

The Oracle ASM disk group attribute COMPATIBLE.ASM is set to release 11.2.0.4, or later.
The CELL.SMART_SCAN_CAPABLE attribute is set to TRUE.
All disks in the disk group are the same type of disk, such as all hard disks or all flash disks.
All disks in the disk group are the same size.
All failure groups in the disk group have an equal number of disks.
No disk in the disk group is offline.

Minimum software: Oracle Exadata Storage Server Software release 11.2.3.3 running Oracle Database 11g Release 2 (11.2) release 11.2.0.4

NOTE: This feature is not available in Oracle Database version 12.1.0.1.

Hidden disk group attributes

Not all disk group attributes are documented. Here are some of the more interesting ones.

_REBALANCE_COMPACT

The _REBALANCE_COMPACT attribute is related to the compacting phase of the rebalance. The attribute value can be TRUE (default) or FALSE. Setting the attribute to FALSE, disables the compacting phase of the disk group rebalance.

_EXTENT_COUNTS

The _EXTENT_COUNTS attribute, is related to the variable extents size feature, that determines the points at which the extent size will be incremented.

The value of the attribute is “20000 20000 214748367”, which means that the first 20000 extent sizes will be 1 AU, next 20000 extents will have the size determined by the second value of the _EXTENT_SIZES attribute, and the rest will have the size determined by the third value of the _EXTENT_SIZES attribute.

_EXTENT_SIZES

The _EXTENT_SIZES is the second attribute relevant to the variable extents size feature, and it determines the extent size increments - in the number of AUs.

In ASM version 11.1 the attribute value is "1 8 64". In ASM version 11.2 and later, the value of the _EXTENT_SIZES is "1 4 16".

V$ASM_ATTRIBUTE view and ASMCMD lsattr command

The disk group attributes can be queried via the V$ASM_ATTRIBUTE view and the ASMCMD lsattr command.

This is one way to list the attributes for disk group PLAY:

$ asmcmd lsattr -G PLAY –l

Name                    Value
access_control.enabled  FALSE
access_control.umask     066
au_size                  4194304
cell.smart_scan_capable FALSE
compatible.asm           11.2.0.0.0
compatible.rdbms         11.2.0.0.0
disk_repair_time         3.6h
sector_size              512
$

The disk group attributes can be modified via SQL ALTER DISKGROUP SET ATTRIBUTE, ASMCMD setattr command and the ASMCA. This is an example of using the ASMCMD setattr command to modify the DISK_REPAIR_TIME attribute for disk group PLAY:

$ asmcmd setattr -G PLAY disk_repair_time '4.5 H'

Check the new value:

$ asmcmd lsattr -G PLAY -l disk_repair_time
Name              Value
disk_repair_time  4.5 H
$

Conclusion

Disk group attributes, introduced in ASM version 11.1, are a great way to fine tune the disk group capabilities. Some of the attributes are Exadata specific (as marked) and some are available in certain versions only (as marked). Most disk group attributes are documented and accessible via the V$ASM_ATTRIBUTE view. Some of the undocumented attributes were also discussed and those should not be modified unless advised by Oracle Support.

March 30, 2014

ASM spfile in a disk group

Starting with ASM version 11.2, the ASM spfile can be stored in an ASM disk group. Indeed, during a new ASM installation, the Oracle Universal Installer (OUI) will place the ASM spfile in the disk group that gets created during the installation. This is true for both Oracle Restart (single instance environments) and Cluster installations. It should be noted that the first disk group created during the installation is the default spfile location, but not a requirement. The spfile can still be on a file system, in say $ORACLE_HOME/dbs directory.

New ASMCMD commands

To support this feature, new ASMCMD commands were introduced to back up, copy and move the ASM spfile. The commands are:
  • spbackup - backs up an ASM spfile to a backup file. The backup file is not a special file type and is not identified as an spfile.
  • spcopy - copies an ASM spfile from the source location to an spfile in the destination location.
  • spmove - moves an ASM spfile from source to destination and automatically updates the GPnP profile.

The SQL commands CREATE PFILE FROM SPFILE and CREATE SPFILE FROM PFILE are still valid for the ASM spfile stored in the disk group.

ASM spfile in disk group DATA

In my environment, the ASM spfile is (somewhere) in the disk group DATA. Let's find it:

$ asmcmd find --type ASMPARAMETERFILE +DATA "*"
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.822856169

As we can see, the ASM spfile is in a special location and it has ASM file number 253. The ASM spfile stored in the disk group is a registry file, and will always be the ASM metadata file number 253.

Of course, we see the same thing from the sqlplus:

$ sqlplus / as sysasm

SQL> show parameter spfile

NAME   TYPE   VALUE
------ ------ -------------------------------------------------
spfile string +DATA/ASM/ASMPARAMETERFILE/registry.253.822856169

SQL>

Let's make a backup of that ASM spfile.

$ asmcmd spbackup +DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.822856169 /tmp/ASMspfile.backup

And check out the contents of the file:

$ strings /tmp/ASMspfile.backup
+ASM.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='RECO','ACFS'#Manual Mount
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

As we can see, this is a copy of the ASM spfile, that includes the parameters and associated comments.

ASM spfile discovery

So, how can the ASM instance read the spfile on startup, if the spfile is in a disk group that is not mounted yet? Not only that - the ASM doesn't really know which disk group has the spfile, or even if the spfile is in a disk group. And what is the value of the ASM discovery string?

The ASM Admin guide says this on the topic:

When an Oracle ASM instance searches for an initialization parameter file, the search order is:
  1. The location of the initialization parameter file specified in the Grid Plug and Play (GPnP) profile.
  2. If the location has not been set in the GPnP profile, then the search order changes to:
    1. SPFILE in the Oracle ASM instance home (e.g. $ORACLE_HOME/dbs/spfile+ASM.ora)
    2. PFILE in the Oracle ASM instance home

This does not tell us anything about the ASM discovery string, but at least it tells us about the spfile and the GPnP profile. It turns out the ASM discovery string is also in the GPnP profile. Here are the values from an Exadata environment:

$ gpnptool getpval -p=profile.xml -asm_dis -o-
o/*/*
$ gpnptool getpval -p=profile.xml -asm_spf -o-
+DBFS_DG/spfileASM.ora

There is no GPnP profile in a single instance set up, so this information is in the ASM resource (ora.asm), stored in the Oracle Local Repository (OLR). Here are the values from a single instance environment:

$ crsctl stat res ora.asm -p | egrep "ASM_DISKSTRING|SPFILE"
ASM_DISKSTRING=
SPFILE=+DATA/ASM/ASMPARAMETERFILE/registry.253.822856169

So far so good. Now the ASM knows where to look for ASM disks and where the spfile is. But the disk group is not mounted yet, as the ASM instance still hasn't started up, so how can ASM read the spfile?

The trick is in the ASM disk headers. To support the ASM spfile in a disk group, two new fields were added to the ASM disk header:
  • kfdhdb.spfile - Allocation unit number of the ASM spfile.
  • kfdhdb.spfflg - ASM spfile flag. If this value is 1, the ASM spfile is on this disk in allocation unit kfdhdb.spfile.

As part of the disk discovery process, the ASM instance reads the disk headers and looks for the spfile information. Once it finds the disks that have the spfile, it can read the actual initialization parameters.

Let's have a look at my disk group DATA. First check the disk group state and redundancy

$ asmcmd lsdg -g DATA | cut -c1-26
Inst_ID  State    Type
     1  MOUNTED  NORMAL

The disk group is mounted and the redundancy is normal. This means the ASM spfile will be mirrored, so we should see two disks with kfdhdb.spfile and kfdhdb.spfflg values set. Let's have a look:

$ for disk in `asmcmd lsdsk -G DATA --suppressheader`
> do
> echo $disk
> kfed read $disk | grep spf
> done
/dev/sdc1
kfdhdb.spfile:                       46 ; 0x0f4: 0x0000002e
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
/dev/sdd1
kfdhdb.spfile:                     2212 ; 0x0f4: 0x000008a4
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
/dev/sde1
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000

As we can see, two disks have the ASM spfile.

Let's check the contents of the Allocation Unit 46 on disk /dev/sdc1:

$ dd if=/dev/sdc1 bs=1048576 skip=46 count=1 | strings
+ASM.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='RECO','ACFS'#Manual Mount
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0352732 s, 29.7 MB/s

The AU 46 on disk /dev/sdc1 indeed contains the ASM spfile.

ASM spfile alias block

In addition to the new ASM disk header fields, there is a new metadata block type - KFBTYP_ASMSPFALS - that describes the ASM spfile alias. The ASM spfile alias block will be the last block in the ASM spfile.

Let's have a look at the last block of the Allocation Unit 46:

$ kfed read /dev/sdc1 aun=46 blkn=255
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           27 ; 0x002: KFBTYP_ASMSPFALS
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     255 ; 0x004: blk=255
kfbh.block.obj:                     253 ; 0x008: file=253
kfbh.check:                   806373865 ; 0x00c: 0x301049e9
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfspbals.incarn:              822856169 ; 0x000: 0x310bc9e9
kfspbals.blksz:                     512 ; 0x004: 0x00000200
kfspbals.size:                        3 ; 0x008: 0x0003
kfspbals.path.len:                    0 ; 0x00a: 0x0000
kfspbals.path.buf:                      ; 0x00c: length=0

There is not much in this metadata block. Most of the entries have the block header info (fields kfbh.*). The actual ASM spfile alias data (fields kfspbals.*) has only few entries. The spfile file incarnation (822856169) is part of the file name (REGISTRY.253.822856169), the block size is 512 (bytes) and the file size is 3 blocks. The path info is empty, meaning I don't actually have the ASM spfile alias.

Let's create one. I will first create a pfile from the existing spfile and then create the spfile alias from that pfile.

$ sqlplus / as sysasm

SQL> create pfile='/tmp/pfile+ASM.ora' from spfile;

File created.

SQL> shutdown abort;
ASM instance shutdown

SQL> startup pfile='/tmp/pfile+ASM.ora';
ASM instance started

Total System Global Area 1135747072 bytes
Fixed Size                  2297344 bytes
Variable Size            1108283904 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted

SQL> create spfile='+DATA/spfileASM.ora' from pfile='/tmp/pfile+ASM.ora';

File created.

SQL> exit

Looking for the ASM spfile again shows two entries:

$ asmcmd find --type ASMPARAMETERFILE +DATA "*"
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.843597139
+DATA/spfileASM.ora

We now see the ASM spfile itself (REGISTRY.253.843597139) and its alias (spfileASM.ora). Having a closer look at spfileASM.ora confirms this is indeed the alias for the registry file:

$ asmcmd ls -l +DATA/spfileASM.ora
Type              Redund  Striped  Time             Sys  Name
ASMPARAMETERFILE  MIRROR  COARSE   MAR 30 20:00:00  N    spfileASM.ora => +DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.843597139

Check the ASM spfile alias block now:

$ kfed read /dev/sdc1 aun=46 blkn=255
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           27 ; 0x002: KFBTYP_ASMSPFALS
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     255 ; 0x004: blk=255
kfbh.block.obj:                     253 ; 0x008: file=253
kfbh.check:                  2065104480 ; 0x00c: 0x7b16fe60
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfspbals.incarn:              843597139 ; 0x000: 0x32484553
kfspbals.blksz:                     512 ; 0x004: 0x00000200
kfspbals.size:                        3 ; 0x008: 0x0003
kfspbals.path.len:                   13 ; 0x00a: 0x000d
kfspbals.path.buf:        spfileASM.ora ; 0x00c: length=13

Now we see that the alias file name appears in the ASM spfile alias block. Note the new incarnation number, as this is a new ASM spfile, created from the pfile.

Conclusion

Starting with ASM version 11.2, the ASM spfile can be stored in an ASM disk group. To support this feature, we now have new ASMCMD commands and, under the covers, we have new ASM metadata structures.