This document provides information on a set of recommended best practices when deploying Data Domain storage systems for data protection and archive with the NFS (Network File System) protocol. This paper will provide insights on how to best tune NFS network components in order to optimize the NFS services on a Data Domain system. Best practices on configuring the NFS server, application clients, network connectivity and security.
Introduction:
Network file system (NFS) protocol originally developed by Sun Microsystems in 1983 is a client-server protocol, allowing a user on a client computer to access files over a network from the server as easily as if the network devices were attached to its local disks.NFS allows UNIX systems to share files, directories and files systems. The protocol is designed to be stateless.
Data Domain support NFS version 3 (NFSv3) the most commonly used version. It uses UDP or TCP and is based on the stateless protocol design. It includes some new features, such as a 64-bit file size, asynchronous writes and additional file attributes to reduce re-fetching.
Configuration:
DataDomain:
Make sure NFS service is running on the DD-- #nfs status
It is best practice to use hostname of the client while creating the nfs export.
Please make sure all the forward and reverse lookups of the hostnames in the nfs export list are correct in the DNS server. #net lookup
To add more than one client use a comma or space or both.
A client can be a fully-qualified domain hostname, class-C IP addresses, IP addresses with either netmasks or length, or an asterisk (*) wildcard with a domain name, such as *.yourcompany.com. An asterisk (*) by itself means no restrictions.
It is a best practice to keep the client entries in the nfs export list to a manageable state. This can be achieved by using acl entries like *.company.com, network address.
A client added to a sub-directory under /backup has access only to that sub-directory.
The are a comma-separated or space-separated or both list bounded by parentheses. With no options specified, the default options are rw, root_squash, no_all_squash, and secure.
The following options are allowed:
NFS Options
Following is the example of adding the nfs export client
#nfs add /backup 192.168.29.30/24 (rw,no_root_squash,no_all_squash,secure)
DD doesn’t support NFS locking. Please look at the NFS locking KB for detailed information.
It is always a best practice to use multiple NFS mounts on the client with multiple IPs on Data Domain for better performance.
For ex. TSM data is backed to /backup/TSM directory and SQL to /backup/SQL
Create separate NFS export for /backup/TSM and /backup/SQL
#nfs add /backup/TSM hostname.company.com (rw,no_root_squash,no_all_squash,secure)
#nfs add /backup/SQL hostname.company.com (rw,no_root_squash,no_all_squash,secure)
On the client mount them as two different mount points using two different IPs of DD.
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,rsize=1048600,wsize=1048600:/backup/TSM /ddr/TSM
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,rsize=1048600,wsize=1048600 <:/backup/SQL /ddr/SQL
Security
Port 2049 (NFS) and Port 2052 (mountd) must be open on the firewall if any existing.
It is a best practice for better security use the “root_squash” nfs export option while configuring the export.
Along with root_squash option set the anongid and anonuid to a specific ID on DD.
For example:
#nfs add /backup/TSM hostname.company.com (rw,root_squash,no_all_squash,secure,anongid=655,anonuid=655)
Configuration
Please make sure nfsd daemon is running on the client.
#/sbin/service nfs status
It is a best practice to create a separate MTREE for each media/Database server. This will help in performing less number of Meta data operations.
Check for stable writes. Run tcpdump when a backup is performed and check for stable flag in write packet. If clients are sending stable small writes less than 256KB, performance will degrade. It is due to pipeline commit behavior in 5.1, where there are long write latency with stable small writes. This issue is fixed in 5.1.1.3.
Mount option we recommend to use:
Linux OS:
# mount -t nfs -o hard,intr,nolock,nfsvers=3,tcp,timeo=1200,rsize=1048600,wsize=1048600,bg HOSTNAME:/backup /ddr/backup
# mount -t nfs -o hard,intr,nolock,nfsvers=3,tcp,timeo=1200rsize=1048600,wsize=1048600,bg HOSTNAME:/data/col1/ /ddr/
Solaris OS:
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,sec=sys,rsize=1048600,wsize=1048600 HOSTNAME:/backup /ddr/backup
# mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,sec=sys,rsize=1048600,wsize=1048600 HOSTNAME:/data/col1/ /ddr/
AIX OS:
# mount –V nfs –o intr,hard,llock,timeo=1200,rsize=65536,wsize=65536,vers=3,proto=tcp,\combehind,timeo=600,retrans=2 -p HOSTNAME:/backup /ddr
# mount –V nfs –o intr,hard,llock,timeo=1200,rsize=65536,wsize=65536,vers=3,proto=tcp,\combehind,timeo=600,retrans=2 -p HOSTNAME:/data/col1/ /ddr/
Network
If the client is having multiple network interfaces please make sure the routing is correct and the NFS I/O operations are using the correct interface.
Set the tcp buffers to the max value the client OS can support. For more detailed information please look at the below tuning guide for each client OS.
Using ping check the RTT between the client and the DD.
If there are any firewall between the media server and the DD make sure NFS port 2049 is open.
Please make sure the MTU size used is same throughout the data path. Verify that the MTU size is consistent between DDR and client. An inconsistent MTU will cause fragmentation which can lead to slow backups. The tracepath tool can be used to check mtu size:
#tracepath -n
Client OS Tuning
Linux OS
The Linux tuning guide describes how to optimize the backup and restore performance to DD. This tuning guide contains the tcp and memory tuning parameters and recommended mount options.
Please use the following tuning guide for more details: Linux tuning guide
Solaris OS
The Solaris tuning guide describes how to optimize the backup and restore performance to DD. This tuning guide contains the tcp and memory tuning parameters and recommended mount options.
Please use the following tuning guide for more details: Solaris Tuning guide
AIX and TSM tuning Guide
The AIX and TSM tuning guide describes how to improve backup performance within AIX/TSM environments through OS level tuning of the AIX server and TSM backup application.
Please use the following tuning guide for more details: AIX and TSM tuning guide
Note: DD recommends that you properly test all planned configurations in a testing environment before applying it to a production environment. You should also back up all your pre-tuning configurations.
Introduction:
Network file system (NFS) protocol originally developed by Sun Microsystems in 1983 is a client-server protocol, allowing a user on a client computer to access files over a network from the server as easily as if the network devices were attached to its local disks.NFS allows UNIX systems to share files, directories and files systems. The protocol is designed to be stateless.
Data Domain support NFS version 3 (NFSv3) the most commonly used version. It uses UDP or TCP and is based on the stateless protocol design. It includes some new features, such as a 64-bit file size, asynchronous writes and additional file attributes to reduce re-fetching.
Configuration:
DataDomain:
Make sure NFS service is running on the DD-- #nfs status
It is best practice to use hostname of the client while creating the nfs export.
Please make sure all the forward and reverse lookups of the hostnames in the nfs export list are correct in the DNS server. #net lookup
To add more than one client use a comma or space or both.
A client can be a fully-qualified domain hostname, class-C IP addresses, IP addresses with either netmasks or length, or an asterisk (*) wildcard with a domain name, such as *.yourcompany.com. An asterisk (*) by itself means no restrictions.
It is a best practice to keep the client entries in the nfs export list to a manageable state. This can be achieved by using acl entries like *.company.com, network address.
A client added to a sub-directory under /backup has access only to that sub-directory.
The
The following options are allowed:
NFS Options
Following is the example of adding the nfs export client
#nfs add /backup 192.168.29.30/24 (rw,no_root_squash,no_all_squash,secure)
DD doesn’t support NFS locking. Please look at the NFS locking KB for detailed information.
It is always a best practice to use multiple NFS mounts on the client with multiple IPs on Data Domain for better performance.
For ex. TSM data is backed to /backup/TSM directory and SQL to /backup/SQL
Create separate NFS export for /backup/TSM and /backup/SQL
#nfs add /backup/TSM hostname.company.com (rw,no_root_squash,no_all_squash,secure)
#nfs add /backup/SQL hostname.company.com (rw,no_root_squash,no_all_squash,secure)
On the client mount them as two different mount points using two different IPs of DD.
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,rsize=1048600,wsize=1048600
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,rsize=1048600,wsize=1048600 <
Security
Port 2049 (NFS) and Port 2052 (mountd) must be open on the firewall if any existing.
It is a best practice for better security use the “root_squash” nfs export option while configuring the export.
Along with root_squash option set the anongid and anonuid to a specific ID on DD.
For example:
#nfs add /backup/TSM hostname.company.com (rw,root_squash,no_all_squash,secure,anongid=655,anonuid=655)
Configuration
Please make sure nfsd daemon is running on the client.
#/sbin/service nfs status
It is a best practice to create a separate MTREE for each media/Database server. This will help in performing less number of Meta data operations.
Check for stable writes. Run tcpdump when a backup is performed and check for stable flag in write packet. If clients are sending stable small writes less than 256KB, performance will degrade. It is due to pipeline commit behavior in 5.1, where there are long write latency with stable small writes. This issue is fixed in 5.1.1.3.
Mount option we recommend to use:
Linux OS:
# mount -t nfs -o hard,intr,nolock,nfsvers=3,tcp,timeo=1200,rsize=1048600,wsize=1048600,bg HOSTNAME:/backup /ddr/backup
# mount -t nfs -o hard,intr,nolock,nfsvers=3,tcp,timeo=1200rsize=1048600,wsize=1048600,bg HOSTNAME:/data/col1/
Solaris OS:
#mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,sec=sys,rsize=1048600,wsize=1048600 HOSTNAME:/backup /ddr/backup
# mount -F nfs -o hard,intr,llock,vers=3,proto=tcp,timeo=1200,sec=sys,rsize=1048600,wsize=1048600 HOSTNAME:/data/col1/
AIX OS:
# mount –V nfs –o intr,hard,llock,timeo=1200,rsize=65536,wsize=65536,vers=3,proto=tcp,\combehind,timeo=600,retrans=2 -p HOSTNAME:/backup /ddr
# mount –V nfs –o intr,hard,llock,timeo=1200,rsize=65536,wsize=65536,vers=3,proto=tcp,\combehind,timeo=600,retrans=2 -p HOSTNAME:/data/col1/
Network
If the client is having multiple network interfaces please make sure the routing is correct and the NFS I/O operations are using the correct interface.
Set the tcp buffers to the max value the client OS can support. For more detailed information please look at the below tuning guide for each client OS.
Using ping check the RTT between the client and the DD.
If there are any firewall between the media server and the DD make sure NFS port 2049 is open.
Please make sure the MTU size used is same throughout the data path. Verify that the MTU size is consistent between DDR and client. An inconsistent MTU will cause fragmentation which can lead to slow backups. The tracepath tool can be used to check mtu size:
#tracepath -n
Client OS Tuning
Linux OS
The Linux tuning guide describes how to optimize the backup and restore performance to DD. This tuning guide contains the tcp and memory tuning parameters and recommended mount options.
Please use the following tuning guide for more details: Linux tuning guide
Solaris OS
The Solaris tuning guide describes how to optimize the backup and restore performance to DD. This tuning guide contains the tcp and memory tuning parameters and recommended mount options.
Please use the following tuning guide for more details: Solaris Tuning guide
AIX and TSM tuning Guide
The AIX and TSM tuning guide describes how to improve backup performance within AIX/TSM environments through OS level tuning of the AIX server and TSM backup application.
Please use the following tuning guide for more details: AIX and TSM tuning guide
Note: DD recommends that you properly test all planned configurations in a testing environment before applying it to a production environment. You should also back up all your pre-tuning configurations.
how to enable setgid? I have the directory permission set with 2775, and files were created not having group as the directory's.
ReplyDeleteThanks,