Client side scripting on DataDomain

The basic technique is to use ssh private/public key pairs to allow login and remote execution of commands without performing an interactive login. A private key is generated on the client, and the corresponding public key is stored on the Data Domain Restorer (DDR) using the adminaccess set ssh-keys command. After adding this public key to the DDR, subsequent logins are authenticated using the private key presented by the client. If the private key presented by the client matches the public key stored on the DDR, the login is permitted (without requiring a password).

The basis for this mechanism is ssh. There are a number of ssh implementations available for many different platforms, including open ssh, putty, etc. This document is written with openssh running on a linux client. A similar setup is possible on a Windows client using openssh with cygwin, or using putty.

Note: Data Domain devices con ONLY handle 10 concurrent ssh client sessions at once. Take this into consideratios when scheduling your scripted tasks.

Creating a public/private key pair

The first step to enabling “passwordless” login is to create a private/public key pair. Using openssh, the process is as shown in figure 1. The user is “fred” on a linux host named “cotton”. This example assumes that Fred’s .ssh directory starts completely empty (please move or rename any existing .sshdirectory before following this example).

Figure 1: Generating a private/public key (user input in red)

fred@cotton ~ $ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/auto/home/fred/.ssh/id_dsa): ¶
Created directory '/auto/home/fred/.ssh'.
Enter passphrase (empty for no passphrase): ¶
Enter same passphrase again: ¶
Your identification has been saved in /auto/home/fred/.ssh/id_dsa.
Your public key has been saved in /auto/home/fred/.ssh/id_dsa.pub.
The key fingerprint is:

df:16:ed:f2:a4:5a:a0:b8:bb:27:99:51:9b:cb:ee:48 fred@cotton.sample.com

fred@cotton ~ $ ls .ssh
id_dsa id_dsa.pub
Note that this example uses an empty passphrase. The return key was pressed at the “Enter passphrase (empty for no passphrase):” prompt (as indicated in blue in figure 1). Please note that this means the private key will be stored unencrypted on the local filesystem (in the file id_dsa). After completing the following steps, anyone with read access to /auto/home/fred/.ssh/id_dsa will be able run any command on the DDR without providing a password.

If the security of the local filesystem cannot be guaranteed, we recommend using a passphrase to encrypt the private key. In order to allow passwordless login, however, a passphrase encrypted private key will require the use of ssh-agent and ssh-add. These commands allow you to enter the passphrase once interactively, with subsequent commands using process memory to store the unencrypted password (hopefully more secure than the filesystem). The usage of these commands is beyond the scope of this document, however. The remainder of this document assumes the private key was stored without a passphrase.

Adding the public key to the DDR

Once the key pair has been created, the next step is to store the public key on the DDR. Note that the private key must be kept secret — anyone possessing this key will be able to run any command they wish on the DDR without providing a password. The public key, however, may be distributed freely (the public key is stored in id_dsa.pub). It is not possible to determine a private key from a public key. It is possible, however, to tell if a given private key matches a given public key.

The easiest way to enter the public key on the DDR is to use “cut and paste” with a mouse. First display the text using the following command:

Figure 2: Display the public key

fred@cotton ~ $ cat .ssh/id_dsa.pub
ssh-dss
AAAAB3NzaC1kc3MAAAEBALi3J6zJW57YWwgyQ6QXF9hq3MNOcKuR+PCk5q0qlE0zgqXtjbNAnJWP7gs7v9pXpBifEoA
0mUC7K2GZQKUMByU90r7j4oQJg/+2LG8AKhopkFHsOR+8cHDU0oppnwyQFoa1HOwrK7m9Uetsb8jWzIZ8D+tvcDcf0Y
zKY0ExI65VCxuthvLdqEZavM1btj/DQtUrUwB/OxbeV77Sl8HM2leQ4iw2VbihfxCfp6VrS5wIpNqfZdAZEhfE5AM2a
vS9/WdRY+ROj7Sb/5JyN2zK/A9MzlOnVQgTU58vyQm4XpLVJKyF7sbPK/Z6tLtbYhobVdVVeLb3IZbRjNbbN/X9CBUA
AAAVAKxCSxSTsrHaxoEeP5ff1QeKhPpPAAABAGKsikkkJA19M7pEByx/G0URB+ukQbnaF3fTuJ71arr0cHMpwqPkbko
2Kibyf7wP4TekPMaFz7vYCm2696lQn8XsUJvaYRNQ1ghEVCXfHm3IKebEPM7437OoKQUSzyKp9hETj8BcMX+RWotYa4
1OC/NiZ708POQxsVlhctdvFx+PxSeAcS1/JVvzLnlWU6U8W7Cyi83J4J+EHfICP0e2JwUfdbJXhrls7sk7QJMvkx7T/
5U8i6kvnPyrxl5J2PKvM5D0W7anEdHP/M/HkZ4kYoj++O8c4Ptautcf60qXDoxLmkhLUoKpZm+G9vaEI4B8G5JWYLFI
RQj8KZPG27O92vAAAAEBAI1I8uFOt71j6/+QHzru9tpY67NAMMlOMU4MiOUwpgWMkOaGH2ciXhNWGv1513BsH6mW8jY
p5aTMSwgyV0abPDM1lYrSSwhrd8bPeCU53m46UeyD1opaMPayKDmEAAaXxBydP2QU8t+WVizK+RzSUCSazpTltIMW5N
uatdp9Kr92A8Iq0E1x99Mq0nhXavskMxgJ6PX1g0SVLarBARDuVtbz7UsHcTZOGns2wLl/OivVhanfCIBv5ZKfRMqUa
OlADVtI26QYllNs5+XEqDvPod8964QZEVbkITIT7Jxj2VOaaR/HQGYA29J8EBD1iY1HBV+PIdgtoEk9/TCwxa2asW0=
fred@sample.com
Select and copy all of the lines of text using your mouse (the text beginning “ssh-dss” and ending “...W0=fred@sample.com”).

Storing the public key on the DDR

Next you will login to the DDR and store this public key. You must, of course, login to the DDR in order to enter these commands. Because the public key has not yet been added, you will need to provide the password for this login. This example will use the DDR named dd410.sample.com and the sysadmin user.

Figure 3: Add the public key to the DDR

fred@cotton ~ $ ssh sysadmin@dd410.sample.com
The authenticity of host 'dd410.sample.com (192.168.4.135)' can't be established.
RSA key fingerprint is 08:1d:ff:cf:2b:00:53:c8:10:e3:01:82:af:df:9f:2e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dd410.sample.com,192.168.4.135' (RSA) to the list of known hosts.
Password:
Last login: Sat Feb 17 16:18:50 2007 from cotton.sample.com

Welcome to Data Domain OS 4.0.5.0-34686
---------------------------------------
sysadmin@dd410-1# adminaccess show ssh-keys
sysadmin@dd410-1# adminaccess add ssh-keys

Enter the key and then press Control-D, or press Control-C to cancel.

ssh-dss
AAAAB3NzaC1kc3MAAAEBALi3J6zJW57YWwgyQ6QXF9hq3MNOcKuR+PCk5q0qlE0zgqXtjbNAnJWP7gs7v9pXpBifEoA
0mUC7K2GZQKUMByU90r7j4oQJg/+2LG8AKhopkFHsOR+8cHDU0oppnwyQFoa1HOwrK7m9Uetsb8jWzIZ8D+tvcDcf0Y
zKY0ExI65VCxuthvLdqEZavM1btj/DQtUrUwB/OxbeV77Sl8HM2leQ4iw2VbihfxCfp6VrS5wIpNqfZdAZEhfE5AM2a
vS9/WdRY+ROj7Sb/5JyN2zK/A9MzlOnVQgTU58vyQm4XpLVJKyF7sbPK/Z6tLtbYhobVdVVeLb3IZbRjNbbN/X9CBUA
AAAVAKxCSxSTsrHaxoEeP5ff1QeKhPpPAAABAGKsikkkJA19M7pEByx/G0URB+ukQbnaF3fTuJ71arr0cHMpwqPkbko
2Kibyf7wP4TekPMaFz7vYCm2696lQn8XsUJvaYRNQ1ghEVCXfHm3IKebEPM7437OoKQUSzyKp9hETj8BcMX+RWotYa4
1OC/NiZ708POQxsVlhctdvFx+PxSeAcS1/JVvzLnlWU6U8W7Cyi83J4J+EHfICP0e2JwUfdbJXhrls7sk7QJMvkx7T/
5U8i6kvnPyrxl5J2PKvM5D0W7anEdHP/M/HkZ4kYoj++O8c4Ptautcf60qXDoxLmkhLUoKpZm+G9vaEI4B8G5JWYLFI
RQj8KZPG27O92vAAAAEBAI1I8uFOt71j6/+QHzru9tpY67NAMMlOMU4MiOUwpgWMkOaGH2ciXhNWGv1513BsH6mW8jY
p5aTMSwgyV0abPDM1lYrSSwhrd8bPeCU53m46UeyD1opaMPayKDmEAAaXxBydP2QU8t+WVizK+RzSUCSazpTltIMW5N
uatdp9Kr92A8Iq0E1x99Mq0nhXavskMxgJ6PX1g0SVLarBARDuVtbz7UsHcTZOGns2wLl/OivVhanfCIBv5ZKfRMqUa
OlADVtI26QYllNs5+XEqDvPod8964QZEVbkITIT7Jxj2VOaaR/HQGYA29J8EBD1iY1HBV+PIdgtoEk9/TCwxa2asW0=
fred@sample.com
^D

Trying to add this much text without some form of “cut and paste” is far too difficult and error prone, but with sufficient effort, manual entry is possible (it’s also possible to generate a shorter key using the -b 512 option when generating the keys with the ssh-keygen command).

Figure 4: Verify only one key is stored and quit

sysadmin@dd410-1# adminaccess show ssh-keys
1         ssh-dss
AAAAB3NzaC1kc3MAAAEBALi3J6zJW57YWwgyQ6QXF9hq3MNOcKuR+PCk5q0qlE0zgqXtjbNAnJWP7gs7v9pXpBifEoA
0mUC7K2GZQKUMByU90r7j4oQJg/+2LG8AKhopkFHsOR+8cHDU0oppnwyQFoa1HOwrK7m9Uetsb8jWzIZ8D+tvcDcf0Y
zKY0ExI65VCxuthvLdqEZavM1btj/DQtUrUwB/OxbeV77Sl8HM2leQ4iw2VbihfxCfp6VrS5wIpNqfZdAZEhfE5AM2a
vS9/WdRY+ROj7Sb/5JyN2zK/A9MzlOnVQgTU58vyQm4XpLVJKyF7sbPK/Z6tLtbYhobVdVVeLb3IZbRjNbbN/X9CBUA
AAAVAKxCSxSTsrHaxoEeP5ff1QeKhPpPAAABAGKsikkkJA19M7pEByx/G0URB+ukQbnaF3fTuJ71arr0cHMpwqPkbko
2Kibyf7wP4TekPMaFz7vYCm2696lQn8XsUJvaYRNQ1ghEVCXfHm3IKebEPM7437OoKQUSzyKp9hETj8BcMX+RWotYa4
1OC/NiZ708POQxsVlhctdvFx+PxSeAcS1/JVvzLnlWU6U8W7Cyi83J4J+EHfICP0e2JwUfdbJXhrls7sk7QJMvkx7T/
5U8i6kvnPyrxl5J2PKvM5D0W7anEdHP/M/HkZ4kYoj++O8c4Ptautcf60qXDoxLmkhLUoKpZm+G9vaEI4B8G5JWYLFI
RQj8KZPG27O92vAAAAEBAI1I8uFOt71j6/+QHzru9tpY67NAMMlOMU4MiOUwpgWMkOaGH2ciXhNWGv1513BsH6mW8jY
p5aTMSwgyV0abPDM1lYrSSwhrd8bPeCU53m46UeyD1opaMPayKDmEAAaXxBydP2QU8t+WVizK+RzSUCSazpTltIMW5N
uatdp9Kr92A8Iq0E1x99Mq0nhXavskMxgJ6PX1g0SVLarBARDuVtbz7UsHcTZOGns2wLl/OivVhanfCIBv5ZKfRMqUa
OlADVtI26QYllNs5+XEqDvPod8964QZEVbkITIT7Jxj2VOaaR/HQGYA29J8EBD1iY1HBV+PIdgtoEk9/TCwxa2asW0=
fred@sample.com

sysadmin@dd410-1#quit
Connection to dd410-1.support.sample.local closed.
After adding the key, verify that the paste was performed correctly and that there are no extraneous characters orlinebreaks. The end result should be a single line returned by adminaccess show ssh-keys. If, instead ofoutput similar to figure 4 you see multiple lines, use the adminaccess del ssh-keys 1 command multiple timesuntil there are no more lines remaining, then repeat the process (using a different form of “cut and paste” or manually typing in the key if necessary).

Running commands remotely without a password

Now, if everything has gone correctly it should be possible to login to the DDR from the client without providing a password:

Figure 5: Verify that ssh from the client no longer requires a password

fred@cotton ~ $ssh sysadmin@dd410-1.support.sample.local
Last login: Mon Feb 19 09:35:59 2007 from cotton.sample.com

Welcome to Data Domain OS 4.0.5.0-34686
---------------------------------------
sysadmin@dd410-1#quit
Connection to dd410-1.support.sample.local closed.


Note that in addition to starting an interactive ddsh login without providing a password, it is also possible to run individual , non-interactive DD OS CLI commands by adding the command to the ssh command line (see figure 6).

Figure 6: Running non-interactive commands

fred@cotton ~ $ssh sysadmin@dd410-1.support.sample.local system show uptime
09:37:41 up 10 days, 2:19, 2 users, load average: 0.01, 0.03, 0.04
Filesystem has been up 0 days, 23:21.

fred@cotton ~ $ssh sysadmin@dd410-1.support.sample.local system show version
Data Domain OS 4.0.5.0-34686

fred@cotton ~ $ssh sysadmin@dd410-1.support.sample.local autosupport show report

========== GENERAL INFO ==========
GENERATED_ON=Mon Feb 19 09:45:34 PST 2007
VERSION=Data Domain OS 4.0.5.0-34686
[remaining text deleted]

Particularly useful commands for monitoring the status of a DDR include, but are obviously not limited to:

system show performance
replication status all
replication show stats all
filesys show space
filesys show compression last 24 hours
Sample Script

This technique can be used to write arbitrary scripts. An example of such a script is shown in figure 7. This script is intended to log the status of a replication context at ten minute intervals.

 Figure 7: Replication status logger

This script emits a single line every ten minutes containing a timestamp, the “destination remaining”, “source remaining, and “compressed remaining” fields from replication show stats; and the “state” and “connection” fields from replication status(refer to the Restorer User’s Guide for explanations for these fields). Sample output is shown in figure 8. Any non-zero values for any of the “remaining” fields indicates replication work remains to be done (the two DDRs aren’t in synch), while any value for “state” other than “normal” indicates a problem of some sort (a connection disruption for example).

Figure 8: Sample output from script

fred@cotton ~ $ ./repl_status_logger.pl
Mon Feb 19 16:19:01 2007 0 0 0 normal connected since Sun Feb 18 11:18:26
Mon Feb 19 16:29:01 2007 0 0 0 normal connected since Sun Feb 18 11:18:26
Mon Feb 19 16:39:01 2007 0 0 0 normal connected since Sun Feb 18 11:18:26
Mon Feb 19 16:49:01 2007 0 0 0 normal connected since Sun Feb 18 11:18:26
^C

As written, this script is not terribly robust. A number of improvements should be considered before using this script in production:

No error checking is performed. If the ssh command fails for any reason (another administrator removing the key, for example) the problem will almost certainly not be caught. If the DDR being monitored cannot be reached, a meaningful log entry should be generated.

The program runs in an endless loop, sleeping 10 seconds between each iteration. If the process were to be killed for any reason, however, the script would not be restarted. This script is a “daemon” and should be run under some form of daemon control that restarts the process if it dies for any reason (while simultaneously ensuring that there is never more than one process running at a time). The daemon should also be restarted at boot time.

The output is written directly to stdout. To make this more useful, the output should be redirected to a file, syslog, or some other logging system.There may be other fields of interest from the replication show stats or replication status commands that might be useful. If desired, other statistics could also be monitored and logged (network interface statistics, system load, etc.).

The configuration for this script is contained in hard-coded variables at the top of the script. To make the script more generally applicable, it might be worth passing in the configuration information via command-line arguments, a config file, or some other mechanism. Some form of error checking should also be performed on these configuration arguments: the INTERVAL variable, for example, should not normally be less than about 5 or 10 seconds to prevent undue load on the DDR being monitored (just the act of logging into the system via ssh or otherwise causes work unrelated to backup and restore).