mirrordir(1)mirrordir(1)NAME
pslogin - secure remote tcp login using strong stream cipher encryption
and diffie-hellman key exchanges.
forward - arbitrary tcp socket forwarding over a secure/encrypted chan‐
nel.
copydir, mirrordir - copy, mirror directory trees via a minimal set of
changes, locally or over FTP, or over a secure tcp connection.
recursdir - recurse through local or remote directories to command/find
files or create tar files.
SYNOPSIS
mirrordir
[-a, --access-times]
[-m, --strict-mtimes]
[--no-mtimes]
[--ignore-size]
[-A, --mtime-threshold ext]
[--time-offset [[+]|-][H]H[:MM]]
[-A, --always-write]
[-r, --restore-access]
[--no-chown]
[--no-chmod]
[-D, --only-delete]
[-b, -S, --backup-extension, --suffix ext]
[-N, --num-backups num]
[-O, --backup-outdate sec]
[-B, --block-size bytes]
[-M, --max-bytes num[K|M|G]]
[-s, --starting-file path]
[-i, --ignore-next-exclude]
[[-i] -X, --exclude path] [[-i] -X, --exclude path] ...
[[-i] -F, --exclude-from file] ...
[[-i] -G, --exclude-glob expr] [[-i] -G, --exclude-glob expr] ...
[[-i] -R, --exclude-regexp expr] [[-i] -R, --exclude-regexp expr] ...
[-C, --exclude-script [expr|file]] [-C, --exclude-script [expr|file]]
...
[-h, --help]
[-v, --verbose] [-v, --verbose] ...
[-V, --version]
[-k, --keep-files]
[-l, --no-hard-links]
[--follow-symlinks]
[-L, --strict-locking]
[-p, --password password]
[-P, --password-exact password]
[--test-login]
[--no-warn-first-login]
[--read-password-from-stdin]
[--allow-empty-ftp-dirs]
[--no-allow-empty-ftp-dirs]
[--netrc]
[--no-netrc]
[--proxy-host host]
[--secure]
[-z, --gzip]
[--gzip-backups]
[--case-insensitive]
[--to-lower]
[--to-upper]
[--no-use-passive-connections]
[-K, --key-size bits]
[--download-scripts]
[--tar-file filename]
[--tar-block-size N]
[-t, --dry-run, --test-only]
[--nice num] control mirror
mirrordir [-c | --copy-mode | --recurs-mode ] -[abBCdDFGhklMm‐
NOopRrstvVX] src [src ...] dest
copydir -[abBCdeFGhklMmNOopRrstvVX] src [src ...] dest
recursdir -[abBCdeFGhklMmNOopRrstvVX] src [src ...]
pslogin [--key-size bits] [mc://][username@]hostname[:portnum‐
ber][/path]
But usually just
mirrordir [--exclude path] control mirror
copydir src [src ...] dest
recursdir src [src ...] [-C program]
pslogin [username@]hostname[:portnumber]
DESCRIPTION
mirrordir is a set of useful utilities for manipulating and mirroring
directories. Included is also the command pslogin - an alternative to
ssh(1), and forward(1) for forwarding arbitrary TCP socket connections
over encrypted secure channels.
mirrordir copies files that are different between the directories con‐
trol and mirror to the directory mirror. Files whose modification times
or sizes differ are copied. File permissions, ownerships, modification
times, access times (only if --access-times is used), sticky bits, and
device types are duplicated. Symlinks are duplicated without any trans‐
lation. Symlink modification and access times (of the symlink itself,
not the file it points to) are not preserved. Hard linked files are
merely copied. Creation times cannot be set with Unix as far as I can
see.
mirrordir is a DANGEROUS command because files or directories that
exist in mirror that don't exist in control are deleted. If control is
entirely empty, then all files and directories in mirror will be
deleted. If mirror is entirely empty, then all files and directories in
control will be copied.
In short, mirrordir forces mirror to be an exact replica of the direc‐
tory tree control in every possible detail suitable for purposes of
timed backup. It naturally descends into subdirectories to all their
depths. mirrordir tries to be as efficient as possible by making the
minimal set of changes necessary to mirror the directory.
Access time duplication is not usually required and creates unnecessary
load. Hence it is given as an option.
The directory control is left untouched. If --restore-access is given
then access times are reset to their original with each read.
If the the --strict-locking option is on, files in control that are
copied are locked for `shared reading´. This will ensure, if another
process is busy writing to that file, that the file is not copied in
its incomplete or corrupted state.
Usually mirrordir will not exit, but will give error messages to stderr
to report any problems, and then will continue.
The directory mirror or dest must exist, even if it is empty.
Before erasing all the files in a directory, mirrordir checks for the
file *--keep-me (where * is zero or one characters). If this file is
present it will abort with an error message. Hence such a file can be
created in all directories that you are fearful of being recursively
erased.
copydir is equivalent to mirrordir -ck --no-erase-directories ...
(although -c implies -k anyway), so copydir is very much like a rigor‐
ous version of cp(1) where filenames can also be URLs, and only out‐
dated files are replaced. Use copydir instead of mirrordir for most
file transfers. Only use mirrordir, when you really want to delete
things.
recursdir is a further program that does nothing but descend into the
directories on the command line. It is equivalent to mirrordir
--recurs-mode ...
It was born after the -C option was added, and can be used as a more
rigorous version of find(1) and can also pack all the files it finds
into a tar file.
pslogin is yet a further program which has almost nothing to do with
the previous three. It envokes a secure login session using secure-
mcserv. It is equivalent to mirrordir --login-mode --secure ...
pslogin should be called logindir. See --login-mode below.
forward is yet a further program which has almost nothing to do with
first three. It can do forwarding of arbitrary services over a secure
channel. See forward(1) for details.
The importance of this package is that you can use URL's instead of
normal filenames, and hence manipulate files over a network. The URL
types currently supported are ftp:// and mc:// (http:// is not a
filesystem and therefore is not supported). mc:// is the Midnight Com‐
mander filesystem and is served by the secure-mcserv daemon. It has the
advantage of serving cryptographically strong secure file transfers and
logins.
You can also use glob expressions in filenames for the recursdir and
copydir commands. These will be recursively expanded.
SECURITY AND ENCRYPTION
mirrordir supports strong stream cipher encryption and Diffie-Hellman
key exchanges with several possible key sizes. Secure connections work
with mc:// type connections. See the options --secure, --key-size,
--download-scripts. See the EXAMPLES section for demo's and the FILES
section for where public/private keys are stored.
OPTIONS--help Print out detailed help, then exit.
--verbose
Specifies verbose output of file modifications made to mirror.
This option can be given multiple times for greater verbosity.
Output is written to stdout.
--restore-access
Restore the access times of control with each read.
--access-times
Duplicate even the access times of control.
--always-write
Force rewrite of every file regardless of whether they appear to
be identical or not.
--recurs-mode
This is set by default with recursdir. Listed directories are
read recursively and nothing is done to them. This option is
useful with -C in order to execute shell commands and search for
files. Note the system(), exec() and popen() functions available
to the C interpretor.
--login-mode
This option is set by default with pslogin. This approximates
rlogin(1) using secure-mcserv as a server. It is a secure
encrypted connection intended as a replacement to ssh(1). pslo‐
gin implies --secure as well. When using this option or when
using pslogin, only one path must be present on the command-
line. The path is of the form [mc://][username@]hostname[:port‐
number][/path].
Immediately after logging in, a cd /path will be executed at the
shell prompt. To determine if a prompt is available, pslogin
searches for a #, $ or > character. Should these not be found,
pslogin blocks indefinately. To avoid this behaviour, specify
/path as exactly /. This will leave you in your login directory.
Alternatively, change your shell prompt to include one of these
characters.
--copy-mode
This is set by default with copydir. Approximate the behaviour
of cp(1) as regards source and destination files. This option
implies --keep-files. It overwrites any existing file or direc‐
tory with the same name as a source file or directory, but does
not delete files or directories unnecessarily. Multiple source
files or directories may be given. The destination path must be
a directory.
--no-erase-directories
With this option, if a target directory conflicts with a source
file, and the directory is not empty, then an error message will
be printed and the program will abort. This is the default be‐
haviour for copydir.
--erase-directories
With this option, if a target directory conflicts with a source
file, then the target directory will be recursively deleted.
This is the default behaviour for mirrordir.
--allow-empty-ftp-dirs
Some ftp servers do not produce a . or .. directory. This makes
it appear as though you did not have permission to read the
directory. This option overrides this by assuming that such com‐
pletely empty directories are merely empty and do not have
errors. If you get unable to open directory: *: Permission
denied errors, then you can use this option. This option is
enabled by default (see next).
--no-allow-empty-ftp-dirs
Because we now cd to the directory to check for permission to
access it, the default behaviour is to allow empty directories.
--only-delete
Do not make any changes to mirror that will cause the total
space occupied by mirror to enlarge. This is a useful option
when backing up onto a drive that has limited space, where
changes have been made to control that might cause mirror to
grow larger during the duration of the transfer. Running mir‐
rordir once with this option and then once normally will ensure
that the available space is not overrun.
-i, --ignore-next-exclude
This option dictates that the next --exclude- type option must
completely overlook those files regardless of their existence or
non-existence in the mirror directory. It has the effect of a
return value of IGNORE for the --exclude-script option below.
This can be used to cause certain files to never be modified,
for example if you would like /etc/named.boot to never be modi‐
fied, use mirrordir /mnt/1 /mnt/2 -i--exclude
/mnt/1/etc/named.boot. Note that all paths must be specified in
the control directory and not the mirror directory, hence
--exclude /mnt/2/etc/named.boot won't work. This has the idiotic
behaviour that if you want to avoid erasing a file, you have to
have that file present in the control directory, even if it has
zero length.
--exclude path
Exclude file or directory path. A large number of excluded paths
on the command-line will slow performance. An excluded path will
be removed from the mirror tree if it already exists, in the
same way as any absent directory or file. Use the --ignore
option to ignore keep rather than delete these files.
If you have a long list of files to exclude, use the --exclude-
from option.
--exclude-glob glob
Exclude file or directory names matching glob style expression
glob. Matches the file without its full path. For shells, the
expression should be enclosed in appropriate quotes to prevent
substitutions.
--exclude-regexp regex
Exclude full pathnames of files or directories matching regex.
For shells, the expression should be enclosed in appropriate
quotes to prevent substitutions.
--exclude-script [expr|file]
Cause execution of the script expr for each file before doing
anything with that file. The script is a C style statement
block, terminating with a `return expression;´. The value of
expression can be one of INCLUDE, EXCLUDE, UNKNOWN or IGNORE
(see -i above) to explain what is to be done with that file. If
expr does not contain a semi-colon (;) then it is assumed to be
the name of a file - which is then loaded. In either case, the
text is byte compiled into reverse-polish notation for fast exe‐
cution. This option can be specified multiple times and the
scripts will be executed in order until a script returns some‐
thing other than UNKNOWN. A return value of UNKNOWN allows the
remaining --exclude- options on the command-line to take effect.
If you find that the interpretor incorrectly reports errors or
segfaults, please report the breaking script to me.
The scripting language itself is a subset of the full C program‐
ming language. For example, the following is a valid script:
/* PATH is the full name of the file including
its path, DIR is the directory, excluding
the trailing slash (/), CWD is the current
directory, and depth() returns the number
of forward slashes (/) less one. */
if (depth (DIR) - depth (CWD) > 3) {
printf ("%s: excluded\n", PATH);
return EXCLUDE;
} else
return INCLUDE;
The scripting language does not support the assignment operator,
and hence does not support user defined variables.
The following predefined macros are available. Note that the
expansion of the macros applies equally well to directories as
to files.
FILE current file without its path
NAME file name without its path or extension or trailing dot
(.)
EXTENSION
file extension without its leading dot (.)
DIR directory without file-name or trailing slash (/)
PATH full file name with path
CWD current working directory
TIME current time in seconds
All of the logical, arithmetic and bitwise C operators are sup‐
ported. These are ( ) >= <= > < != == && || ! - + * / % & ^ and
have the same meanings and precedences as in C.
The following further predefined macros are available. Each
returns an integer (type long int in C). These are based on a C
lstat (or stat if --follow-symlinks is used) on the file. See
stat(2) for a detailed explanation.
stat.st_dev - device
stat.st_ino - inode
stat.st_mode - permissions
stat.st_nlink - number of hard links
stat.st_uid - user id of owner
stat.st_gid - group id of owner
stat.st_rdev - device type
stat.st_size - file size in bytes
stat.st_blksize - block-size for file-system I/O
stat.st_blocks - number of blocks allocate
stat.st_atime - time of last accessed in seconds
stat.st_mtime - time of last modification in seconds
stat.st_ctime - time of creation
The following functions return boolean values:
strncmp(string1, string2, integer);
returns an integer less than, equal to, or greater than
zero if string1 is found, respectively, to be less than,
to match, or be greater than string2.
glob(glob, string);
returns zero if string matches glob expression glob. Try
to use only one glob expression in your code for effi‐
ciency of the underlying implementation.
regexp(regexp, string);
returns zero if string matches regular expression regexp.
Try to use only one regular expression in your code for
efficiency of the underlying implementation.
strstr(string1, string2);
returns the first occurance of string2 in string1 up to
the length of string1, or zero if it did not occur.
The following functions also return a boolean value and are
analogous to the corresponding macros explained in stat(2).
They return non-zero if the specified condition is true.
S_ISLNK(integer); - file is a sym-link
S_ISREG(integer); - file is a regular file
S_ISDIR(integer); - file is a directory
S_ISCHR(integer); - file is a character device
S_ISBLK(integer); - file is a block device
S_ISFIFO(integer); - file is an fifo
S_ISSOCK(integer); - file is a socket
The following functions manipulate strings:
strcat(string1, string2);
returns the concatenation of string1 with string2. Note
that the + operator also concatenates strings.
depth(string);
returns one less than the number of forward slashes (/)
in string.
printf(format, ...);
behaves like printf(3) with an important exception: only
long int format specifiers should be used. The behaviour
of anything that results in conversion of other than a
long int is undefined. For example, use "%ld" instead of
"%d". This function prints to stdout.
The following functions do system calls:
system(command);
executes /bin/sh -c command, but unlike the C version, it
returns the exit code of the command. I.e. it executes a
single line of shell script, command.
exec(argv0, argv1, ...);
executes process argv0 with arguments argv1.... argv0
must be a full path name. This is faster than system
because it need not envoke sh.
popen([string, ] shell_command);
like system, but returns the output of shell_command as a
string. If string is given, this writes string into the
standard input of shell_command and return zero on suc‐
cess.
The following further integer constants are available and are
analogous to the macros defined in stat.h and explained in
stat(2).
S_IFMT S_IFSOCK S_IFLNK S_IFREG S_IFBLK S_IFDIR S_IFCHR S_IFIFO
S_ISUID S_ISGID S_ISVTX S_IRWXU S_IRUSR S_IWUSR S_IXUSR S_IRWXG
S_IRGRP S_IWGRP S_IXGRP S_IRWXO S_IROTH S_IWOTH S_IXOTH
One of the the following constants should be returned using the
return keyword, and imply to the caller as follows. If nothing
is returned, the return value is assumed to be UNKNOWN.
UNKNOWN
didn't know what to do, continue with other --exclude-
options
INCLUDE
include the file
IGNORE do nothing with the file regardless of its existence or
non-existence in the mirror directory
EXCLUDE
consider the file to be non-existent in the control
directory and hence must be removed from the mirror
directory (this does not override the option --keep-
files)
The following perform flow control analogous to C:
The if clause causes statement1, statement2, etc. to be executed
if integer is true (i.e. non-zero), or otherwise causes state‐
ment1, statement2, etc. to be executed. The else {...} part is
optional.
if (integer) {
statement1;
statement2;
.
.
.
} else {
statementA;
statementB;
.
.
.
}
The return clause gives a value back to mirrordir and causes the
script to exit:
return expression;
The exit function cause mirrordir to exit with the specified
exit code.
exit(integer);
C scripts would typically be used to exclude types of files.
Note that this is an excessive implementation of a scripting
language, and all the features are not meant for general use. A
typical script will, for example, do nothing more than cause
device files to be excluded:
if (S_ISSOCK(stat.st_mode) || S_ISFIFO(stat.st_mode)
|| S_ISBLK(stat.st_mode) || S_ISCHR(stat.st_mode)) {
return EXCLUDE;
} else
return INCLUDE;
C scripts can also be used to search for files with the
--recurs-mode option (same as the recursdir command):
/* removes all core files */
/* This example has been moved to the EXAMPLES section. */
--exclude-from file
Exclude from a list of paths listed in the file file. Empty
lines and comment lines (with a # as the first character of the
line) are ignored. This list of files is sorted and binary
searched, so if you have lots of filenames to exclude, it is
best to include them here for performance. This option can be
specified multiple times with different files. BUG: the last
path in file must end with a newline.
--backup-extension level
Create backups of files before deleting or replacing them.
extension is a C style format string e.g. .ORIG.%d (be careful
of shell substitutions with the %). level is the highest number
of revision to keep. extension is appended to the filename, the
oldest file having the highest number.
--backup-outdate sec
Delete backup files older than sec seconds.
--nice num
Be nice to other processes by sleeping occasionally. num is a
small integer. --nice causes the process to sleep for as long
as it is active, times a factor of num. Hence a value of 1 will
(very roughly) double the time it takes to do a copy, and a
value of 3 will quadruple the time. This can be used where one
would like timed backups to place less load on the CPU. --nice
may not be available on your system.
--no-chmod
Normally the permissions of files are set. If you have
restricted access and cannot change permissions, than this can
be used to disable setting of permissions.
--no-chown
Normally the ownerships of files are set. If you have restricted
access and cannot change ownerships, than this can be used to
disable setting of ownerships.
--mtime-threshold sec
This is the deviation in mtime that is allowed for a file before
it is overwritten. If you have mirrored an ftp site, the mtimes
are accurate to within a minute only, thereafter an nfs mirror
will cause every file to be copied: you can then use --mtime-
threshold 60 to fix this.
--time-offset [[+]|-][H]H[:MM]
This sets the time offset of any vfs (i.e. non-local) directory.
For instance, I am 8 hours east of New York, so when I mirror
from New York, I use --time-offset -08:00.
--test-only, --dry-run
Do not make any changes. If used with --verbose, this will show
what changes would be made. This is an effective way of compar‐
ing directory trees. This is untested - i.e. I don't know
whether this option actually makes any changes or not!
--skip-symlinks
Symlinks are treated as though they were not read - hence if
they are found in the mirror directory then they are deleted.
--keep-files
Don't remove files from mirror, even if they don't exist in con‐
trol. This makes mirrordir somewhat like cp(1).
--no-hard-links
mirrordir mirrors hard-links properly unless this option is set,
in which case hard-links are copied as regular files.
--follow-symlinks
mirrordir mirrors symlinks properly unless this option is set,
in which case symlinks are copied as regular files. Useful for
mirroring the Debian tree. Note that this follows symlinks in
both the control and mirror directories, hence if symlinks
already exist in the mirror directory, these will be left as
symlinks. Note that --follow-symlinks implies --no-hard-links
also.
--strict-locking
Create shared read locks on files as they are read. This pre‐
vents clashes especially when copying files from the mail direc‐
tory - mail programs will be trying to write to those files at
the same time as mirrordir is trying to to read from them. This
option has no effect with virtual file systems.
--max-bytes [[num[k|M|G]]|num]
Maximum number of bytes to write before giving the message,
`filled up all blocks - first file/dir not mirrored: path',
which is written to stdout. The remaining files are deleted from
mirror, but in their listed order - hence it is possible that
the archive will grow greater than num while mirrordir is still
running. You should make allowance for this eventuality by mak‐
ing num smaller than the maximum available space. Also note that
some filesystems will give an error message `No space left on
device' before the device is completely full. You can restart
mirrordir with the option --starting-file path and hence con‐
tinue with this file on another volume. This enables mirrordir
to back up across different devices. num can be appended by k, M
or G (case insensitive) to specify kilobytes, megabytes or giga‐
bytes respectively. If any single file is greater than this num‐
ber, then an error message will be given. See also --block-size.
--password password
Specify the password for FTP and mc:// connections. The anony‐
mous password defaults to your login name @ your local machine
name. You will be prompted for other login passwords. As usual,
you are warned that including passwords in scripts is a security
risk. It is much better to put the password in your ~/.netrc
file and then not use the --no-netrc option; see man ftp for
details.
--password-exact password]
Don't prepend a - to the anonymous password. With ftp anonymous
passwords, a - is usually prepended to the password string. I
don't know why the Midnight Commander vfs did this, but one user
had problems with it, hence this option is given to send the
password exactly as specified by password.
--test-login
When using --login-mode or pslogin, you may want to test access
non-interactively (eg. in a shell script). To do this you can
run pslogin with this option and then check its exit status.
secure-mcserv uses this to verify if the user can login on the
password server.
--no-warn-first-login
The first time you try a secure connection to a machine, no pub‐
lic key exists on the local machine. There is hence no security
against a man-in-the-middle attack. A warning to this effect is
printed and the user is prompted if they want to continue. This
option disables this warning and goes ahead regardless.
--read-password-from-stdin
Instead of specifying the password on the command line, you can
write it into the command via stdin. This is not the same as
typing the invisible password as one normally does, because it
can be used even if there is no terminal. This is useful for
usage within other programs using, say popen(3). secure-mcserv
uses this to verify if the user can login on the password
server.
--netrc
Scan the ~/.netrc file. By default, this option is on.
--no-netrc
Turn off reading of ~/.netrc file.
--proxy-host host
Set the proxy for ftp downloads. Don't know how or if this
works. Consult the mc(1) man page for info in proxy support.
--secure
(This feature is BETA) I have implemented a secure socket layer
for mirrordir. It is enabled with this option and applies to
connections to secure-mcserv (i.e. using mc:// type URLs). The
secure socket library consists of libdiffie.a and a header file
diffie-socket.h. If you include diffie-socket.h after you
include sys/socket.h you can recompile the resulting program
with all normal sockets turned into secure sockets. (This
applies to any program that uses Unix socket calls, but is
untested). The first service supporting this is secure-mcserv
which compiles and installs by default. Hence you can use the
mc:// file system with the --secure option, provided the remote
host is running secure-mcserv (try secure-mcserv -h to for
help). --secure uses a stream cipher (much more secure and
faster than block ciphers like DES) with discrete logarithm key
exchanges with public key server authentication (Diffie-Hellman
and p-NEW schemes). For the full details, see the file diffie-
socket.h in the source distribution. The default key size is 512
bits. Note that if you are using gcc you should compile mir‐
rordir with the options -O3 -fomit-frame-pointer -s -Wall to
speed key generation.
--key-size bits
The default key size is 512 bits. Sizes require primes to be
generated and compiled into the file field.c, hence only those
primes listed in field.c are supported, which are at the moment
512, 768, 1024 and 1536. 768 is the recommended value for medium
security if you have slow computer. Otherwise 1536 is not an
unreasonably paranoid size for long term security. 512 can be
used if you are not worried about large corporations, well con‐
nected hackers, or governments snooping your connections. Note
that the the stream cipher has a length of bits/2, which means
that you have more chance of being hit by a meteor while cashing
in at the state lottery, than cracking it. Be aware that a per‐
son can sniff your connection and then keep the output for as
long as it takes to break it. In twenty years time a 1536 bit
key will be considered small. Note also that the discrete loga‐
rithm problem (used here) is considered to be more difficult to
solve than the factorisation problem, hence the key is effec‐
tively a little larger than for RSA. This is all my (rather
uneducated) opinion.
--download-scripts
Mirrordir comes in two versions, an International version and a
US version (see --version). The US version contains no encryp‐
tion code whatsoever. Instead it downloads the needed algorithms
from encrypt.obsidian.co.za (in South Africa). These are writ‐
ten in a fast, native, C-style interpreted language. There are
four scripts: one for the server Diffie-Hellman key exchange,
one for the client Diffie-Hellman key exchange, one for initial‐
ising the stream cipher and one for actually performing the
encryption using the stream cipher. Mirrordir automatically
downloads these scripts when you try to use the security fea‐
tures. The option --download-scripts however can be used to
force a download at any time. The International version contains
compiled-in versions of the stream cipher, hence only two
scripts are used, which need never be downloaded since they come
with the distribution. No speed penalty is incurred by having
the Diffie-Hellman exchanges in scripts, however, you will
notice a significant speed different when using the encryption
scripts compared to compiled-in encryption.
--version
Prints out the version number as well as whether this is an
International or US version of mirrordir. See --download-
scripts.
-z, --gzip
Enable compression for mc:// connections. This actually envokes
a compressed socket layer at a lower level than the encryption.
Compression uses the libz library of gzip(1). The degree of com‐
pression is dynamically set to minimise transfer time. It will
drop to no compression for fast ethernet connections, and will
go to maximum compression for slow modem connections. The algo‐
rithm adjusts the compression level so that the time for a TCP
write call is between 2% and 5% of the time for the same amount
of data to be deflated (i.e. compressed).
--gzip-backups
Backups are usually just a copy of the file. With this option
they are compressed and a .gz is added to the default extension.
Note that if you specify your own extension using --backup-
extension, then it must have a .gz at the end for compression to
work.
--case-insensitive, --for-Robert-Seese
Ignores case in comparison of filenames and linknames. This
option is useful when interacting with certain brain-dead oper‐
ating systems. I am not sure if this option behaves correctly
under all circumstances.
--to-lower
--to-upper
Convert all new filenames to upper or lower case. When used with
--case-insensitive, it will apply only when creating new files.
When used without --case-insensitive, all files, existing or
not, will be converted to uppercase in the inefficient way of
deleting the old file and then copying the new file. This option
is useful when interacting with certain brain-dead operating
systems. I am not sure if this options behave correctly under
all circumstances.
--no-use-passive-connections
If you get an error message could not setup passive mode it is
likely because you need to enable this option. I don't really
understand what `passive' means, so don't ask me.
--tar-file filename
This is to be used only with recursdir. It creates a tar archive
in the same format as GNU tar(1) and stores it in filename.
Leading special prefixes and leading slashes are removed, i.e.
ftp://machine/dir/file becomes dir/file. If the first character
of the filename is a | character, the remaining text is consid‐
ered as a command through which the output is to be piped. Hence
a gzipped archive can be created for example with
recursdir ftp://machine/dir --tar-file '| gzip -d > foo.tar.gz'
--tar-block-size N
Sets the block size for tar output to 512 * N. These are the
units in which data is written to the archive. The default is
20. This is significant only when writing to block devices. This
must not be confused with --block-size.
--block-size bytes
The default block size is 1024 bytes. File sizes are rounded up
to the nearest block when calculating the total number of blocks
consumed. If the actual block size is greater than assumed then
it is possible for fewer blocks to be counted than are actually
consumed during writing. Hence it is essential to specify a
block size greater than or equal to the actual block size when
using the option --max-bytes.
--strict-mtimes
When copying regular files, mirrordir normally only overwrites
the mirror file if it is older than the control file. This
option forces a copy if there is any discrepancy at all in the
modified times of the files.
--no-mtimes
Copy files only if their sizes differ. Ignore the file modifica‐
tion time.
--ignore-size
Copy files depending on mtime, but ignore size differences.
--starting-file path
path can be a file or directory. Until path is read, files or
directories will be processed as though they were excluded files
(i.e. if they exist in mirror they will be deleted). Directories
that contain path will be created if they don't exist. After
path is read, files are mirrored as usual. path itself will also
be mirrored. If path or any of its subdirectories are not
present, then mirrordir exits immediately. This is the only
time mirrordir exits prematurely. This is to prevent the entire
filesystem from being erased on account of path never being
found.
FTP SUPPORT
Ftp transfers are supported using the Midnight Commander's Virtual File
System (VFS), see mc(1). In short, this means that full URLs as well as
local directories are supported. The following example demonstrates:
mirrordir --verbose \
ftp://lava.obsidian.co.za/pub/mirrordir \
/home/mirrordir
or alternatively,
mirrordir --verbose /home/mirrordir \
ftp://psheer@lava.obsidian.co.za/home/ftp/pub/mirrordir
also works, but will prompt me for a password first. If you are upload‐
ing to an ftp server, you should not have the --strict-mtimes options
on, because modification times cannot be set via ftp, and hence every‐
thing will be copied.
You will notice (with --verbose) that mirrordir repeatedly tries to set
modification and access times over ftp. I am going to leave these mes‐
sages in to remind users of the limitations of the VFS type in use.
These attempts do not appreciably detract from performance. You can use
mc:// instead although performance is poor when uploading with this
protocol. Downloading is always preferable to uploading.
In general you should NOT have an ftp upload in your cron jobs or use
ftp upload to keep directories in sync. Always download from the other
end when trying to keep directories in sync. Uploading is good for once
off uploads only.
EXAMPLES
Here are some nifty things you can do with mirrordir.
Pedantic minimalist copy
I have two source trees. I like to keep an old copy before I
apply a patch. I just do mkdir tree.OLD and then
mirrordir -v tree tree.OLD
If I run mirrordir again, then only a minimal change is made,
i.e. only the updated files are copied. (cp(1) can in fact do
this).
System backups
Some systems used timed backups onto tape archives. Others still
use RAID devices which constantly maintain an identical copy of
a partition. mirrordir provides a further alternative. You can
install two drives in a system - one for use and one for backups
- and place mirrordir in your cron(8) tables. The various backup
options can be set to make backups of files that have changed.
The backup directory might also be readable by users so that
they can see their backed up files. Previous versions of files
would also be available for users that might like to retrieve
any older instances of the file. Because mirrordir executes
only the absolutely minimal set of changes, it is extremely
fast. It can be run several times a day or even continuously
with the --nice option.
Backups can also be made by FTP onto a remote machine for addi‐
tional security against the destruction of the machine.
Backing up hourly between two machines:
On one machine called dar2 I have a cron job that runs every six
hours:
#!/bin/sh
# (this is just in case of any bugs I don't know about,
# but I don't think it is necessary)
killall -9 tee
killall mirrordir >& /dev/null
sleep 2
killall -9 mirrordir >& /dev/null
( \
date ; \
echo "mirrordir says (if it said nothing it is bad):" ; \
mirrordir mc://dar1:12346/ -p abcdefg /mnt/dar1/ \
-i --exclude-regexp '^mc://dar1:12346/var/lock/subsys/atd' \
--exclude-regexp '^mc://dar1:12346/proc/' \
--exclude-regexp '^mc://dar1:12346/mnt/[^/]*/.*$' \
-i --exclude-regexp '^mc://dar1:12346/boot/' \
-i --exclude-regexp '^mc://dar1:12346/etc/lilo.conf' \
-C \
´
if (S_ISDIR (stat.st_mode)) {
if (!regexp ("^mc://dar1:12346/[^/]*$", PATH))
printf ("Backing up: %s0, PATH);
}
´ ; \
date ; \
echo "Done" ; \
) 2>&1 \
| tee --ignore-interrupts --append /var/log/mirrordir.log \
| mail -s 'dar1 backup results' psheer@obsidian.co.za
Secure transfers and logins
On the machine turing.co.uk execute
secure-mcserv -p 12345 -d
On some foreign machine
copydir--secure -K 512 -z \
mc://alan@turing.co.uk:12345/usr/src/linux/.config .
to securely copy a file using a security key of 512 bits and
using compression, or
pslogin mc://alan@turing.co.uk:12345/
to login to the machine securely.
Mirroring FTP sites
Ftp sites often disable their -R option on ls so that a tradi‐
tional mirror(1) will fail (?). mirrordir does not suffer from
this limitation.
mirrordir -v ftp://metalab.unc.edu/pub /home/ftp/pub
FTP transfers
Typical FTP transfers can be made easily with a single command
using the --copy-mode option. You can copy multiple files in
both directions and even between two ftp sites (albeit indi‐
rectly), just like with cp(1). I use
copydir-v mirrordir-0.9.15.tar.gz \
mirrordir.lsm ftp://metalab.unc.edu/incoming/Linux
to upload mirrordir to the sunsite.
Passwords for non-anonymous ftp transfers are best placed in the
~/.netrc file according to standard ftp convention, and the
option --netrc used. Alternatively use ftp://myname@machine/
instead.
Finding files
recursdir / -C
'if (!glob ("*.c", FILE)) printf ("%s\n", PATH);'
will print out all the C files on your system.
recursdir / -C
'if (S_ISCHR(stat.st_mode)) printf ("%s\n", PATH);'
will print out all character devices on your system.
Backup FTP sites onto tape
Use
recursdir ftp://user@remote.machine/ \
--exclude-regexp '//[^/]*/proc/' --tar-file /dev/mt
to back-up a remote site onto tape.
Removing core files
This removes all core files from your system:
recursdir / -C ´
long l;
if (strncmp (PATH, "/proc", 5)) {
if (S_ISREG (stat.st_mode) && !strcmp ("core", FILE)) {
if (strstr (popen ("file " + PATH), "ELF 32-bit LSB core")) {
l = l + stat.st_size;
printf ("removing: %s, cumu. total = %ldkB\n", PATH, l >> 10);
exec ("rm", "-f", PATH); /* could also use system() */
}
}
}
´
ENVIRONMENT VARIABLES
TMPDIR The directory where you would like temporary files to be stored.
The ftp filesystem downloads files first into the this temporary
directory, and then copies the file into its correct place. See
BUGS below.
If TMPDIR is not specified, then it defaults to the directory of
the current file in progress.
RETURN VALUE
mirrordir returns the following:
0 Success.
1 Some kind of error occurred like a write error, a permissions
error or the like. In this case, the precise error would have
been written to stderr.
2 A file was in use and therefore could not be copied, but other‐
wise a success. In this case the error `unable to open control
file for writing' would have been written to stderr. If you grep
for these in stderr, you can then run mirrordir with just these
files. See grep(1).
BUGS
Some versions of the atd daemon create a lock (or pid?) file with a
lock on the file that cause secure-mcserv to block indefinately. You
will have to kill atd or exclude (?) this file before hand to prevent
this.
Mirroring over ftp where there is some daylight savings correction,
seems to produce a one hour time offset. Use --time-offset as a tempo‐
rary work-around. I don't know if this is mirrordir's fault though.
A bug that caused mirrordir to eat CPU and appear to halt has been
fixed.
A large number of command-line --exclude expressions will slow perfor‐
mance. This is noted for the purposes of those wanting to exclude a
large list of files. This list should be placed in a text file and
excluded with the --exclude-from option.
Modification and access times of symlinks (not the files they point to)
are not duplicated.
Note that the previous limitation that hardlinked files are treated as
regular files has now been alleviated. The option --no-hard-links is
provided to emulate the behaviour of versions prior to 0.9.8.
No check is made to see if a hard-link would be created across devices,
in which case an appropriate error will be reported.
The ftp file-system downloads files first into the temporary directory,
as a file with the name *ftpfs*. This is a waste of space but is the
default behavior of the vfs library. If there is not enough space in
this directory, then mirrordir will probably hang. See ENVIRONMENT
VARIABLES above.
The C scripting language used to exclude files is an excessive imple‐
mentation.
FILES
~/.netrc
List of machines and corresponding passwords. See the option
--netrc.
/etc/ssocket/accept.cs
This script performs the key exchange and signature generation
on the server side of the connection.
/etc/ssocket/connect.cs
This script performs the key exchange and signature authentica‐
tion on the client side of the connection.
/etc/ssocket/arcinit.cs
This initialises the stream cipher encryption. (Not present in
International versions.)
/etc/ssocket/arcencrypt.cs
This performs the stream cipher encryption. (Not present in
International versions.)
/etc/ssocket/private/
This directory contains private keys for the host. Each key will
be held in a file: filenames are 512, 1024 etc. This is a
preferable alternative to databases of keys as it obviates the
need for a key management utility. Soon the Reiser (spelling)
filesystem will be standard and database files will become obso‐
lete anyway.
/etc/ssocket/public/
Analogous to /etc/ssocket/private/ for public keys.
STANDARDS
mirrordir is an invention of its author and does not belong to any
operating system standard (although it should!).
AVAILABILITY
The latest version of the program can be found at either ftp://meta‐
lab.unc.edu/pub/Linux/system/backup, or ftp://lava.obsid‐
ian.co.za/pub/linux/mirrordir.
AUTHOR
Paul Sheer <psheer@obsidian.co.za> <psheer@icon.co.za>
SEE ALSOmirror(1), pavuk(1), cp(1), scp(1), find(1), mc(1), ftp(1), ssh(1),
tar(1), rlogin(1), rlogind(8), forward
Linux 1998 November 1 mirrordir(1)