File: ange-ftp.el.html

This package attempts to make accessing files and directories using FTP from within GNU Emacs as simple and transparent as possible. A subset of the common file-handling routines are extended to interact with FTP.

Usage:

Some of the common GNU Emacs file-handling operations have been made FTP-smart. If one of these routines is given a filename that matches
'/user@host:name' then it will spawn an FTP process connecting to machine
'host' as account 'user' and perform its operation on the file 'name'.

For example: if find-file is given a filename of:

  /ange@anorman:/tmp/notes

then ange-ftp spawns an FTP process, connects to the host 'anorman' as user 'ange', gets the file '/tmp/notes' and pops up a buffer containing the contents of that file as if it were on the local filesystem. If ange-ftp needs a password to connect then it reads one in the echo area.

Extended filename syntax:

The default extended filename syntax is '/user@host:name', where the
'user@' part may be omitted. This syntax can be customized to a certain
extent by changing ange-ftp-name-format. There are limitations. The host part has an optional suffix #port which may be used to specify a non-default port number for the connection.

If the user part is omitted then ange-ftp generates a default user instead whose value depends on the variable ange-ftp-default-user.

Passwords:

A password is required for each host/user pair. Ange-ftp reads passwords as needed. You can also specify a password with ange-ftp-set-passwd, or in a *valid* ~/.netrc file.

Passwords for user "anonymous":

Passwords for the user "anonymous" (or "ftp") are handled specially. The variable ange-ftp-generate-anonymous-password controls what happens: if the value of this variable is a string, then this is used as the password; if non-nil (the default), then the value of user-mail-address is used; if nil then the user is prompted for a password as normal.

"Dumb" UNIX hosts:

The FTP servers on some UNIX machines have problems if the 'ls' command is used.

The routine ange-ftp-add-dumb-unix-host can be called to tell ange-ftp to limit itself to the DIR command and not 'ls' for a given UNIX host. Note that this change will take effect for the current GNU Emacs session only. See below for a discussion of non-UNIX hosts. If a large number of machines with similar hostnames have this problem then it is easier to set the value of ange-ftp-dumb-unix-host-regexp in your init file. ange-ftp is unable to automatically recognize dumb unix hosts.

File name completion:

Full file-name completion is supported on UNIX, VMS, CMS, and MTS hosts. To do filename completion, ange-ftp needs a listing from the remote host. Therefore, for very slow connections, it might not save any time.

FTP processes:

When ange-ftp starts up an FTP process, it leaves it running for speed purposes. Some FTP servers will close the connection after a period of time, but ange-ftp should be able to quietly reconnect the next time that the process is needed.

Killing the "*ftp user@host*" buffer also kills the ftp process. This should not cause ange-ftp any grief.

Binary file transfers:

By default ange-ftp transfers files in ASCII mode. If a file being transferred matches the value of ange-ftp-binary-file-name-regexp then binary mode is used for that transfer.

Account passwords:

Some FTP servers require an additional password which is sent by the ACCOUNT command. ange-ftp partially supports this by allowing the user to specify an account password by either calling ange-ftp-set-account, or by specifying an account token in the .netrc file. If the account password is set by either of these methods then ange-ftp will issue an ACCOUNT command upon starting the FTP process.

Preloading:

ange-ftp can be preloaded, but must be put in the site-init.el file and not the site-load.el file in order for the documentation strings for the functions being overloaded to be available.

Status reports:

Most ange-ftp commands that talk to the FTP process output a status message on what they are doing. In addition, ange-ftp can take advantage of the FTP client's HASH command to display the status of transferring files and listing directories. See the documentation for the variables ange-ftp-{ascii,binary}-hash-mark-size, ange-ftp-send-hash and ange-ftp-process-verbose for more details.

Gateways:

Sometimes it is necessary for the FTP process to be run on a different machine than the machine running GNU Emacs. This can happen when the local machine has restrictions on what hosts it can access.

ange-ftp has support for running the ftp process on a different (gateway) machine. The way it works is as follows:

1) Set the variable 'ange-ftp-gateway-host' to the name of a machine
   that doesn't have the access restrictions.

2) Set the variable 'ange-ftp-local-host-regexp' to a regular expression
   that matches hosts that can be contacted from running a local ftp
   process, but fails to match hosts that can't be accessed locally. For
   example:

   "\\\\.hp\\\\.com$\\\\|^[^.]*$"

   will match all hosts that are in the .hp.com domain, or don't have an
   explicit domain in their name, but will fail to match hosts with
   explicit domains or that are specified by their ip address.

3) Using NFS and symlinks, make sure that there is a shared directory with
   the *same* name between the local machine and the gateway machine.
   This directory is necessary for temporary files created by ange-ftp.

4) Set the variable 'ange-ftp-gateway-tmp-name-template' to the name of
   this directory plus an identifying filename prefix. For example:

   "/nfs/hplose/ange/ange-ftp"

   where /nfs/hplose/ange is a directory that is shared between the
   gateway machine and the local machine.

The simplest way of getting a ftp process running on the gateway machine is if you can spawn a remote shell using either 'rsh' or 'remsh'. If you can't do this for some reason such as security then points 7 onwards will discuss an alternative approach.

5) Set the variable ange-ftp-gateway-program to the name of the remote
   shell process such as 'remsh' or 'rsh' if the default isn't correct.

6) Set the variable ange-ftp-gateway-program-interactive to nil if it
   isn't already. This tells ange-ftp that you are using a remote shell
   rather than logging in using telnet or rlogin.

That should be all you need to allow ange-ftp to spawn a ftp process on the gateway machine. If you have to use telnet or rlogin to get to the gateway machine then follow the instructions below.

7) Set the variable ange-ftp-gateway-program to the name of the program
   that lets you log onto the gateway machine. This may be something like
   telnet or rlogin.

8) Set the variable ange-ftp-gateway-prompt-pattern to a regular
   expression that matches the prompt you get when you login to the
   gateway machine. Be very specific here; this regexp must not match
   *anything* in your login banner except this prompt.
   shell-prompt-pattern is far too general as it appears to match some
   login banners from Sun machines. For example:

   "^$*$ *"

9) Set the variable ange-ftp-gateway-program-interactive to t to let
   ange-ftp know that it has to "hand-hold" the login to the gateway
   machine.

10) Set the variable ange-ftp-gateway-setup-term-command to a UNIX command
    that will put the pty connected to the gateway machine into a
    no-echoing mode, and will strip off carriage-returns from output from
    the gateway machine. For example:

    "stty -onlcr -echo"

    will work on HP-UX machines, whereas:

    "stty -echo nl"

    appears to work for some Sun machines.

That's all there is to it.

Smart gateways:

If you have a "smart" ftp program that allows you to issue commands like
"USER foo@bar" which do nice proxy things, then look at the variables
ange-ftp-smart-gateway and ange-ftp-smart-gateway-port.

Otherwise, if there is an alternate ftp program that implements proxy in a transparent way (i.e. without specifying the proxy host), that will connect you directly to the desired destination host: Set ange-ftp-gateway-ftp-program-name to that program's name. Set ange-ftp-local-host-regexp to a value as stated earlier on. Leave ange-ftp-gateway-host set to nil. Set ange-ftp-smart-gateway to t.

Tips for using ange-ftp:

1. For dired to work on a host which marks symlinks with a trailing @ in
   an ls -alF listing, you need to (setq dired-ls-F-marks-symlinks t).
   Most UNIX systems do not do this, but ULTRIX does. If you think that
   there is a chance you might connect to an ULTRIX machine (such as
   prep.ai.mit.edu), then set this variable accordingly. This will have
   the side effect that dired will have problems with symlinks whose names
   end in an @. If you get yourself into this situation then editing
   dired's ls-switches to remove "F", will temporarily fix things.

2. If you know that you are connecting to a certain non-UNIX machine
   frequently, and ange-ftp seems to be unable to guess its host-type,
   then setting the appropriate host-type regexp
   (ange-ftp-vms-host-regexp, ange-ftp-mts-host-regexp, or
   ange-ftp-cms-host-regexp) accordingly should help. Also, please report
   ange-ftp's inability to recognize the host-type as a bug.

3. For slow connections, you might get "listing unreadable" error
   messages, or get an empty buffer for a file that you know has something
   in it. The solution is to increase the value of ange-ftp-retry-time.
   Its default value is 5 which is plenty for reasonable connections.
   However, for some transatlantic connections I set this to 20.

4. Beware of compressing files on non-UNIX hosts. Ange-ftp will do it by
   copying the file to the local machine, compressing it there, and then
   sending it back. Binary file transfers between machines of different
   architectures can be a risky business. Test things out first on some
   test files. See "Bugs" below. Also, note that ange-ftp copies files by
   moving them through the local machine. Again, be careful when doing
   this with binary files on non-Unix machines.

5. Beware that dired over ftp will use your setting of dired-no-confirm
   (list of dired commands for which confirmation is not asked). You
   might want to reconsider your setting of this variable, because you
   might want confirmation for more commands on remote direds than on
   local direds. For example, I strongly recommend that you not include
   compress and uncompress in this list. If there is enough demand it
   might be a good idea to have an alist ange-ftp-dired-no-confirm of
   pairs ( TYPE . LIST ), where TYPE is an operating system type and LIST
   is a list of commands for which confirmation would be suppressed. Then
   remote dired listings would take their (buffer-local) value of
   dired-no-confirm from this alist. Who votes for this?

---------------------------------------------------------------------
Non-UNIX support:
---------------------------------------------------------------------

VMS support:

Ange-ftp has full support for VMS hosts. It should be able to automatically recognize any VMS machine. However, if it fails to do this, you can use the command ange-ftp-add-vms-host. Also, you can set the variable ange-ftp-vms-host-regexp in your init file. We would be grateful if you would report any failures to automatically recognize a VMS host as a bug.

Filename Syntax:

For ease of *implementation*, the user enters the VMS filename syntax in a UNIX-y way. For example:
 PUB$:[ANONYMOUS.SDSCPUB.NEXT]README.TXT;1
would be entered as:
 /PUB$$:/ANONYMOUS/SDSCPUB/NEXT/README.TXT;1
i.e. to log in as anonymous on ymir.claremont.edu and grab the file:
 [.CSV.POLICY]RULES.MEM
you would type:
 C-x C-f /anonymous@ymir.claremont.edu:CSV/POLICY/RULES.MEM

A valid VMS filename is of the form: FILE.TYPE;## where FILE can be up to 39 characters
      TYPE can be up to 39 characters
      ## is a version number (an integer between 1 and 32,767)
Valid characters in FILE and TYPE are A-Z 0-9 _ - $
$ cannot begin a filename, and - cannot be used as the first or last
character.

Tips:
1. Although VMS is not case sensitive, EMACS running under UNIX is.
   Therefore, to access a VMS file, you must enter the filename with upper
   case letters.
2. To access the latest version of file under VMS, you use the filename
   without the ";" and version number. You should always edit the latest
   version of a file. If you want to edit an earlier version, copy it to a
   new file first. This has nothing to do with ange-ftp, but is simply
   good VMS operating practice. Therefore, to edit FILE.TXT;3 (say 3 is
   latest version), do C-x C-f /ymir.claremont.edu:FILE.TXT. If you
   inadvertently do C-x C-f /ymir.claremont.edu:FILE.TXT;3, you will find
   that VMS will not allow you to save the file because it will refuse to
   overwrite FILE.TXT;3, but instead will want to create FILE.TXT;4, and
   attach the buffer to this file. To get out of this situation, M-x
   write-file /ymir.claremont.edu:FILE.TXT will attach the buffer to
   latest version of the file. For this reason, in dired "f"
   (dired-find-file), always loads the file sans version, whereas "v",
   (dired-view-file), always loads the explicit version number. The
   reasoning being that it reasonable to view old versions of a file, but
   not to edit them.
3. EMACS has a feature in which it does environment variable substitution
   in filenames. Therefore, to enter a $ in a filename, you must quote it
   by typing $$.

MTS support:

Ange-ftp has full support for hosts running the Michigan terminal system. It should be able to automatically recognize any MTS machine. However, if it fails to do this, you can use the command ange-ftp-add-mts-host. As well, you can set the variable ange-ftp-mts-host-regexp in your init file. We would be grateful if you would report any failures to automatically recognize a MTS host as a bug.

Filename syntax:

MTS filenames are entered in a UNIX-y way. For example, if your account was YYYY, the file FILE in the account XXXX: on mtsg.ubc.ca would be entered as
  /YYYY@mtsg.ubc.ca:/XXXX:/FILE
In other words, MTS accounts are treated as UNIX directories. Of course, to access a file in another account, you must have access permission for it. If FILE were in your own account, then you could enter it in a relative name fashion as
  /YYYY@mtsg.ubc.ca:FILE
MTS filenames can be up to 12 characters. Like UNIX, the structure of the filename does not contain a TYPE (i.e. it can have as many "."'s as you like.) MTS filenames are always in upper case, and hence be sure to enter them as such! MTS is not case sensitive, but an EMACS running under UNIX is.

CMS support:

Ange-ftp has full support for hosts running CMS. It should be able to automatically recognize any CMS machine. However, if it fails to do this, you can use the command ange-ftp-add-cms-host. As well, you can set the variable ange-ftp-cms-host-regexp in your init file. We would be grateful if you would report any failures to automatically recognize a CMS host as a bug.

Filename syntax:

CMS filenames are entered in a UNIX-y way. In other words, minidisks are treated as UNIX directories. For example to access the file READ.ME in minidisk *.311 on cuvmb.cc.columbia.edu, you would enter
  /anonymous@cuvmb.cc.columbia.edu:/*.311/READ.ME
If *.301 is the default minidisk for this account, you could access FOO.BAR on this minidisk as
  /anonymous@cuvmb.cc.columbia.edu:FOO.BAR
CMS filenames are of the form FILE.TYPE, where both FILE and TYPE can be up to 8 characters. Again, beware that CMS filenames are always upper case, and hence must be entered as such.

Tips:
1. CMS machines, with the exception of anonymous accounts, nearly always
   need an account password. To have ange-ftp send an account password,
   you can either include it in your .netrc file, or use
   ange-ftp-set-account.
2. Ange-ftp cannot send "write passwords" for a minidisk. Hopefully, we
   can fix this.

BS2000 support:

Ange-ftp has full support for BS2000 hosts. It should be able to automatically recognize any BS2000 machine. However, if it fails to do this, you can use the command ange-ftp-add-bs2000-host. As well, you can set the variable ange-ftp-bs2000-host-regexp in your .emacs file. We would be grateful if you would report any failures to auto- matically recognize a BS2000 host as a bug.

If you want to access the POSIX subsystem on BS2000 you MUST use command ange-ftp-add-bs2000-posix-host for that particular hostname. ange-ftp can't decide if you want to access the native filesystem or the POSIX filesystem, so it accesses the native filesystem by default. And if you have an ASCII filesystem in your BS2000 POSIX subsystem you must use ange-ftp-binary-file-name-regexp to access its files.

Filename Syntax:

For ease of *implementation*, the user enters the BS2000 filename syntax in a UNIX-y way. For example:
 :PUB:$PUBLIC.ANONYMOUS.SDSCPUB.NEXT.README.TXT
would be entered as:
 /:PUB:/$$PUBLIC/ANONYMOUS.SDSCPUB.NEXT.README.TXT
You don't have to type pubset and account, if they have default values, i.e. to log in as anonymous on bs2000.anywhere.com and grab the file IMPORTANT.TEXT.ON.BS2000 on the default pubset X on userid PUBLIC
(there are only 8 characters in a valid username), you could type:
 C-x C-f /public@bs2000.anywhere.com:/IMPORTANT.TEXT.ON.BS2000
or
 C-x C-f /anonym@bs2000.anywhere.com:/:X:/$$PUBLIC/IMPORTANT.TEXT.ON.BS2000

If X is not your default pubset, you could add it as 'subdirectory' (BS2000 has a flat architecture) with the command
(setq ange-ftp-bs2000-additional-pubsets '(":X:"))
and then you could type:
 C-x C-f /anonym@bs2000.anywhere.com:/:X:/IMPORTANT.TEXT.ON.BS2000

Valid characters in an BS2000 filename are A-Z 0-9 $ # @ . - If the first character in a filename is # or @, this is replaced with ange-ftp-bs2000-special-prefix because names starting with # or @ are reserved for temporary files. This is especially important for auto-save files. Valid file generations are ending with ([+|-|*]0-9...) . File generations are not supported yet! A filename must at least contain one character (A-Z) and cannot be longer than 41 characters.

Tips:
1. Although BS2000 is not case sensitive, EMACS running under UNIX is.
   Therefore, to access a BS2000 file, you must enter the filename with
   upper case letters.
2. EMACS has a feature in which it does environment variable substitution
   in filenames. Therefore, to enter a $ in a filename, you must quote it
   by typing $$.
3. BS2000 machines, with the exception of anonymous accounts, nearly
   always need an account password. To have ange-ftp send an account
   password, you can either include it in your .netrc file, or use
   ange-ftp-set-account.

------------------------------------------------------------------
Bugs:
------------------------------------------------------------------

1. Umask problems:
   Be warned that files created by using ange-ftp will take account of the
   umask of the ftp daemon process rather than the umask of the creating
   user. This is particularly important when logging in as the root user.
   The way that I tighten up the ftp daemon's umask under HP-UX is to make
   sure that the umask is changed to 027 before I spawn /etc/inetd. I
   suspect that there is something similar on other systems.

2. Some combinations of FTP clients and servers break and get out of sync
   when asked to list a non-existent directory. Some of the ai.mit.edu
   machines cause this problem for some FTP clients. Using
   ange-ftp-kill-ftp-process can restart the ftp process, which
   should get things back in sync.

3. Ange-ftp does not check to make sure that when creating a new file,
   you provide a valid filename for the remote operating system.
   If you do not, then the remote FTP server will most likely
   translate your filename in some way. This may cause ange-ftp to
   get confused about what exactly is the name of the file. The
   most common causes of this are using lower case filenames on systems
   which support only upper case, and using filenames which are too
   long.

4. Null (blank) passwords confuse both ange-ftp and some FTP daemons.

5. Ange-ftp likes to use pty's to talk to its FTP processes. If GNU Emacs
   for some reason creates a FTP process that only talks via pipes then
   ange-ftp won't be getting the information it requires at the time that
   it wants it since pipes flush at different times to pty's. One
   disgusting way around this problem is to talk to the FTP process via
   rlogin which does the 'right' things with pty's.

6. For CMS support, we send too many cd's. Since cd's are cheap, I haven't
   worried about this too much. Eventually, we should have some caching
   of the current minidisk.

7. Some CMS machines do not assign a default minidisk when you ftp them as
   anonymous. It is then necessary to guess a valid minidisk name, and cd
   to it. This is (understandably) beyond ange-ftp.

8. Remote to remote copying of files on non-Unix machines can be risky.
   Depending on the variable ange-ftp-binary-file-name-regexp, ange-ftp
   will use binary mode for the copy. Between systems of different
   architecture, this still may not be enough to guarantee the integrity
   of binary files. Binary file transfers from VMS machines are
   particularly problematical. Should ange-ftp-binary-file-name-regexp be
   an alist of OS type, regexp pairs?

9. The code to do compression of files over ftp is not as careful as it
   should be. It deletes the old remote version of the file, before
   actually checking if the local to remote transfer of the compressed
   file succeeds. Of course to delete the original version of the file
   after transferring the compressed version back is also dangerous,
   because some OS's have severe restrictions on the length of filenames,
   and when the compressed version is copied back the "-Z" or ".Z" may be
   truncated. Then, ange-ftp would delete the only remaining version of
   the file. Maybe ange-ftp should make backups when it compresses files
   (of course, the backup "~" could also be truncated off, sigh...).
   Suggestions?

10. If a dir listing is attempted for an empty directory on (at least
    some) VMS hosts, an ftp error is given. This is really an ftp bug, and
    I don't know how to get ange-ftp work to around it.

11. Bombs on filenames that start with a space. Deals well with filenames
    containing spaces, but beware that the remote ftpd may not like them
    much.

12. The dired support for non-Unix-like systems does not currently work.
    It needs to be reimplemented by modifying the parse-...-listing
functions to convert the directory listing to ls -l format.

13. The famous @ bug. As mentioned above in TIPS, ULTRIX marks symlinks
    with a trailing @ in a ls -alF listing. In order to account for this
    ange-ftp looks to chop trailing @'s off of symlink names when it is
    parsing a listing with the F switch. This will cause ange-ftp to
    incorrectly get the name of a symlink on a non-ULTRIX host if its name
    ends in an @. ange-ftp will correct itself if you take F out of the
    dired ls switches (C-u s will allow you to edit the switches). The
    dired buffer will be automatically reverted, which will allow ange-ftp
    to fix its files hashtable. A cookie to anyone who can think of a
    fast, sure-fire way to recognize ULTRIX over ftp.

If you find any bugs or problems with this package, PLEASE report a bug to the Emacs maintainers via M-x report-emacs-bug.

-----------------------------------------------------------
Technical information on this package:
-----------------------------------------------------------

ange-ftp works by putting a handler on file-name-handler-alist which is called by many primitives, and a few non-primitives, whenever they see a file name of the appropriate sort.

Checklist for adding non-UNIX support for TYPE

The following functions may need TYPE versions:
(not all functions will be needed for every OS)

ange-ftp-fix-name-for-TYPE ange-ftp-fix-dir-name-for-TYPE ange-ftp-TYPE-host ange-ftp-TYPE-add-host ange-ftp-parse-TYPE-listing ange-ftp-TYPE-delete-file-entry ange-ftp-TYPE-add-file-entry ange-ftp-TYPE-file-name-as-directory ange-ftp-TYPE-make-compressed-filename ange-ftp-TYPE-file-name-sans-versions

Variables:

ange-ftp-TYPE-host-regexp May need to add TYPE to ange-ftp-dumb-host-types

Check the following functions for OS dependent coding:

ange-ftp-host-type ange-ftp-guess-host-type ange-ftp-allow-child-lookup

Host type conventions:

The function ange-ftp-host-type and the variable ange-ftp-dired-host-type
(mostly) follow the following conventions for remote host types. At
least, I think that future code should try to follow these conventions, and the current code should eventually be made compliant.

nil = local host type, whatever that is (probably unix).
      Think nil as in "not a remote host". This value is used by
      ange-ftp-dired-host-type for local buffers.

t = a remote host of unknown type. Think t as in true, it's remote.
    Currently, unix is used as the default remote host type.
    Maybe we should use t.

TYPE = a remote host of TYPE type.

TYPE:LIST = a remote host of TYPE type, using a specialized ftp listing
            program called list. This is currently only used for Unix
            dl (descriptive listings), when ange-ftp-dired-host-type
            is set to unix:dl.

Bug report codes:

Because of their naive faith in this code, there are certain situations which the writers of this program believe could never happen. However, being realists they have put calls to error in the program at these points. These errors provide a code, which is an integer, greater than 1. To aid debugging. the error codes, and the functions in which they reside are listed below.

1: See ange-ftp-ls


-----------------------------------------------------------
Hall of fame:
-----------------------------------------------------------

Thanks to Roland McGrath for improving the filename syntax handling, for suggesting many enhancements and for numerous cleanups to the code.

Thanks to Jamie Zawinski for bugfixes and for ideas such as gateways.

Thanks to Ken Laprade for improved .netrc parsing, password reading, and dired / shell auto-loading.

Thanks to Sebastian Kremer for dired support and for many ideas and bugfixes.

Thanks to Joe Wells for bugfixes, the original non-UNIX system support, VOS support, and hostname completion.

Thanks to Nakagawa Takayuki for many good ideas, filename-completion, help with file-name expansion, efficiency worries, stylistic concerns and many bugfixes.

Thanks to Sandy Rutherford who re-wrote most of ange-ftp to support VMS, MTS, CMS and UNIX-dls. Sandy also added dired-support for non-UNIX OS and auto-recognition of the host type.

Thanks to Dave Smith who wrote the info file for ange-ftp.

Finally, thanks to Keith Waclena, Mark D. Baushke, Terence Kelleher, Ping Zhou, Edward Vielmetti, Jack Repenning, Mike Balenger, Todd Kaufmann, Kjetil Svarstad, Tom Wurgler, Linus Tolke, Niko Makila, Carl Edman, Bill Trost, Dave Brennan, Dan Jacobson, Andy Scott, Steve Anderson, Sanjay Mathur, the folks on the ange-ftp-lovers mailing list and many others whose names I've forgotten who have helped to debug and fix problems with ange-ftp.el.

Defined variables (82)

Defined functions (203)

Defined faces (0)