maspack.fileutil
Class FileGrabber

java.lang.Object
  extended by maspack.fileutil.FileGrabber
Direct Known Subclasses:
ArtisynthFileGrabber, ArtisynthWebGrabber

public class FileGrabber
extends java.lang.Object

Downloads files from URIs satisfying the generic/zip URI syntax according to specifications RFC 3986 and proposed jar/zip extension.

A typical use will look like this:

  FileGrabber grabber = new FileGrabber("data", "http://www.fileserver.com");
 grabber.get("localFile", "remoteFile");
 
By default, the FileGrabber will simply return the localFile if it exists. To force updates from a remote location, you can set options:
  int options = FileGrabber.CHECK_HASH;
 grabber.get("localFile", "remoteFile", options);
 
The CHECK_HASH flag downloads the remote file if its sha1 hash differs from that of your local copy. There is also a FORCE_REMOTE flag that forces the remote file to be downloaded always.

For fine control and catching errors, the workflow is slightly different:
  boolean download = true;
 File dest = new File("localFile");
 URI source = new URI("remoteFile");
 File local = grabber.getLocal(dest);
 
 if (local != null) {
    boolean match = false;
    try {
       match = grabber.equalsHash(dest, source);
    } catch (FileGrabberException e) {
       System.out.println(e.getMessage());
    }
    download = !match;
 }
 
 if (download) {
    try {
       local = grabber.getRemote(dest, source);
    } catch (FileGrabberException e) {
       System.out.println(e.getMessage());
    }
 } 
 
 if (local == null) {
    throw new RuntimeException("Unable to get file.");
 } 
 
See VFS2 File Systems for further details regarding supported remote filesystems and URI syntax.

Author:
"Antonio Sanchez" Creation date: 13 Nov 2012

Field Summary
static int CHECK_HASH
          Check file hashes, and if different, get from remote
static int DEFAULT_LOG_LEVEL
           
static Logger DEFAULT_LOGGER
           
static int DEFAULT_OPTIONS
           
static int DOWNLOAD_ZIP
          If file is in a remote zip file, get a local copy of the entire zip file first
static int FORCE_REMOTE
          Always read from remote if possible
 int myOptions
           
 
Constructor Summary
FileGrabber()
          Default constructor, sets the local directory to the current path, and sets the default URI to be empty.
FileGrabber(java.io.File downloadDir, URIx remoteSource)
          Sets default paths
FileGrabber(java.lang.String downloadPath)
          Sets default download directory, leaves source as null
FileGrabber(java.lang.String downloadPath, java.lang.String remoteSourceName)
          Sets local download path and remote URI source by parsing the supplied strings
 
Method Summary
 void addIdentityRepository(URIxMatcher matcher, SimpleIdentityRepository repo)
          Adds an identity repository consisting of a set of private RSA keys for use with SFTP authentication.
 void addTransferListener(FileTransferListener listener)
          Adds a FileTransferListener object that responds to transfer events.
 void addUserAuthenticator(URIxMatcher matcher, org.apache.commons.vfs2.UserAuthenticator auth)
          Adds an authenticator for HTTPS, SFTP, that can respond to domain/username/password requests.
 void closeStreams()
          Must be called to free resources after streams have been read.
 boolean equalsHash(java.io.File file, URIx uri)
          Compares sha1 hash values between a local file and remote URI
 boolean equalsHash(java.lang.String relPath)
           
 java.io.File get(java.io.File path, URIx source)
          Downloads a file using the default options
 java.io.File get(java.io.File dest, URIx source, int options)
          Returns a file handle to a local version of the requested file.
 java.io.File get(java.lang.String sourceName)
          Downloads a file using same relative path for source and destination and default options
 java.io.File get(java.lang.String sourceName, int options)
          Downloads a file using same relative path for source and destination
 java.io.File get(java.lang.String destName, java.lang.String sourceName)
          Downloads a file with default options
 java.io.File get(java.lang.String destName, java.lang.String sourceName, int options)
          Converts the supplied destination path and source URI to a File and URI object, respectively, and downloads the remote file according to the supplied options.
 java.io.File getAbsoluteFile(java.io.File relFile)
          Converts a relative file to an absolute one using the object's download directory.
 java.io.File getAbsoluteFile(java.lang.String relPath)
          Converts a relative file to an absolute one using the object's download directory.
 URIx getAbsoluteURI(java.lang.String relURIstr)
          Converts a relative URI to an absolute one, using the remoteSource as a base.
 URIx getAbsoluteURI(URIx relURI)
          Converts a relative URI to an absolute one, using the remote source as a base.
 boolean getConsoleProgressPrinting()
          Returns true if default printing of file transfer progress to the console is enabled.
 java.io.File getDownloadDir()
           
 java.io.InputStream getInputStream(java.io.File localCopy, URIx source, int options)
          Returns an input stream for the file located at either localCopy or source (according to options).
 java.io.InputStream getInputStream(java.lang.String sourceName)
           
 java.io.InputStream getInputStream(java.lang.String sourceName, int options)
           
 java.io.InputStream getInputStream(java.lang.String localCopy, java.lang.String sourceName)
           
 java.io.InputStream getInputStream(java.lang.String localCopy, java.lang.String sourceName, int options)
           
 java.io.InputStream getInputStream(URIx source)
           
 java.io.InputStream getInputStream(URIx source, int options)
           
 java.io.File getLastFile()
          Gets the last file retrieved.
 java.io.File getLocal(java.io.File file)
          Retrieves a local file if it exists, null otherwise.
 java.io.File getLocal(java.lang.String fileName)
          Retrieves a local file if it exists, null otherwise.
 java.lang.String getLocalHash(java.io.File file)
          Gets the sha1 hash of a local file
 java.lang.String getLocalHash(java.lang.String fileName)
          Gets the sha1 hash of a local file
static Logger getLogger()
          Gets the logger for printing messages
 int getOptions()
           
 java.io.File getRemote(java.io.File dest, URIx source)
          Retrieves a remote file if it exists, null otherwise.
 java.io.File getRemote(java.lang.String sourceName)
          getRemote(String, String) with dest=null
 java.io.File getRemote(java.lang.String destName, java.lang.String sourceName)
          Retrieves a remote file if it exists, null otherwise.
 java.io.File getRemote(URIx source)
          getRemote(File, URIx) with dest=null
 java.lang.String getRemoteHash(java.lang.String uriStr)
          Fetches a hash file from a remote location.
 java.lang.String getRemoteHash(URIx uri)
          Fetches a hash file from a remote location.
 URIx getRemoteSource()
           
static FileGrabber getStaticGrabber()
          Returns a FileGrabber object used for staticGet(File, URIx).
 FileTransferListener[] getTransferListeners()
          Gets the set of listeners for all transfers handled by this FileGrabber object
 FileTransferMonitor getTransferMonitor()
          Returns the FileTransferMonitor object that is is responsible for detecting changes to the destination file and firing FileTransferEvents to the set of FileTransferListeners.
 void removeTransferListener(FileTransferListener listener)
          Removes a listener that is listener to transfer events.
 void setConsoleProgressPrinting(boolean enable)
          Enables or disables default printing of file transfer progress to the console.
 void setDownloadDir(java.io.File dir)
          Sets directory where files are downloaded
 void setDownloadDir(java.lang.String path)
          Sets directory where files are downloaded
static void setLogger(Logger log)
          Sets the logger for printing messages, defaults to printing to console.
 void setMonitorSleep(double seconds)
          Sets the sleep time between checking for transfer updates.
 void setOptions(int options)
           
 void setRemoteSource(java.lang.String uriStr)
          Sets the base URI for remote files, this is attached to any relative URIs provided in the get(...) methods
 void setRemoteSource(URIx uri)
          Sets the base URI for remote files, this is attached to any relative URIs provided in the get(...) methods
static void setStaticGrabber(FileGrabber grabber)
          Sets the FileGrabber used for staticGet(File, URIx).
 void setTransferMonitor(FileTransferMonitor monitor)
          Sets the FileTransferMonitor which is responsible for detecting when the destination file has changed in order to fire update events.
 void setVerbosityLevel(int level)
          Sets the verbosity level of this FileGrabber.
 void setVerbosityLevel(Logger.LogLevel level)
          Sets the verbosity level of this FileGrabber.
static java.io.File staticGet(java.io.File dest, URIx source)
          Static convenience method for downloading a file.
static java.io.File staticGet(java.io.File dest, URIx source, int options)
          Static convenience method for downloading a file in a single shot.
static java.io.File staticGet(java.lang.String destName, java.lang.String sourceName)
          Static convenience method for downloading a file with default options.
static java.io.File staticGet(java.lang.String destName, java.lang.String sourceName, int options)
          Static convenience method for downloading a file.
 boolean wasLastRemote()
          Returns true if the last file was fetched from remote, false if last file was a local copy
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

FORCE_REMOTE

public static final int FORCE_REMOTE
Always read from remote if possible

See Also:
Constant Field Values

CHECK_HASH

public static final int CHECK_HASH
Check file hashes, and if different, get from remote

See Also:
Constant Field Values

DOWNLOAD_ZIP

public static final int DOWNLOAD_ZIP
If file is in a remote zip file, get a local copy of the entire zip file first

See Also:
Constant Field Values

DEFAULT_OPTIONS

public static int DEFAULT_OPTIONS

DEFAULT_LOG_LEVEL

public static int DEFAULT_LOG_LEVEL

DEFAULT_LOGGER

public static Logger DEFAULT_LOGGER

myOptions

public int myOptions
Constructor Detail

FileGrabber

public FileGrabber()
Default constructor, sets the local directory to the current path, and sets the default URI to be empty.


FileGrabber

public FileGrabber(java.io.File downloadDir,
                   URIx remoteSource)
Sets default paths

Parameters:
downloadDir - the local path to save files to
remoteSource - the remote base URI to download files from

FileGrabber

public FileGrabber(java.lang.String downloadPath)
Sets default download directory, leaves source as null

Parameters:
downloadPath - the local path to save files to

FileGrabber

public FileGrabber(java.lang.String downloadPath,
                   java.lang.String remoteSourceName)
Sets local download path and remote URI source by parsing the supplied strings

Parameters:
downloadPath - the local path to save files to
remoteSourceName - the remote base URI
Method Detail

setDownloadDir

public void setDownloadDir(java.io.File dir)
Sets directory where files are downloaded

Parameters:
dir - default download directory

setDownloadDir

public void setDownloadDir(java.lang.String path)
Sets directory where files are downloaded

Parameters:
path - default download directory

getDownloadDir

public java.io.File getDownloadDir()
Returns:
the default download directory

setRemoteSource

public void setRemoteSource(URIx uri)
Sets the base URI for remote files, this is attached to any relative URIs provided in the get(...) methods

Parameters:
uri -

setRemoteSource

public void setRemoteSource(java.lang.String uriStr)
                     throws URIxSyntaxException
Sets the base URI for remote files, this is attached to any relative URIs provided in the get(...) methods

Parameters:
uriStr -
Throws:
URIxSyntaxException - if uriStr is malformed

getRemoteSource

public URIx getRemoteSource()
Returns:
the default URI source location for resolving relative URIs

getAbsoluteURI

public URIx getAbsoluteURI(java.lang.String relURIstr)
                    throws URIxSyntaxException
Converts a relative URI to an absolute one, using the remoteSource as a base. If the supplied URI string is absolute, the corresponding URI object is returned.

Parameters:
relURIstr - the relative URI
Returns:
converted URI
Throws:
URIxSyntaxException - if URI string is malformed

getAbsoluteURI

public URIx getAbsoluteURI(URIx relURI)
Converts a relative URI to an absolute one, using the remote source as a base. If the supplied URI is absolute, this is returned.

Parameters:
relURI -
Returns:
converted URI

getAbsoluteFile

public java.io.File getAbsoluteFile(java.io.File relFile)
Converts a relative file to an absolute one using the object's download directory. If the supplied file is absolute, this is returned.

Parameters:
relFile - the relative file
Returns:
absolute file

getAbsoluteFile

public java.io.File getAbsoluteFile(java.lang.String relPath)
Converts a relative file to an absolute one using the object's download directory. If the supplied file is absolute, this is returned.

Parameters:
relPath -
Returns:
absolute file

getRemoteHash

public java.lang.String getRemoteHash(java.lang.String uriStr)
                               throws FileGrabberException,
                                      URIxSyntaxException
Fetches a hash file from a remote location. The hash file is assumed to have the form <uri>.sha1

Parameters:
uriStr - the URI of the file to obtain the hash
Returns:
the hex-encoded hash value string
Throws:
FileGrabberException - if cannot retrieve remote hash
URIxSyntaxException - if uriStr is malformed

getRemoteHash

public java.lang.String getRemoteHash(URIx uri)
                               throws FileGrabberException
Fetches a hash file from a remote location. The hash file is assumed to have the form <uri>.sha1

Parameters:
uri - the URI of the file to obtain the hash
Returns:
the hex-encoded hash value string
Throws:
FileGrabberException

getLocalHash

public java.lang.String getLocalHash(java.lang.String fileName)
Gets the sha1 hash of a local file

Parameters:
fileName - file to compute the hash of
Returns:
the 20-byte hash as a hex-encoded String
Throws:
FileGrabberException - if fails to generate hash of local file

getLocalHash

public java.lang.String getLocalHash(java.io.File file)
Gets the sha1 hash of a local file

Parameters:
file - to compute the hash of
Returns:
the 20-byte hash as a hex-encoded String
Throws:
FileGrabberException - if fails to generate hash of local file

equalsHash

public boolean equalsHash(java.io.File file,
                          URIx uri)
                   throws FileGrabberException
Compares sha1 hash values between a local file and remote URI

Parameters:
file - local file of which to compute hash
uri - remote file of which to determine hash
Returns:
true if hashes are equal, false otherwise
Throws:
FileGrabberException - if can't get either hash

equalsHash

public boolean equalsHash(java.lang.String relPath)
                   throws FileGrabberException
Throws:
FileGrabberException

getLocal

public java.io.File getLocal(java.io.File file)
Retrieves a local file if it exists, null otherwise. If the file path is relative, then it prepends the download directory.

Parameters:
file - path for the local file
Returns:
the file handle

getLocal

public java.io.File getLocal(java.lang.String fileName)
Retrieves a local file if it exists, null otherwise.

Parameters:
fileName -
Returns:
File handle

getRemote

public java.io.File getRemote(java.io.File dest,
                              URIx source)
                       throws FileGrabberException
Retrieves a remote file if it exists, null otherwise. If dest is null or a directory, appends source filename

Parameters:
dest - the destination file (local)
source - the source URI
Returns:
a File reference to the new local copy
Throws:
FileGrabberException - if downloading the remote file fails

getRemote

public java.io.File getRemote(java.lang.String destName,
                              java.lang.String sourceName)
                       throws FileGrabberException,
                              URIxSyntaxException
Retrieves a remote file if it exists, null otherwise. If dest is null, or a directory, appends source filename

Parameters:
destName - the destination file (local)
sourceName - the source URI
Returns:
a File reference to the new local copy
Throws:
URIxSyntaxException - if the source URI is malformed
FileGrabberException - if grabbing the remote file fails

getRemote

public java.io.File getRemote(java.lang.String sourceName)
                       throws FileGrabberException
getRemote(String, String) with dest=null

Throws:
FileGrabberException

getRemote

public java.io.File getRemote(URIx source)
                       throws FileGrabberException
getRemote(File, URIx) with dest=null

Throws:
FileGrabberException

get

public java.io.File get(java.io.File dest,
                        URIx source,
                        int options)
                 throws FileGrabberException
Returns a file handle to a local version of the requested file. Downloads from URI if required (according to options). Works with absolute paths and source URIs, otherwise combines path and source URI with downloadDir and remoteSource, respectively. If the destination is null or a directory, then the filename of source is appended. If there is any internal problem, (such as failing to obtain a hash, or failing to download a file), the function will log the error message and continue.

Parameters:
dest - the local path (relative or absolute) to download file to
source - the remote URI to cache
options - set of options, either FORCE_REMOTE or CHECK_HASH
Returns:
File handle to local file
Throws:
FileGrabberException - only if there is no local copy of the file at the end of the function call

get

public java.io.File get(java.io.File path,
                        URIx source)
                 throws FileGrabberException
Downloads a file using the default options

Throws:
FileGrabberException
See Also:
get(File, URIx, int)

get

public java.io.File get(java.lang.String sourceName,
                        int options)
                 throws FileGrabberException
Downloads a file using same relative path for source and destination

Throws:
FileGrabberException
See Also:
get(String, String, int)

get

public java.io.File get(java.lang.String sourceName)
                 throws FileGrabberException
Downloads a file using same relative path for source and destination and default options

Throws:
FileGrabberException
See Also:
get(String, String, int)

get

public java.io.File get(java.lang.String destName,
                        java.lang.String sourceName,
                        int options)
                 throws URIxSyntaxException,
                        FileGrabberException
Converts the supplied destination path and source URI to a File and URI object, respectively, and downloads the remote file according to the supplied options.

Throws:
URIxSyntaxException - if the sourceName is malformed
FileGrabberException - if download fails.
See Also:
get(File, URIx, int)

get

public java.io.File get(java.lang.String destName,
                        java.lang.String sourceName)
                 throws FileGrabberException
Downloads a file with default options

Throws:
FileGrabberException
See Also:
get(String, String, int)

setLogger

public static void setLogger(Logger log)
Sets the logger for printing messages, defaults to printing to console.

Parameters:
log -

getLogger

public static Logger getLogger()
Gets the logger for printing messages

Returns:
log

setVerbosityLevel

public void setVerbosityLevel(int level)
Sets the verbosity level of this FileGrabber. A level of 0 will not print anything, level of 6 will print everything

Parameters:
level - between 0 (off) and 6 (print everything).

setVerbosityLevel

public void setVerbosityLevel(Logger.LogLevel level)
Sets the verbosity level of this FileGrabber. Any messages ranked higher than the supplied "level" will be printed.

Parameters:
level - The minimum message level to print, TRACE printing everything, NONE printing nothing.

setConsoleProgressPrinting

public void setConsoleProgressPrinting(boolean enable)
Enables or disables default printing of file transfer progress to the console. Disabled by default. Progress printing is done by installing a special transfer listener, which will not be returned by getTransferListeners().

Parameters:
enable - enables or disables console progress printing

getConsoleProgressPrinting

public boolean getConsoleProgressPrinting()
Returns true if default printing of file transfer progress to the console is enabled.

Returns:
true if default progress printing is enabled.

addTransferListener

public void addTransferListener(FileTransferListener listener)
Adds a FileTransferListener object that responds to transfer events. Useful for displaying download progress.

Parameters:
listener - The file listener object

removeTransferListener

public void removeTransferListener(FileTransferListener listener)
Removes a listener that is listener to transfer events.

Parameters:
listener - FileTransferListener to remove

getTransferListeners

public FileTransferListener[] getTransferListeners()
Gets the set of listeners for all transfers handled by this FileGrabber object

Returns:
Array of FileTransferListeners

getTransferMonitor

public FileTransferMonitor getTransferMonitor()
Returns the FileTransferMonitor object that is is responsible for detecting changes to the destination file and firing FileTransferEvents to the set of FileTransferListeners. By default, a MultiFileTransferMonitor is created which can monitor several file transfers at once.

Returns:
the current monitor
See Also:
addTransferListener(FileTransferListener)

setTransferMonitor

public void setTransferMonitor(FileTransferMonitor monitor)
Sets the FileTransferMonitor which is responsible for detecting when the destination file has changed in order to fire update events.

Parameters:
monitor - the monitor to use

setMonitorSleep

public void setMonitorSleep(double seconds)
Sets the sleep time between checking for transfer updates. Transfers are monitored by a FileTransferMonitor object that runs in a separate thread. This periodically polls the destination files to see if the transfer has progressed, and fires events to any FileTransferListeners supplied by the addTransferListener(FileTransferListener) function.

Parameters:
seconds -

addUserAuthenticator

public void addUserAuthenticator(URIxMatcher matcher,
                                 org.apache.commons.vfs2.UserAuthenticator auth)
Adds an authenticator for HTTPS, SFTP, that can respond to domain/username/password requests. This authenticator is used for any URIs that match patterns provided in `matcher'

Parameters:
matcher - object that checks whether a supplied URI matches a given set of criteria
auth - the authenticator object

addIdentityRepository

public void addIdentityRepository(URIxMatcher matcher,
                                  SimpleIdentityRepository repo)
Adds an identity repository consisting of a set of private RSA keys for use with SFTP authentication. The keys are used whenever a URI matches a set of parameters described in 'matcher'

Parameters:
matcher - object that checks whether supplied URI matches a given set of criteria
repo - repository containing a set of private RSA keys

getStaticGrabber

public static FileGrabber getStaticGrabber()
Returns a FileGrabber object used for staticGet(File, URIx). Creates one if it does not yet exist.

Returns:
the static FileGrabber object

setStaticGrabber

public static void setStaticGrabber(FileGrabber grabber)
Sets the FileGrabber used for staticGet(File, URIx).

Parameters:
grabber - the FileGrabber to set for static operations

staticGet

public static java.io.File staticGet(java.io.File dest,
                                     URIx source,
                                     int options)
                              throws FileGrabberException
Static convenience method for downloading a file in a single shot. This uses a static copy of a FileGrabber objects, creating one with default options if it doesn't exist.

Throws:
FileGrabberException
See Also:
get(File, URIx, int)

staticGet

public static java.io.File staticGet(java.io.File dest,
                                     URIx source)
                              throws FileGrabberException
Static convenience method for downloading a file. Uses default options.

Throws:
FileGrabberException
See Also:
staticGet(File, URIx, int)

staticGet

public static java.io.File staticGet(java.lang.String destName,
                                     java.lang.String sourceName,
                                     int options)
                              throws FileGrabberException
Static convenience method for downloading a file.

Throws:
FileGrabberException
See Also:
get(String, String, int)

staticGet

public static java.io.File staticGet(java.lang.String destName,
                                     java.lang.String sourceName)
                              throws FileGrabberException
Static convenience method for downloading a file with default options.

Throws:
FileGrabberException
See Also:
get(String, String, int)

setOptions

public void setOptions(int options)

getOptions

public int getOptions()

getInputStream

public java.io.InputStream getInputStream(java.io.File localCopy,
                                          URIx source,
                                          int options)
Returns an input stream for the file located at either localCopy or source (according to options). Works with absolute paths and source URIs, otherwise combines path and source URI with the default download directory and remote source uri, respectively. If the destination is null or a directory, then the filename of source is appended. If there is any internal problem, (such as failing to obtain a hash, or failing to download a file), the function will log the error message and continue.

Parameters:
localCopy - the local path (relative or absolute) to check for overriding file
source - the remote URI to read from
options - set of options, from FORCE_REMOTE, DOWNLOAD_ZIP, CHECK_HASH
Returns:
File input stream
Throws:
FileGrabberException - if we cannot open the stream

getInputStream

public java.io.InputStream getInputStream(URIx source,
                                          int options)
                                   throws FileGrabberException
Throws:
FileGrabberException

getInputStream

public java.io.InputStream getInputStream(URIx source)

getInputStream

public java.io.InputStream getInputStream(java.lang.String localCopy,
                                          java.lang.String sourceName,
                                          int options)
                                   throws FileGrabberException
Throws:
FileGrabberException

getInputStream

public java.io.InputStream getInputStream(java.lang.String localCopy,
                                          java.lang.String sourceName)
                                   throws FileGrabberException
Throws:
FileGrabberException

getInputStream

public java.io.InputStream getInputStream(java.lang.String sourceName,
                                          int options)
                                   throws FileGrabberException
Throws:
FileGrabberException

getInputStream

public java.io.InputStream getInputStream(java.lang.String sourceName)
                                   throws FileGrabberException
Throws:
FileGrabberException

closeStreams

public void closeStreams()
Must be called to free resources after streams have been read.


getLastFile

public java.io.File getLastFile()
Gets the last file retrieved.


wasLastRemote

public boolean wasLastRemote()
Returns true if the last file was fetched from remote, false if last file was a local copy