A typical use will look like this:
FileGrabber grabber = new FileGrabber("data", "http://www.fileserver.com");
grabber.get("localFile", "remoteFile");
By default, the FileGrabber will simply return the localFile if it exists. To
force updates from a remote location, you can set options:
int options = FileGrabber.CHECK_HASH;
grabber.get("localFile", "remoteFile", options);
The CHECK_HASH flag downloads the remote file if its sha1 hash differs from
that of your local copy. There is also a FORCE_REMOTE flag that forces the
remote file to be downloaded always.
For fine control and catching errors, the workflow is slightly different:
boolean download = true;
File dest = new File("localFile");
URI source = new URI("remoteFile");
File local = grabber.getLocal(dest);
if (local != null) {
boolean match = false;
try {
match = grabber.equalsHash(dest, source);
} catch (FileGrabberException e) {
System.out.println(e.getMessage());
}
download = !match;
}
if (download) {
try {
local = grabber.getRemote(dest, source);
} catch (FileGrabberException e) {
System.out.println(e.getMessage());
}
}
if (local == null) {
throw new RuntimeException("Unable to get file.");
}
See VFS2 File
Systems for further details regarding supported remote filesystems and
URI syntax.
- Author:
- "Antonio Sanchez" Creation date: 13 Nov 2012
Constructor Summary |
FileGrabber()
Default constructor, sets the local directory to the current path, and
sets the default URI to be empty. |
FileGrabber(java.io.File downloadDir,
URIx remoteSource)
Sets default paths |
FileGrabber(java.lang.String downloadPath)
Sets default download directory, leaves source as null |
FileGrabber(java.lang.String downloadPath,
java.lang.String remoteSourceName)
Sets local download path and remote URI source by parsing the supplied
strings |
Method Summary |
void |
addIdentityRepository(URIxMatcher matcher,
SimpleIdentityRepository repo)
Adds an identity repository consisting of a set of private RSA keys for
use with SFTP authentication. |
void |
addTransferListener(FileTransferListener listener)
Adds a FileTransferListener object that responds to transfer events. |
void |
addUserAuthenticator(URIxMatcher matcher,
org.apache.commons.vfs2.UserAuthenticator auth)
Adds an authenticator for HTTPS, SFTP, that can respond to
domain/username/password requests. |
void |
closeStreams()
Must be called to free resources after streams have been read. |
boolean |
equalsHash(java.io.File file,
URIx uri)
Compares sha1 hash values between a local file and remote URI |
boolean |
equalsHash(java.lang.String relPath)
|
java.io.File |
get(java.io.File path,
URIx source)
Downloads a file using the default options |
java.io.File |
get(java.io.File dest,
URIx source,
int options)
Returns a file handle to a local version of the requested file. |
java.io.File |
get(java.lang.String sourceName)
Downloads a file using same relative path for source and destination
and default options |
java.io.File |
get(java.lang.String sourceName,
int options)
Downloads a file using same relative path for source and destination |
java.io.File |
get(java.lang.String destName,
java.lang.String sourceName)
Downloads a file with default options |
java.io.File |
get(java.lang.String destName,
java.lang.String sourceName,
int options)
Converts the supplied destination path and source URI to a File and URI
object, respectively, and downloads the remote file according to the
supplied options. |
java.io.File |
getAbsoluteFile(java.io.File relFile)
Converts a relative file to an absolute one using the object's download
directory. |
java.io.File |
getAbsoluteFile(java.lang.String relPath)
Converts a relative file to an absolute one using the object's download
directory. |
URIx |
getAbsoluteURI(java.lang.String relURIstr)
Converts a relative URI to an absolute one, using the remoteSource as a
base. |
URIx |
getAbsoluteURI(URIx relURI)
Converts a relative URI to an absolute one, using the remote source as a
base. |
boolean |
getConsoleProgressPrinting()
Returns true if default printing of file transfer progress to the
console is enabled. |
java.io.File |
getDownloadDir()
|
java.io.InputStream |
getInputStream(java.io.File localCopy,
URIx source,
int options)
Returns an input stream for the file located at either localCopy
or source (according to options). |
java.io.InputStream |
getInputStream(java.lang.String sourceName)
|
java.io.InputStream |
getInputStream(java.lang.String sourceName,
int options)
|
java.io.InputStream |
getInputStream(java.lang.String localCopy,
java.lang.String sourceName)
|
java.io.InputStream |
getInputStream(java.lang.String localCopy,
java.lang.String sourceName,
int options)
|
java.io.InputStream |
getInputStream(URIx source)
|
java.io.InputStream |
getInputStream(URIx source,
int options)
|
java.io.File |
getLastFile()
Gets the last file retrieved. |
java.io.File |
getLocal(java.io.File file)
Retrieves a local file if it exists, null otherwise. |
java.io.File |
getLocal(java.lang.String fileName)
Retrieves a local file if it exists, null otherwise. |
java.lang.String |
getLocalHash(java.io.File file)
Gets the sha1 hash of a local file |
java.lang.String |
getLocalHash(java.lang.String fileName)
Gets the sha1 hash of a local file |
static Logger |
getLogger()
Gets the logger for printing messages |
int |
getOptions()
|
java.io.File |
getRemote(java.io.File dest,
URIx source)
Retrieves a remote file if it exists, null otherwise. |
java.io.File |
getRemote(java.lang.String sourceName)
getRemote(String, String) with dest=null |
java.io.File |
getRemote(java.lang.String destName,
java.lang.String sourceName)
Retrieves a remote file if it exists, null otherwise. |
java.io.File |
getRemote(URIx source)
getRemote(File, URIx) with dest=null |
java.lang.String |
getRemoteHash(java.lang.String uriStr)
Fetches a hash file from a remote location. |
java.lang.String |
getRemoteHash(URIx uri)
Fetches a hash file from a remote location. |
URIx |
getRemoteSource()
|
static FileGrabber |
getStaticGrabber()
Returns a FileGrabber object used for staticGet(File, URIx) . |
FileTransferListener[] |
getTransferListeners()
Gets the set of listeners for all transfers handled by this FileGrabber
object |
FileTransferMonitor |
getTransferMonitor()
Returns the FileTransferMonitor object that is is responsible for
detecting changes to the destination file and firing FileTransferEvents to
the set of FileTransferListener s. |
void |
removeTransferListener(FileTransferListener listener)
Removes a listener that is listener to transfer events. |
void |
setConsoleProgressPrinting(boolean enable)
Enables or disables default printing of file transfer progress to the
console. |
void |
setDownloadDir(java.io.File dir)
Sets directory where files are downloaded |
void |
setDownloadDir(java.lang.String path)
Sets directory where files are downloaded |
static void |
setLogger(Logger log)
Sets the logger for printing messages, defaults
to printing to console. |
void |
setMonitorSleep(double seconds)
Sets the sleep time between checking for transfer updates. |
void |
setOptions(int options)
|
void |
setRemoteSource(java.lang.String uriStr)
Sets the base URI for remote files, this is attached to any relative URIs
provided in the get(...) methods |
void |
setRemoteSource(URIx uri)
Sets the base URI for remote files, this is attached to any relative URIs
provided in the get(...) methods |
static void |
setStaticGrabber(FileGrabber grabber)
Sets the FileGrabber used for staticGet(File, URIx) . |
void |
setTransferMonitor(FileTransferMonitor monitor)
Sets the FileTransferMonitor which is responsible for detecting when the
destination file has changed in order to fire update events. |
void |
setVerbosityLevel(int level)
Sets the verbosity level of this FileGrabber. |
void |
setVerbosityLevel(Logger.LogLevel level)
Sets the verbosity level of this FileGrabber. |
static java.io.File |
staticGet(java.io.File dest,
URIx source)
Static convenience method for downloading a file. |
static java.io.File |
staticGet(java.io.File dest,
URIx source,
int options)
Static convenience method for downloading a file in a single shot. |
static java.io.File |
staticGet(java.lang.String destName,
java.lang.String sourceName)
Static convenience method for downloading a file with default options. |
static java.io.File |
staticGet(java.lang.String destName,
java.lang.String sourceName,
int options)
Static convenience method for downloading a file. |
boolean |
wasLastRemote()
Returns true if the last file was fetched from remote,
false if last file was a local copy |
Methods inherited from class java.lang.Object |
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
FORCE_REMOTE
public static final int FORCE_REMOTE
- Always read from remote if possible
- See Also:
- Constant Field Values
CHECK_HASH
public static final int CHECK_HASH
- Check file hashes, and if different, get from remote
- See Also:
- Constant Field Values
DOWNLOAD_ZIP
public static final int DOWNLOAD_ZIP
- If file is in a remote zip file, get a local copy
of the entire zip file first
- See Also:
- Constant Field Values
DEFAULT_OPTIONS
public static int DEFAULT_OPTIONS
DEFAULT_LOG_LEVEL
public static int DEFAULT_LOG_LEVEL
DEFAULT_LOGGER
public static Logger DEFAULT_LOGGER
myOptions
public int myOptions
FileGrabber
public FileGrabber()
- Default constructor, sets the local directory to the current path, and
sets the default URI to be empty.
FileGrabber
public FileGrabber(java.io.File downloadDir,
URIx remoteSource)
- Sets default paths
- Parameters:
downloadDir
- the local path to save files toremoteSource
- the remote base URI to download files from
FileGrabber
public FileGrabber(java.lang.String downloadPath)
- Sets default download directory, leaves source as null
- Parameters:
downloadPath
- the local path to save files to
FileGrabber
public FileGrabber(java.lang.String downloadPath,
java.lang.String remoteSourceName)
- Sets local download path and remote URI source by parsing the supplied
strings
- Parameters:
downloadPath
- the local path to save files toremoteSourceName
- the remote base URI
setDownloadDir
public void setDownloadDir(java.io.File dir)
- Sets directory where files are downloaded
- Parameters:
dir
- default download directory
setDownloadDir
public void setDownloadDir(java.lang.String path)
- Sets directory where files are downloaded
- Parameters:
path
- default download directory
getDownloadDir
public java.io.File getDownloadDir()
- Returns:
- the default download directory
setRemoteSource
public void setRemoteSource(URIx uri)
- Sets the base URI for remote files, this is attached to any relative URIs
provided in the get(...) methods
- Parameters:
uri
-
setRemoteSource
public void setRemoteSource(java.lang.String uriStr)
throws URIxSyntaxException
- Sets the base URI for remote files, this is attached to any relative URIs
provided in the get(...) methods
- Parameters:
uriStr
-
- Throws:
URIxSyntaxException
- if uriStr is malformed
getRemoteSource
public URIx getRemoteSource()
- Returns:
- the default URI source location for resolving relative URIs
getAbsoluteURI
public URIx getAbsoluteURI(java.lang.String relURIstr)
throws URIxSyntaxException
- Converts a relative URI to an absolute one, using the remoteSource as a
base. If the supplied URI string is absolute, the corresponding URI object
is returned.
- Parameters:
relURIstr
- the relative URI
- Returns:
- converted URI
- Throws:
URIxSyntaxException
- if URI string is malformed
getAbsoluteURI
public URIx getAbsoluteURI(URIx relURI)
- Converts a relative URI to an absolute one, using the remote source as a
base. If the supplied URI is absolute, this is returned.
- Parameters:
relURI
-
- Returns:
- converted URI
getAbsoluteFile
public java.io.File getAbsoluteFile(java.io.File relFile)
- Converts a relative file to an absolute one using the object's download
directory. If the supplied file is absolute, this is returned.
- Parameters:
relFile
- the relative file
- Returns:
- absolute file
getAbsoluteFile
public java.io.File getAbsoluteFile(java.lang.String relPath)
- Converts a relative file to an absolute one using the object's download
directory. If the supplied file is absolute, this is returned.
- Parameters:
relPath
-
- Returns:
- absolute file
getRemoteHash
public java.lang.String getRemoteHash(java.lang.String uriStr)
throws FileGrabberException,
URIxSyntaxException
- Fetches a hash file from a remote location. The hash file is assumed to
have the form <uri>.sha1
- Parameters:
uriStr
- the URI of the file to obtain the hash
- Returns:
- the hex-encoded hash value string
- Throws:
FileGrabberException
- if cannot retrieve remote hash
URIxSyntaxException
- if uriStr is malformed
getRemoteHash
public java.lang.String getRemoteHash(URIx uri)
throws FileGrabberException
- Fetches a hash file from a remote location. The hash file is assumed to
have the form <uri>.sha1
- Parameters:
uri
- the URI of the file to obtain the hash
- Returns:
- the hex-encoded hash value string
- Throws:
FileGrabberException
getLocalHash
public java.lang.String getLocalHash(java.lang.String fileName)
- Gets the sha1 hash of a local file
- Parameters:
fileName
- file to compute the hash of
- Returns:
- the 20-byte hash as a hex-encoded String
- Throws:
FileGrabberException
- if fails to generate hash of local file
getLocalHash
public java.lang.String getLocalHash(java.io.File file)
- Gets the sha1 hash of a local file
- Parameters:
file
- to compute the hash of
- Returns:
- the 20-byte hash as a hex-encoded String
- Throws:
FileGrabberException
- if fails to generate hash of local file
equalsHash
public boolean equalsHash(java.io.File file,
URIx uri)
throws FileGrabberException
- Compares sha1 hash values between a local file and remote URI
- Parameters:
file
- local file of which to compute hashuri
- remote file of which to determine hash
- Returns:
- true if hashes are equal, false otherwise
- Throws:
FileGrabberException
- if can't get either hash
equalsHash
public boolean equalsHash(java.lang.String relPath)
throws FileGrabberException
- Throws:
FileGrabberException
getLocal
public java.io.File getLocal(java.io.File file)
- Retrieves a local file if it exists, null otherwise. If the file
path is relative, then it prepends the download directory.
- Parameters:
file
- path for the local file
- Returns:
- the file handle
getLocal
public java.io.File getLocal(java.lang.String fileName)
- Retrieves a local file if it exists, null otherwise.
- Parameters:
fileName
-
- Returns:
- File handle
getRemote
public java.io.File getRemote(java.io.File dest,
URIx source)
throws FileGrabberException
- Retrieves a remote file if it exists, null otherwise. If dest is null or a
directory, appends source filename
- Parameters:
dest
- the destination file (local)source
- the source URI
- Returns:
- a File reference to the new local copy
- Throws:
FileGrabberException
- if downloading the remote file fails
getRemote
public java.io.File getRemote(java.lang.String destName,
java.lang.String sourceName)
throws FileGrabberException,
URIxSyntaxException
- Retrieves a remote file if it exists, null otherwise. If dest is null, or
a directory, appends source filename
- Parameters:
destName
- the destination file (local)sourceName
- the source URI
- Returns:
- a File reference to the new local copy
- Throws:
URIxSyntaxException
- if the source URI is malformed
FileGrabberException
- if grabbing the remote file fails
getRemote
public java.io.File getRemote(java.lang.String sourceName)
throws FileGrabberException
getRemote(String, String)
with dest=null
- Throws:
FileGrabberException
getRemote
public java.io.File getRemote(URIx source)
throws FileGrabberException
getRemote(File, URIx)
with dest=null
- Throws:
FileGrabberException
get
public java.io.File get(java.io.File dest,
URIx source,
int options)
throws FileGrabberException
- Returns a file handle to a local version of the requested file. Downloads
from URI if required (according to options). Works with absolute paths and
source URIs, otherwise combines path and source URI with downloadDir and
remoteSource, respectively. If the destination is null or a directory,
then the filename of source is appended.
If there is any internal problem, (such as failing to obtain a hash, or
failing to download a file), the function will log the error message and
continue.
- Parameters:
dest
- the local path (relative or absolute) to download file tosource
- the remote URI to cacheoptions
- set of options, either FORCE_REMOTE or CHECK_HASH
- Returns:
- File handle to local file
- Throws:
FileGrabberException
- only if there is no local copy of the file
at the end of the function call
get
public java.io.File get(java.io.File path,
URIx source)
throws FileGrabberException
- Downloads a file using the default options
- Throws:
FileGrabberException
- See Also:
get(File, URIx, int)
get
public java.io.File get(java.lang.String sourceName,
int options)
throws FileGrabberException
- Downloads a file using same relative path for source and destination
- Throws:
FileGrabberException
- See Also:
get(String, String, int)
get
public java.io.File get(java.lang.String sourceName)
throws FileGrabberException
- Downloads a file using same relative path for source and destination
and default options
- Throws:
FileGrabberException
- See Also:
get(String, String, int)
get
public java.io.File get(java.lang.String destName,
java.lang.String sourceName,
int options)
throws URIxSyntaxException,
FileGrabberException
- Converts the supplied destination path and source URI to a File and URI
object, respectively, and downloads the remote file according to the
supplied options.
- Throws:
URIxSyntaxException
- if the sourceName is malformed
FileGrabberException
- if download fails.- See Also:
get(File, URIx, int)
get
public java.io.File get(java.lang.String destName,
java.lang.String sourceName)
throws FileGrabberException
- Downloads a file with default options
- Throws:
FileGrabberException
- See Also:
get(String, String, int)
setLogger
public static void setLogger(Logger log)
- Sets the logger for printing messages, defaults
to printing to console.
- Parameters:
log
-
getLogger
public static Logger getLogger()
- Gets the logger for printing messages
- Returns:
- log
setVerbosityLevel
public void setVerbosityLevel(int level)
- Sets the verbosity level of this FileGrabber. A level of 0 will not print
anything, level of 6 will print everything
- Parameters:
level
- between 0 (off) and 6 (print everything).
setVerbosityLevel
public void setVerbosityLevel(Logger.LogLevel level)
- Sets the verbosity level of this FileGrabber. Any messages ranked higher
than the supplied "level" will be printed.
- Parameters:
level
- The minimum message level to print, TRACE printing everything, NONE
printing nothing.
setConsoleProgressPrinting
public void setConsoleProgressPrinting(boolean enable)
- Enables or disables default printing of file transfer progress to the
console. Disabled by default. Progress printing is done by installing a
special transfer listener, which will not be returned by
getTransferListeners()
.
- Parameters:
enable
- enables or disables console progress printing
getConsoleProgressPrinting
public boolean getConsoleProgressPrinting()
- Returns true if default printing of file transfer progress to the
console is enabled.
- Returns:
- true if default progress printing is enabled.
addTransferListener
public void addTransferListener(FileTransferListener listener)
- Adds a FileTransferListener object that responds to transfer events.
Useful for displaying download progress.
- Parameters:
listener
- The file listener object
removeTransferListener
public void removeTransferListener(FileTransferListener listener)
- Removes a listener that is listener to transfer events.
- Parameters:
listener
- FileTransferListener to remove
getTransferListeners
public FileTransferListener[] getTransferListeners()
- Gets the set of listeners for all transfers handled by this FileGrabber
object
- Returns:
- Array of FileTransferListeners
getTransferMonitor
public FileTransferMonitor getTransferMonitor()
- Returns the FileTransferMonitor object that is is responsible for
detecting changes to the destination file and firing FileTransferEvents to
the set of
FileTransferListener
s. By default, a
MultiFileTransferMonitor
is created which can monitor several file
transfers at once.
- Returns:
- the current monitor
- See Also:
addTransferListener(FileTransferListener)
setTransferMonitor
public void setTransferMonitor(FileTransferMonitor monitor)
- Sets the FileTransferMonitor which is responsible for detecting when the
destination file has changed in order to fire update events.
- Parameters:
monitor
- the monitor to use
setMonitorSleep
public void setMonitorSleep(double seconds)
- Sets the sleep time between checking for transfer updates. Transfers are
monitored by a
FileTransferMonitor
object that runs in a separate
thread. This periodically polls the destination files to see if the
transfer has progressed, and fires events to any FileTransferListeners
supplied by the addTransferListener(FileTransferListener)
function.
- Parameters:
seconds
-
addUserAuthenticator
public void addUserAuthenticator(URIxMatcher matcher,
org.apache.commons.vfs2.UserAuthenticator auth)
- Adds an authenticator for HTTPS, SFTP, that can respond to
domain/username/password requests. This authenticator is used for any URIs
that match patterns provided in `matcher'
- Parameters:
matcher
- object that checks whether a supplied URI matches a given set of criteriaauth
- the authenticator object
addIdentityRepository
public void addIdentityRepository(URIxMatcher matcher,
SimpleIdentityRepository repo)
- Adds an identity repository consisting of a set of private RSA keys for
use with SFTP authentication. The keys are used whenever a URI matches a
set of parameters described in 'matcher'
- Parameters:
matcher
- object that checks whether supplied URI matches a given set of criteriarepo
- repository containing a set of private RSA keys
getStaticGrabber
public static FileGrabber getStaticGrabber()
- Returns a FileGrabber object used for
staticGet(File, URIx)
.
Creates one if it does not yet exist.
- Returns:
- the static FileGrabber object
setStaticGrabber
public static void setStaticGrabber(FileGrabber grabber)
- Sets the FileGrabber used for
staticGet(File, URIx)
.
- Parameters:
grabber
- the FileGrabber to set for static operations
staticGet
public static java.io.File staticGet(java.io.File dest,
URIx source,
int options)
throws FileGrabberException
- Static convenience method for downloading a file in a single shot. This
uses a static copy of a FileGrabber objects, creating one with default
options if it doesn't exist.
- Throws:
FileGrabberException
- See Also:
get(File, URIx, int)
staticGet
public static java.io.File staticGet(java.io.File dest,
URIx source)
throws FileGrabberException
- Static convenience method for downloading a file. Uses default options.
- Throws:
FileGrabberException
- See Also:
staticGet(File, URIx, int)
staticGet
public static java.io.File staticGet(java.lang.String destName,
java.lang.String sourceName,
int options)
throws FileGrabberException
- Static convenience method for downloading a file.
- Throws:
FileGrabberException
- See Also:
get(String, String, int)
staticGet
public static java.io.File staticGet(java.lang.String destName,
java.lang.String sourceName)
throws FileGrabberException
- Static convenience method for downloading a file with default options.
- Throws:
FileGrabberException
- See Also:
get(String, String, int)
setOptions
public void setOptions(int options)
getOptions
public int getOptions()
getInputStream
public java.io.InputStream getInputStream(java.io.File localCopy,
URIx source,
int options)
- Returns an input stream for the file located at either
localCopy
or source
(according to options). Works with absolute paths and
source URIs, otherwise combines path and source URI with the default
download directory and remote source uri, respectively. If the
destination is null or a directory, then the filename of source is
appended.
If there is any internal problem, (such as failing to obtain a hash, or
failing to download a file), the function will log the error message and
continue.
- Parameters:
localCopy
- the local path (relative or absolute) to check for overriding filesource
- the remote URI to read fromoptions
- set of options, from FORCE_REMOTE, DOWNLOAD_ZIP, CHECK_HASH
- Returns:
- File input stream
- Throws:
FileGrabberException
- if we cannot open the stream
getInputStream
public java.io.InputStream getInputStream(URIx source,
int options)
throws FileGrabberException
- Throws:
FileGrabberException
getInputStream
public java.io.InputStream getInputStream(URIx source)
getInputStream
public java.io.InputStream getInputStream(java.lang.String localCopy,
java.lang.String sourceName,
int options)
throws FileGrabberException
- Throws:
FileGrabberException
getInputStream
public java.io.InputStream getInputStream(java.lang.String localCopy,
java.lang.String sourceName)
throws FileGrabberException
- Throws:
FileGrabberException
getInputStream
public java.io.InputStream getInputStream(java.lang.String sourceName,
int options)
throws FileGrabberException
- Throws:
FileGrabberException
getInputStream
public java.io.InputStream getInputStream(java.lang.String sourceName)
throws FileGrabberException
- Throws:
FileGrabberException
closeStreams
public void closeStreams()
- Must be called to free resources after streams have been read.
getLastFile
public java.io.File getLastFile()
- Gets the last file retrieved.
wasLastRemote
public boolean wasLastRemote()
- Returns true if the last file was fetched from remote,
false if last file was a local copy