Document Loader Job: Move invalid documents to error directory
Description
If a document loader is scheduled to run on a regular basis and it encounters errors, it can generate a lot of user messages as the same files are processed each time. To avoid this, documents that have generated errors should be moved to an optional 'error directory', specified as part of the job configuration.
Environment
None
Activity
Show:
Tim Anderson November 11, 2015 at 2:57 AM
Changes applied in revision 6813.
The entity.jobDocumentLoader now has an optional errorDir node to specify the error directory. The docload tool now accepts a --err argument to specify the error directory.
If a document loader is scheduled to run on a regular basis and it encounters errors, it can generate a lot of user messages as the same files are processed each time.
To avoid this, documents that have generated errors should be moved to an optional 'error directory', specified as part of the job configuration.