Introduction
Planning
Installation
Loading Documents
Exporting Documents
Deleting Documents
Troubleshooting
Error Recovery
Change Log
Tyler Technologies’ iasWorld Document Loader is a standalone tool that is intended to assist users with loading large numbers of photos or documents into iasWorld’s Document Manager module. It is most commonly used for initial bulk loads of photos or documents when a new Document Manager installation is being set up, but it can also be used to automate loading of documents resulting from scanners and other automated processes.
This release of Document Loader has the following restrictions.
In general, best performance is achieved when file access does not require network requests. This means that dodcuments and photos should be located on the system the loader is run from. In practice, given the size and difficulty of moving large files around, it is often easiest to simply install the loader on the system where the files are.
If local access to document and photo files is impossible, maximize the network speed on file requests by using gigabit network segments if possible, or by using network segements that carry little other traffic.
When large photo/document sets are being loaded, overall performance can be improved by splitting the photo set into multiple segments (e.g. 150,000 into 3 50,000 segments). The separate segments can then be loaded concurrently by using separate document loader instances on separate physical or virtual machines. This will improve performance by leveraging the ability of the server to handle multiple requests simultaneously.
It is essential to ensure that sufficient database capacity exists in the Document Manager schema to allow all documents to successfully load. The capacity required is directly related to the binary size of the files being loaded; metadata and other internal data required by Document Manager makes up a very tiny portion of the overall space required.
Photos and documents require space to load the full size of the document. No internal compression is used, for the reason that most current file formats that would be stored are already compressed and cannot be compressed further. In addition, photos require an additional 30 KB to allow for the creation of a thumbnail image.
In addition, a 20% slack factor should be allowed for internal Oracle needs and to allow for growth.
A rough calculation of size required is therefore:
In configuring Oracle schemas for Document Manager use, the following adjustments can be made to Oracle:
Note that these adjustments are specifically for Document Manager use and should not be used for other schemas.
The iasWorld Document Loader requires .NET 4.5.2 to be installed on the machine where the loader is to be installed. The .NET 4.5.2 installer may be downloaded from Microsoft. NOTE: the use of .NET 4.5.2 is a new requirement, as older versions of this tool used .NET 4.0 and .NET 4.5.
The loader may be installed and used from any system that has the following:
Best performance will be obtained by minimizing the network pathways from the loader to the documents or photos being loaded. Ideally, the loader should be installed on the systemwhere the documents and photos are stored so that they may be loaded directly from the filesystem.
It is also possible to run the loader directly from the iasWorld server, thereby minimizing the network pathways in connecting to iasWorld. If doing this, the following guidelines should be followed:
One way to do this is to launch the loader from the command line using the following syntax:
START /HIGH <loaderpathname>
To install the document loader:
When you start Document Loader, you are asked for a URL for the iasWorld Document Manager site you are connecting to. This URL will be the URL to the iasWorld site’s root folder (e.g. http://myserver/iasworld).
This URL may be HTTP or HTTPS. However, when using HTTPS, you must ensure that the URL you provide matches the URL on the site’s SSL certificate exactly. If, for example, the certificate is registered to a fully qualified name with a domain such as myserver.myorganization.com, you must provide that full name. Further, you may need need to ensure that a root certificate is installed on the client computer if the server’s SSL certificate is a self-generated one. These restrictions are because code-based connections to SSL do not provide the option to connect anyway when a certificate error occurs.
In addition, you must provide a valid iasWorld username and password. The user must have “Create” access to Document Manager.
One option for the input source to iasWorld’s Document Loader is a CSV (comma-separated values) file . This file lists all documents or photos to be listed within the CSV along with all field values that should be used for each document. The CSV file must be located in the top-level folder where the photos or documents are stored.
The CSV file format must be exactly as listed below, will all fields in this order.
Fields may not be removed from the format.
In addition, Document Manager must have field definitions matching exactly the fields below, with no missing fields. Document Manager may have additional fields defined that are not present in in the CSV; these fields will have null values created when loading and can later be modified via the Document Manager user interface or directly via the Document Manager schema.
The first line of the file is assumed to be a header line containing field names, and will be skipped. The header is present to make the file more understandable to humans and is not used by the Loader in any way. Processing of the CSV file occurs positionally, with the first column assumed to be filename, the second column assumed to be parid, and so forth.
The fields that must be included in the CSV are as follows:
When using the CSV source, the CSV file must be placed in the parent folder under which all the photos are located.
Parid, Jur, and Filename are required in all cases. Other fields can be left as empty values (using subsequent commas such as test,,1)
Parcel Photos and Parcel Documents share a common CSV format. However, the Rank field is ignored for Parcel Documents and may be left blank.
If Parid and/or Jur are incorrect, the documents loaded using the incorrect values may be difficult or impossible to retrieve in iasWorld. These values are used to query for documents within iasWorld. Please ensure that these values are correct in the CSV file.
If fields are missing, an error will be logged and the photo or document referred to will not be loaded.
A short sample of the CSV format follows. A longer sample is included with this application as “photo sample.csv”.
Filename,Parid,Rank,Jur,Year,Card,Category,Title,Notes,CapturedBy,CaptureDate
C:\temp\027149.jpg,027149,1,000,2011,1,Front,Test,New porch,,
C:\temp\027907.jpg,027907,,000,2011,,,,,,
Important: When preparing or editing a CSV file, use a text editor such as Notepad or Notepad++ to ensure the file formatting is correct. Avoid using Microsoft Excel or other spreadsheet tools if possible as they can sometimes handle CSV files in a nonstandard way.
A second option for the input source is to load files directly from a folder, using the filename of each file for the parid (and for photos, rank) and providing the other required details directly to Document Loader. This option is particularly appropriate when loading files that have been exported from Landisc, as this is Landisc’s native export format.
When using the Folder source, the following is expected:
When using the Folder source, all photos or documents being loaded must be located “underneath” the selected folder.
This release of Document Loader loads documents via a direct connection to the database for maximum performance, making it several times faster than previous releases.
In addition, when the current Document Type in Document Manager has been configured within iasWorld to use the “Filesystem” storage instead of the “iasWorld Internal” storage, this system stores the photo or document binaries directly on a filesystem or Storage Area Network (SAN) rather than within Oracle, although Oracle is still used for storage of document and photo metadata.
Support for loading to a filesystem is automatic when iasWorld has been configured to use the filesystem.
However, all photos must be already be stored in or underneath the permanent filesystem storage folder that will be used by Document Manager to hold the photos. This location must therefore be on a server accessible to the Document Manager service, and the Document Manager service should be configured to access the files from that location before loading is begun.
The Document Loader will not move the files to another location. If the files are not in the correct permanent storage location, the loaded documents and folders will not be accessible in Document Manager and attempts to access them will produce an error. It is essential to ensure that the files are in their correct permanent location before loading them, moving them first if necessary.
Start Document Loader by clicking on iasWorld Document Loader in the Start menu.
Click the Start button with Document Loader. Document Loader will begin loading document or photos. As documents load, you will see them removed from the Documents Remaining count to the Documents Completed count.
Start Document Loader by clicking on iasWorld Document Loader in the Start menu.
When importing to a Document Manager service configured to use “Filesystem” storage, the folder you pick must be the permanent storage location. For example, if all photos are placed in \san-server\photos, then you must pick that folder in Document Loader.
Click the Start button with Document Loader. Document Loader will begin loading document or photos. As documents load, you will see them removed from the Documents Remaining count to the Documents Completed count.
Document Loader also supports running a post processing job against the photos or documents loaded in the batch. This is done by creating a stored procedure that is passed the timestamp for the moment the batch job started. This timestamp is stored in IDOCX_DOCUMENT.JOB_STARTED_ON and therefore can be used to identify the specific documents loaded during that job. Once the documents are identified, any logic that is needed can be performed.
Some important points to note are:
1) This feature is disabled by default. To enable, open the file DocumentLoader.exe.config from the Document Loader folder in a text editor such as Notepad, and find the following value value, and change “false” to “true”:
<add key="CallCustomPostProcessProcedure" value="false"/>
2) Once enabled, an error will be displayed if Document Loader cannot find a custom post processing procedure.
3) To create a procedure, create an Oracle package with the following signature. In the body, use the i_batch_guid parameter to provide an implementation of the business logic you desire, by matching it against document/photo records with the same value stored in the IDOCX_DOCUMENT table. Any records matching the value provided to the script will be the photos or documents loaded during the just-finished batch load. An example of such a script is included within the Document Loader zip file under the name IDOCX_CUSTOM example.pck; the example sets the category for all previous photos for the parcels in the just-finished batch to “Archive”.
create or replace package IDOCX_CUSTOM is
PROCEDURE post_batch_load(i_batch_guid VARCHAR2);
end IDOCX_CUSTOM;
4) For assistance with this feature, please contact your Tyler representative.
In addition to loading documents, Document Loader provides the ability to export documents from Document Manager’s Oracle storage to the filesystem.
The following restrictions apply: * Exports are only possible from Document Manager Internal or Filesystem storage. If Document Manager is configured to use an EDMS integration, exports should be performed using the EDMS’s own tools. * Documents can be restricted by tax year and/or category, or run unrestricted. Please be aware that unrestricted exports will process all documents or photos, and will consequently take a long time to run. * The total file sizes of documents or photos can be very large. Please ensure that the drive/folder selected for export has sufficient free space available to hold the documents or photos.
To export documents or photos:
Start Document Loader by clicking on iasWorld Document Loader in the Start menu.
Click the Export tab.
Note that photos are written to the file system using Landisc format, using “parid-rank.ext”. Documents are written out using “parid-originalFileName.ext” as the format.
Document manager also provides the ability to perform bulk deletions of content, with optional archiving. This is intended to be used as a way of removing aged-out photos and documents in an efficient manner. The deletion operation requires a CSV file containing a column of document IDs. Please see “delete sample.csv” as a simple sample csv. Note that additional columns may be present in the CSV if desired; deletion operations will not look at those columns and will not be affected by their presence.
To delete documents:
Start Document Loader by clicking on iasWorld Document Loader in the Start menu.
Click the Delete tab.
Click Start. You will be warned that the process will be deleting documents permanently. Document Loader will begin deleted document or photos. As documents are deleted , you will see them removed from the Documents Remaining count to the Documents Completed count.
If problems occur, click the View Logs tab to see the information and error logs. The information log tracks each document that is successfully loaded. The error log shows internal error details when they occur. Please note that when an error is shown for a document, that document has not been loaded and will need to be reloaded once the issue is resolved. Please see Error Recovery for more detail.
To forward logs to Document Loader support, click the View In Explorer button. This will open the log file folder in Explorer, allowing the files to be manipulated or attached to an email message.
All loader activities are logged. If error occurs during loading, resulting in a need to reload documents or photos, the “Skip loaded documents” feature allows loader to skip parcels that were successfully loaded.
To use this, simply set up your CSV file or your Filesystem folder in the loader as you would normally, then click the “Skip loaded documents” button.
The loader will process the log files to determine which parcels may be skipped, and it will remove those parcels from the list to be loaded.
The displayed number of parcels remaining will be decreased.
Note that this feature depends on the log files being intact. If the logs have been deleted or moved, this feature will not work.
Please see the Change List for full details on changes in this release.