DISCLAIMERS:
1) I only have handy access to the VB code, so I can't offer actual plugins, but this will hopefully get you some answers.
2) I've anonymised/generified this code in NotePad++, and I have no easy way of testing this, so there are probably bugs, possibly even compile-time errors.
My folder watcher solution is split over several VB plugins.
Doing it this way allowed me to setup a 'base' class and have multiple different folder watchers; where the 'folder watcher' code is common, but the 'document processing' part is separate.
If you only have one importer, then you can put all of these into one big plugin, but that will make it less flexible.
If using multiple plugins, you need to set the priority of the plugins to match the dependencies, or you will get compilation errors. You may also need to restart JIWA in between adding individual plugins so that you can compile the following dependent plugins. If you have any challenges making plugins dependent on other plugins, just raise them back here.
The plugins (in priority/compilation order) are:
1) File Importer Interface & Base Class - As in a .NET Class interface
IAerpFileImporter, and a base class
AerpFileImporterBase for inheriting from. Specific implementations are instantiated by the generic folder watcher. (this will hopefully make more sense with some code).
In hindsight, the Interface is probably unnecessary, but it's something I've got into the habit of using, so I'll leave it here.
- Code: Select all
Imports System
Namespace AERP
Public Interface IAerpFileImporter
Property WatcherKey As String
Property WatchFolder As String
Property FileFilter As String
Property ProcessingFolder As String
Property FailedFolder As String
Property CompletedFolder As String
Property LogFolder As String
Function LoadDocument(fileName As String) As Integer
End Interface
Public MustInherit Class AerpFileImporterBase : Implements IAerpFileImporter
Public Property WatcherKey As String Implements IAerpEdiFileImporter.WatcherKey
Public Property WatchFolder As String Implements IAerpEdiFileImporter.WatchFolder
Public Property FileFilter As String Implements IAerpEdiFileImporter.FileFilter
Public Property ProcessingFolder As String Implements IAerpEdiFileImporter.ProcessingFolder
Public Property FailedFolder As String Implements IAerpEdiFileImporter.FailedFolder
Public Property CompletedFolder As String Implements IAerpEdiFileImporter.CompletedFolder
Public Property LogFolder As String Implements IAerpEdiFileImporter.LogFolder
Public MustOverride Function LoadDocument(fileName As String) As Integer Implements IAerpEdiFileImporter.LoadDocument
End Class
End Namespace
2) File Importer Implementations - one or more plugins with classes that implement
IAerpFileImporter (or Inherit from
AerpFileImporterBase)
This is a very bare example. Since every import will be very specific, you'd need to roll your own from this base.
In our case, simple XML transforms were never going to be enough, since the customer orders we were loading needed to be validated against business rules during the import. And we were dealing with different XML formats from several different large franchise operations. So we have five different importer classes.
This sample is based on one our imports that could have one or many documents, and had slightly different XML for each scenario. Mostly, I'd expect a file to be a bit more consistent, and have only one code path per file.
If using multiple plugins, this could also be done in C#, even if you retained the rest of my code in VB.
- Code: Select all
Imports System
Imports Microsoft.VisualBasic
Imports System.Xml
Imports System.Xml.Linq
Namespace AERP
Public Class AerpSampleImporter
Inherits AerpFileImporterBase
Public Overrides Function LoadDocument(fileName As String) As Integer
Dim _xmlDocument As XDocument = XDocument.Load(fileName)
Dim result As Integer = 0
Dim root = _xmlDocument.Root()
' very primitive sanity checks.
If root.Name = "Document" Then
' single Document file
result += ProcessOneDocument(fileName, root)
ElseIf root.Name = "Documents" Then
' multiple Documents in one file
For Each item As XElement In root.Elements("Document")
result += ProcessOneDocument(fileName, item)
Next
Else
' error
Return -1
End If
Return result
End Function
Public Function ProcessOneDocument(fileName As String, document As XElement) As Integer
Try
' TODO: process the document
Return 1
Catch ex As System.Exception
' TODO: Report Error
Return 0
End Try
End Function
End Class
End Namespace
3) The folder watcher class itself. This is the actual 'watcher'. One of its properties is an implementation of IAerpFileImporter, which is called to do the importing of an individual file.
It creates any missing folders (Processing, Failed, Complete). By default, sub-folders of the 'watched' folder, but can also be explicitly specified by setting the relevant properties on your IAerpFileImporter-derived class.
It also includes some logging to file, which I found more useful than logging to an event log, because it's easier to find it over the network.
- Code: Select all
Imports JiwaFinancials.Jiwa
Imports System
Imports System.IO
Imports AERP
Namespace AERP
Public Class AerpEdiFileWatcherEventArgs
Inherits System.EventArgs
Public Property Importer As IAerpFileImporter
Public Property FilePath As String
Public Property ItemsImported As Integer
Public Sub New()
End Sub
Public Sub New(importer As IAerpFileImporter, filePath As String, itemsImported As Integer)
Me.Importer = importer
Me.FilePath = filePath
Me.ItemsImported = itemsImported
End Sub
End Class
Public Class AerpFileWatcher
Public Plugin As JiwaApplication.Plugin.Plugin
Public Property WatcherKey As String
Public Property Importer As IAerpFileImporter
Private FileSystemWatcher As System.IO.FileSystemWatcher
Private _watcherLogFile As String
Public Event FileImport(sender As Object, e As AerpEdiFileWatcherEventArgs)
Public Sub New(Plugin As JiwaApplication.Plugin.Plugin, importer As IAerpFileImporter)
Me.Plugin = Plugin
Me.Importer = importer
End Sub
Public Sub Start()
If String.IsNullOrWhiteSpace(Importer.WatcherKey) Then Throw New InvalidOperationException("WatcherKey not specified")
If String.IsNullOrWhiteSpace(Importer.WatchFolder) Then Throw New InvalidOperationException("WatchFolder not specified")
If Not IO.Directory.Exists(Importer.WatchFolder) Then Throw New DirectoryNotFoundException(String.Format("Folder '{0}' not found.", Importer.WatchFolder))
If String.IsNullOrWhiteSpace(Importer.LogFolder) Then
Importer.LogFolder = IO.Path.Combine(Importer.WatchFolder, "Logs")
End If
If Not System.IO.Directory.Exists(Importer.LogFolder) Then
System.IO.Directory.CreateDirectory(Importer.LogFolder)
End If
_watcherLogFile = GetVersionedFileName(Path.Combine(Importer.LogFolder, "FileWatcher.Log"))
If String.IsNullOrWhiteSpace(Importer.ProcessingFolder) Then
Importer.ProcessingFolder = IO.Path.Combine(Importer.WatchFolder, "Processing")
LogMessage(_watcherLogFile, String.Format("AERP File Watcher using default folder: '{0}' for {1}", Importer.ProcessingFolder, Me.WatcherKey))
End If
If Not System.IO.Directory.Exists(Importer.ProcessingFolder) Then
System.IO.Directory.CreateDirectory(Importer.ProcessingFolder)
LogMessage(_watcherLogFile, String.Format("AERP File Watcher created folder: '{0}' for {1}", Importer.ProcessingFolder, Me.WatcherKey))
End If
If String.IsNullOrWhiteSpace(Importer.CompletedFolder) Then
Importer.CompletedFolder = IO.Path.Combine(Importer.WatchFolder, "Complete")
LogMessage(_watcherLogFile, String.Format("AERP File Watcher using default folder: '{0}' for {1}", Importer.CompletedFolder, Me.WatcherKey))
End If
If Not System.IO.Directory.Exists(Importer.CompletedFolder) Then
System.IO.Directory.CreateDirectory(Importer.CompletedFolder)
LogMessage(_watcherLogFile, String.Format("AERP File Watcher created folder: '{0}' for {1}", Importer.CompletedFolder, Me.WatcherKey))
End If
If String.IsNullOrWhiteSpace(Importer.FailedFolder) Then
Importer.FailedFolder = IO.Path.Combine(Importer.WatchFolder, "Failed")
LogMessage(_watcherLogFile, String.Format("AERP File Watcher using default folder: '{0}' for {1}", Importer.FailedFolder, Me.WatcherKey))
End If
If Not System.IO.Directory.Exists(Importer.FailedFolder) Then
System.IO.Directory.CreateDirectory(Importer.FailedFolder)
LogMessage(_watcherLogFile, String.Format("AERP File Watcher created folder: '{0}' for {1}", Importer.FailedFolder, Me.WatcherKey))
End If
ImportAllFilesInWatchFolder()
FileSystemWatcher = New FileSystemWatcher
FileSystemWatcher.Path = Importer.WatchFolder
FileSystemWatcher.NotifyFilter = (NotifyFilters.LastWrite Or NotifyFilters.FileName Or NotifyFilters.Size)
FileSystemWatcher.Filter = Importer.FileFilter
FileSystemWatcher.EnableRaisingEvents = True
AddHandler FileSystemWatcher.Created, AddressOf OnCreated
' because moving things into this folder from another folder on the same disk is a rename.
AddHandler FileSystemWatcher.Renamed, AddressOf OnRenamed
ImportAllFilesInWatchFolder()
LogMessage(_watcherLogFile, String.Format("AERP File Watcher started monitoring folder: '{0}' for {1}", Importer.WatchFolder, Me.WatcherKey))
End Sub
Public Sub ImportAllFilesInWatchFolder()
' Import any files already in the watch folder
For Each file As String In System.IO.Directory.EnumerateFiles(Importer.WatchFolder, Importer.FileFilter)
OnFileImport(file)
System.Threading.Thread.Sleep(200) ' wait a few ticks before doing the next one
Next
End Sub
Private Sub OnFileImport(ByVal FullPath As String)
If File.Exists(FullPath) Then
Try
' Wait until we can get exclusive access, or until we exceed the retry period.
WaitForFile(FullPath)
Catch ex As Exception
LogMessage(_watcherLogFile, String.Format("Failed to get exclusive access to: '{0}'.", FullPath))
Return
End Try
Dim processingFileName As String = Path.GetFileName(FullPath)
Dim processingFileNameAndPath As String = Path.Combine(Importer.ProcessingFolder, processingFileName)
Try
' prefer processing file to have original name, but don't want to block if there's already one there.
If File.Exists(processingFileNameAndPath) Then processingFileNameAndPath = GetVersionedFileName(processingFileNameAndPath)
' Move to processing folder
File.Move(FullPath, processingFileNameAndPath)
LogMessage(_watcherLogFile, String.Format("Moved File '{0}' to '{1}'", FullPath, processingFileNameAndPath))
' Hack to try and eliminate invoice number collisions on import.
Dim randomWait As Integer = Guid.NewGuid().ToByteArray(0)
System.Threading.Thread.Sleep(randomWait)
' Create EventArgs to hold results of the import
Dim fileWatcherEventArgs = New AerpEdiFileWatcherEventArgs(Importer, processingFileNameAndPath, 0)
' Raise the event to do the actual file import
RaiseEvent FileImport(Me, fileWatcherEventArgs)
Dim targetFileName As String
If fileWatcherEventArgs.ItemsImported > 0 Then
' Move to succeeded folder
targetFileName = GetVersionedFileName(Path.Combine(Importer.CompletedFolder, processingFileName))
Else
' Move to failed folder
targetFileName = GetVersionedFileName(Path.Combine(Importer.FailedFolder, processingFileName))
End If
File.Move(processingFileNameAndPath, targetFileName)
File.SetLastWriteTimeUtc(targetFileName, DateTime.UtcNow)
LogMessage(_watcherLogFile, String.Format("Moved File '{0}' to '{1}'", processingFileNameAndPath, targetFileName))
Catch ex As System.Exception
LogMessage(_watcherLogFile, String.Format("Failed to import: '{0}': {1}", processingFileNameAndPath, ex.ToString()))
End Try
End If
End Sub
Private Shared Sub WaitForFile(FullPath As String)
Dim retryCount As Integer = 0
Dim maxRetries As Integer = 5
Do While True
Try
Using fs As New FileStream(FullPath, FileMode.Open, FileAccess.ReadWrite, FileShare.None, 100)
fs.ReadByte()
Return
End Using
Catch ex As System.Exception
If retryCount > maxRetries Then
Throw New ApplicationException(String.Format("Unable to read {0}", FullPath))
Else
retryCount += 1
System.Threading.Thread.Sleep(500)
End If
End Try
Loop
End Sub
Private Sub OnCreated(source As Object, e As FileSystemEventArgs)
OnFileImport(e.FullPath)
End Sub
Private Sub OnRenamed(source As Object, e As RenamedEventArgs)
OnFileImport(e.FullPath)
End Sub
Public Sub StopWatching()
RemoveHandler FileSystemWatcher.Created, AddressOf OnCreated
RemoveHandler FileSystemWatcher.Renamed, AddressOf OnRenamed
End Sub
Public Sub LogMessage(logFileName As String, messageText As String)
Dim lfi = New FileInfo(logFileName)
Dim lf = lfi.AppendText()
lf.WriteLine("{0:yyyy-MM-dd HH:mm:ss.ff}: {1}", Date.Now, messageText)
lf.Close()
End Sub
End Class
End Namespace
4) The schedule plugin. Doesn't actually run the folder watcher on a schedule, since the folder-watcher is event-driven in the first place. When the service starts, it instantiates one or more watcher instances. You could even start the same one multiple times, each pointing at a different folder.
- Code: Select all
Imports JiwaFinancials.Jiwa
Imports System
Imports System.Collections.Generic
Imports AERP
Namespace AERP
Public Class ScheduledExecutionPlugin
Inherits System.MarshalByRefObject
Implements JiwaApplication.IJiwaScheduledExecutionPlugin
' XXXXXXXX put your log file somewhere useful
Private LogFileName As String = "I:\FTP\EdiServiceLogs\ServicePlugin.Log"
Private _plugin As JiwaApplication.Plugin.Plugin
Public Sub OnServiceStart(ByVal Plugin As JiwaApplication.Plugin.Plugin) Implements JiwaApplication.IJiwaScheduledExecutionPlugin.OnServiceStart
_plugin = Plugin
' Start watching the folders
LogMessage(LogFileName, "Starting")
' XXXXXXXX put your log file somewhere useful
AddWatcher(New AerpSampleImporter() With {.WatcherKey = "Folder1", .WatchFolder = GetAerpSystemSetting("Import Folder 1","I:\FTP\Folder1\Received"), .FileFilter = "*.XML"})
AddWatcher(New AerpSampleImporter() With {.WatcherKey = "Folder2", .WatchFolder = GetAerpSystemSetting("Import Folder 2","I:\FTP\Folder2\Received"), .FileFilter = "*.XML"})
End Sub
Public Sub OnServiceStopping(ByVal Plugin As JiwaApplication.Plugin.Plugin) Implements JiwaApplication.IJiwaScheduledExecutionPlugin.OnServiceStopping
LogMessage(LogFileName, "Stopping")
For Each i As ImportWatcher In _importerList.Values
If i.Watcher IsNot Nothing Then
i.Watcher.StopWatching()
End If
Next
End Sub
Public Sub Execute(ByVal Plugin As JiwaApplication.Plugin.Plugin, ByVal Schedule As JiwaApplication.Schedule.Schedule) Implements JiwaApplication.IJiwaScheduledExecutionPlugin.Execute
End Sub
Private Structure ImportWatcher
Public Key As String
Public Watcher As AerpFileWatcher
Public Importer As IAerpFileImporter
Public Sub New(key As String, watcher As AerpFileWatcher, importer As IAerpFileImporter)
Me.Key = key
Me.Watcher = watcher
Me.Importer = importer
End Sub
End Structure
Private Shared _importerList As New Dictionary(Of String, ImportWatcher)
''' Perform a file import.
Private Sub FileImport(sender As Object, e As AerpFileWatcherEventArgs)
LogMessage(LogFileName, String.Format("Importing {0} for {1}...", e.FilePath, e.Importer.WatcherKey))
Dim importer = _importerList(e.Importer.WatcherKey).Importer
e.ItemsImported = importer.LoadDocument(e.FilePath)
End Sub
''' Register the folder watcher.
Private Sub AddWatcher(importer As IAerpFileImporter)
LogMessage(LogFileName, String.Format("Adding watcher for {0}...", importer.WatchFolder, importer.WatcherKey))
Dim watcher As AerpFileWatcher = New AerpFileWatcher(_plugin, importer)
watcher.WatcherKey = importer.WatcherKey
If Not IO.Directory.Exists(importer.WatchFolder) Then IO.Directory.CreateDirectory(importer.WatchFolder)
_importerList.Add(importer.WatcherKey, New ImportWatcher(importer.WatcherKey, watcher, importer))
watcher.Start()
AddHandler watcher.FileImport, AddressOf FileImport
LogMessage(LogFileName, String.Format("Watching {0} for {1}...", importer.WatchFolder, importer.WatcherKey))
End Sub
Public Sub LogMessage(logFileName As String, messageText As String)
Dim lfi = New FileInfo(logFileName)
Dim lf = lfi.AppendText()
lf.WriteLine("{0:yyyy-MM-dd HH:mm:ss.ff}: {1}", Date.Now, messageText)
lf.Close()
End Sub
End Class
End Namespace