File Watcher polling interval

[DannyC]

Hi,

In the CONFIG file for JiwaPluginSchedulerService.exe there is a polling interval which I have set for 60000 milliseconds. So every 1 minute, it will poll the Watch folder(s).
What is actually occurring is that the service picks up the files in Watch almost instantly, the service doesn't seem to be polling according to the PollInterval tag.

Is there another trick to get the File Watcher to poll the folder at a defined interval?

Cheers

Danny

[Mike.Sheen]

Hi Danny,

That's not how the File Watcher plugin we provide works.

The poll interval set in the JiwaPluginSchedulerService.exe.config applies to the frequency the service looks for schedules to be run. When due to run, it will invoke the Execute method in the ScheduledExecutionPlugin class of each plugin which has a schedule attached that is due or overdue to be run.

The standard File Watcher Plugin in Jiwa doesn't use poll intervals - you'll note the Execute method in there is empty.

So how does it know when a file appears in a watch folder? The OnServiceStart method of the File Watcher Plugin tells the operating system that it wants to be notified as soon as a file appears - no polling - this is achieved by the following code:

Code: Select all

FileSystemWatcher = New FileSystemWatcher
FileSystemWatcher.Path = Me.WatchFolder
FileSystemWatcher.NotifyFilter = (NotifyFilters.LastWrite Or NotifyFilters.FileName)
FileSystemWatcher.Filter = "*.*"
FileSystemWatcher.EnableRaisingEvents = True
   
AddHandler FileSystemWatcher.Created, AddressOf OnCreated

This tells the operating system (via the .Net framework) to invoke the OnCreated method of the plugin whenever a file is created in the watch folder.

I can see how you were misled - the "JiwaPluginSchedulerService" is being used to start watching a folder on service start, and not using any scheduling - we opted to do this to simplify things and not introduce another dedicated service to watch folders, when we could just use the JiwaPluginSchedulerService.

If you really want to only perform actions on files in a folder based on a schedule, then you can do that by moving some code to the Execute method of the ScheduledExecutionPlugin class from the OnServiceStart method. You'll need to remove the code which I showed above.

It'll take a little bit of refactoring, but you'll mostly be deleting and moving chunks of code - so not too much effort involved to do what you want.

Mike

[DannyC]

Understood Mike. Thanks for the comprehensive answer.

The problem I am facing is that Jiwa is picking up the file in Watch a few milliseconds before the software which is writing the file has a chance to finish populating & closing the file write. The file ends up being zero bytes & nothing gets imported to Jiwa.
I will fiddle around with the plugin based on your recommendations. I am thinking an interval of about 30 seconds would suffice.

Cheers

Danny

[Mike.Sheen]

The problem I am facing is that Jiwa is picking up the file in Watch a few milliseconds before the software which is writing the file has a chance to finish populating & closing the file write. The file ends up being zero bytes & nothing gets imported to Jiwa.

Easiest way to solve this is to do a Thread.Sleep(x) in the file watcher in the OnCreated method before it does anything.

[DannyC]

I reckon this may have done the trick!

[Mike.Sheen]

I reckon this may have done the trick!

That probably means then the protections we put in place to prevent against exactly your problem are not working well enough - or at all.

We actually do a System.Threading.Thread.Sleep(500) up to 5 times when trying to move a newly detected file to the pending folder:

Code: Select all

Private Sub OnFileImport(ByVal FullPath As String)
   If System.IO.File.Exists(FullPath) Then
      Dim pendingFileNameAndPath As String = Path.Combine(Me.PendingFolder, System.IO.Path.GetFileName(FullPath)) & "." & now.ToString("yyyy-MM-ddTHmmss.FFF")
         
      Try
         ' Wait until we can get exclusive access, or until we exceed the retry period.
         Dim retryCount As Integer = 0
         Dim maxRetries As Integer = 5               
         
         Do While True
            Try
               Using fs As New FileStream(FullPath, FileMode.Open, FileAccess.ReadWrite, FileShare.None, 100)
                  fs.ReadByte
                  Exit Do
               End Using
            Catch ex As System.Exception
               If retryCount > maxRetries Then
                  Exit Do
               Else
                  retryCount += 1
                  System.Threading.Thread.Sleep(500)
               End If
            End Try
         Loop
            
         ' Move to pending folder            
         System.IO.File.Move(Path.Combine(Me.WatchFolder, System.IO.Path.GetFileName(FullPath)), pendingFileNameAndPath)

What sleep interval did you set before you achieved successful results? Given that we already had up to 2.5 seconds wait time (500ms * 5 retries), then your other application writing the file is taking an awfully long time to finish writing, or our check for exclusive access is flawed.

Also, if our cumulative wait / retry period wasn't long enough we should see an exception when trying to move the file to the pending folder - not a 0 byte file as you describe. Puzzling.

Mike

[DannyC]

What sleep interval did you set before you achieved successful results?

10,000 milliseconds.
But I didn't try any other wait time, that was my first shot and so far I haven't seen any zero byte file come through. 10 seconds is plenty long enough I figured.