CPU spike/performance issue

[perry]

Hi All,
Jiwa 7.2.1 with latest SR.

Does Jiwa.exe do something every 300 seconds that spikes CPU usage for couple seconds? This is when Jiwa is completely idle and doing nothing.
Below are my testing from different setup, e.g. local SQL demo DB, Azure SQL customer DB, Azure SQL demo DB, customer Server+Azure DB, they all behave the same.

Can you point the to the right direction on what exactly Jiwa application maybe doing every 5mins? checking


The customer has a remote desktop server hosted on Azure. Each instance has about 7-10 Jiwa users (I can only assume most users only have 1 Jiwa client running).
The CPU usage reaches 100% (across all users) every few mins and sometime last for more than a min. According to the end user, the Jiwa performance is unstable.

Few things I dont understand
- My observation is, this spike finishes within 5s but it is different to what they see on server (a min+).
- this could be process power is not enough to handle the spike from all users at once, so 1 min instead of 5 seconds
- why all clients start spiking at the same time?? SQL connections are pooled and the pool is being cleared?
- Am I on the right track?

[Mike.Sheen]

Does Jiwa.exe do something every 300 seconds that spikes CPU usage for couple seconds?

The only timers we have is in the Search Window - when a search window is displayed we start a time on key press, and when that lapses (user setting defines the duration - default is 1750ms) if no keys were pressed we generate and execute a SQL query. If you're not displaying a search window then there is no timer involvement.

The only other timers we have is in the Jiwa Plugin Scheduler service.

What are your CPU usage graphs of? Have you somehow captured only the Jiwa.exe CPU usage? How would I generate the same on my local machine here to compare?

[SBarnes]

You are obviously looking for something that fires on a timer or even something that is hitting Azure SQL every five minutes so some simple things I would suggest you look at:

Are todos enabled and what is the polling if they are?
Is the REST API and web hooks in use?
Looking at Azure SQL is there a spike in usage for the same period?
Is there any customisations that rely on timers?
Is the plugin scheduler in use?

Things you could look at
Lower the retries on transient failure value and see if the client loses its connection its retrying to connect can look like freezing
Make sure that the database is being maintained and reindexed, Azure SQL does not do this on its own and its amazing what some maintenance will do for performance
Try turning the Jiwa services off temporarily to see if that isolates the issue is there
Identify anything outside of the Jiwa world that could be connecting to the database and temporarily disable them if you can
If this is related to a long running process in Jiwa on the manger object there are events for when they are added etc you could try logging to those to see if something is going on there.
Is the database in a pool and something else is using up the resources of the pool?

[perry]

I built a console app to log CPU % per PID, so yes, the CPU % is only Jiwa.exe. (code below) or you can download from
https://lonicera-my.sharepoint.com/personal/perry_ma_lonicera_com_au/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fperry%5Fma%5Flonicera%5Fcom%5Fau%2FDocuments%2FShared%20with%20Everyone%2FDebug%2Ezip&parent=%2Fpersonal%2Fperry%5Fma%5Flonicera%5Fcom%5Fau%2FDocuments%2FShared%20with%20Everyone&ga=1


I do below before I start the console
1. start Jiwa
2. open sales order, sales quote and inventory maintenance form (customer automatically start these forms)
3. wait for forms finish loading then start tracking from console app.

I dont have search dialog and no services nor API service running on my PC.

Code: Select all

Imports System.Text
Imports System.Threading

Module Module1

    'Public logger As Logger = NLog.LogManager.GetCurrentClassLogger()

    Sub Main()


        Console.WriteLine("Please enter the PID: ")
        Dim processName As String = Console.ReadLine()

        processName = GetProcessInstanceName(processName)

        Dim myAppCpu As PerformanceCounter = New PerformanceCounter("Process", "% Processor Time", processName, True)
        Console.WriteLine("Press the any key to stop..." & vbLf)

        Dim t1 As DateTime = Now



        While Not Console.KeyAvailable
            Dim pct As Double = myAppCpu.NextValue()
            pct = Math.Round(pct, 2)
            'logger.Trace($"{processName},{DateDiff(DateInterval.Second, t1, Now)},{pct}")


            Console.WriteLine($"{processName},{DateDiff(DateInterval.Second, t1, Now)},{pct}")

            If pct >= 10 Then
                Console.Beep(5000, 3000)
            End If

            Thread.Sleep(3000)
        End While
    End Sub


    Public Function GetProcessInstanceName(ByVal PID As Integer) As String
        Dim processCategory As PerformanceCounterCategory = New PerformanceCounterCategory("Process")
        Dim instanceNames As String() = processCategory.GetInstanceNames()

        For Each name As String In instanceNames

            If name.StartsWith("Jiwa") Then

                Using processIdCounter As PerformanceCounter = New PerformanceCounter("Process", "ID Process", name, True)

                    If PID = CInt(processIdCounter.RawValue) Then

                        Return name
                    End If
                End Using
            End If
        Next

        Return "asdfasdfasdfasdf"
    End Function


End Module


[SBarnes]

Try disabling every plugin and see if the same thing happens i.e. does Jiwa do it in the raw.

If it doesn't then try adding the plugins back one at a time.

[perry]

Are todos enabled and what is the polling if they are?
Is the REST API and web hooks in use?
Looking at Azure SQL is there a spike in usage for the same period?
Is there any customisations that rely on timers?
Is the plugin scheduler in use?

- no todos,
- API service is disabled on my PC
- I haven't done any DB trace together with my idle session. I will get too much noise from customer DB but since I can replicate this on my local server, thats what I'm gonna do next.
- no timer in any plugins
- plugin scheduler service is disabled

Lower the retries on transient failure value and see if the client loses its connection its retrying to connect can look like freezing
Make sure that the database is being maintained and reindexed, Azure SQL does not do this on its own and its amazing what some maintenance will do for performance
Try turning the Jiwa services off temporarily to see if that isolates the issue is there
Identify anything outside of the Jiwa world that could be connecting to the database and temporarily disable them if you can
If this is related to a long running process in Jiwa on the manger object there are events for when they are added etc you could try logging to those to see if something is going on there.
Is the database in a pool and something else is using up the resources of the pool?

I dont have prove yet but I dont think the client CPU spike causes (or caused by) heavy DB activities. Customer's IT did monitor DB activity and CPU% at the same time and cannot see DB spike during CPU spike.

I haven't checked the pooling yet, it was just a thought.

[perry]

Try disabling every plugin and see if the same thing happens i.e. does Jiwa do it in the raw.

If it doesn't then try adding the plugins back one at a time.

Yes, will do that as well... the demo database I tested with doesn't have much plugins in there but will make sure all are disabled before my next round of testing...

[Mike.Sheen]

or you can download from
https://lonicera-my.sharepoint.com/personal/perry_ma_lonicera_com_au/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fperry%5Fma%5Flonicera%5Fcom%5Fau%2FDocuments%2FShared%20with%20Everyone%2FDebug%2Ezip&parent=%2Fpersonal%2Fperry%5Fma%5Flonicera%5Fcom%5Fau%2FDocuments%2FShared%20with%20Everyone&ga=1

I'll make a console app from your code, but just so you know that link only works for users in your tenant.

Screenshot 2022-07-05 155639.png (25.81 KiB) Viewed 3634 times

[Mike.Sheen]

I can repro the shrilling scream from your console app.

I can make it occur by simply moving the mouse around the inventory maintenance form - doing so and your app logs 29.x% CPU usage. In fact if any form is open, just moving the mouse will see that figure reported if you move the mouse around constantly for the 3 second sample window.

Even printing a report to screen and moving the mouse around on that will trigger it.

However, task manager at the same time shows Jiwa.exe never going above 1%.

Something funny is going on. I'll need to look into the .NET PerformanceCounter class you're using to see why it's reporting 30% but Task Manager says 1%.

[Mike.Sheen]

Opening the Inventory Maintenance form sees a 77% CPU usage claim:

Screenshot 2022-07-05 164144.png (12.7 KiB) Viewed 3634 times

Which is kinda impossible on my 3950X with 16 physical cores.

The metric you're capturing is Process: % Processor Time - from the docs:

Process : % Processor Time. Each process will show up as an instance when selecting this counter. This counter will break down how much processor time each process is taking on the CPU. Don't forget to exclude the Idle and the Total counts when looking at all of the instances.

The statement "This counter will break down how much processor time each process is taking on the CPU" indicates something has to be measured wrong here - Jiwa is single threaded so on a 16 core machine it can't possibly be using 77% or even 30% at a time.

The investigation continues...