Stack

Home > Select Project > Analysis > Stack

Select a project in the initial screen of the WhaTap monitoring service, and then select Analysis > Stack under Project Menu. This feature allows you to check Top stack, Unique stack, and Active stack.

Note

The application that can use the stack analysis function is Java and Python, .NET.

WhaTap can use the thread stacks collected every 10 seconds (default value) to analyze the performance delay sections in the method level.

In the example stack, the top line is IL_STUB_ReversePInvoke.

DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64,Int64,Int64,Int32)

Top line indicates that the thread to be dumped is executing that method. It may be caught momentarily, but it appears on the stack as a percentage of the sum of the module processing times. By calculating the frequency of this top line method, you can determine the method-level performance. WhaTap calls the top line frequency statistics as Top stack.

You can analyze the frequencies of the methods that called the methods derived through the Top stack analysis.

The original active stack was difficult to identify in the top stack hierarchy analysis. Thus, WhaTap provides the Unique stack function to view Active stack after collecting the same stacks.

Top stack

Based on each step on the stack trace, it provides the call rates between steps as percentage. The backlog frequencies of the top step are calculated as a percentage and the results are sorted in descending order.

If each step is clicked, the upper step that calls the step is calculated and provided as percentage based on the call frequency.

The Top stack statistics are good to make decision with enough data. If the number of collected stacks is a prime number of less than 10, it is insufficient to have a statistical significance.

Top stack is useful to find tuning points that were difficult to recognize while tuning. The most frequent stacks can be determined to cause the most response delays in the application server. The rate that appears on the utmost left is how much it affects the application server's performance.

Even on a stable application server, frequently appearing stacks have the potential to cause performance degradation, so keep an eye on those classes.

If you click Top stack, the frequency of calls on the top stack can be checked. Because the call relationship of the Top stack is one-to-one, the data accuracy may decrease as the depth of the Top stack goes down. Use the information on the sub-depths for reference purposes and proceed with tuning.

To improve the application performance, the module with a high backlog rate in the top-level step must be checked for bottleneck. For modules with high backlog rates, even a small performance improvement can lead to significant enhancement throughout all applications.

whatap.util.ThreadUtil.sleep
// The call rate of jdbc.Control.exec is 40.02%.
jdbc.Control.exec 
// The call rate of jdbc.FakePreparedStatement.executeQuery is 68.06%.

The call rate of whatap.util.ThreadUtil.sleep ← jdbc.Control.exec ← jdbc.FakePreparedStatement.executeQuery does not indicate 40.02% * 68.06%. It is because the possibility of calling other modules exist in jdbc.Control.exec.

When determining the call rate through Top stack, do not multiply the call rates between steps to calculate the overall call relationship. Because the call rate of the top stack is the result of the call rates between steps of the data exposed on the stack trace, the total call rate derived with the call rates between steps can lead to a distorted result.

In the Top stack statistics, the history of collection counts and rate changes over time based on a specific period of time are provided.

• Percent

  • It displays changes in rate of the top stacks selected during the query period.
  • It is useful to identify the status of failure and to compare before and after the improvement.

• Count

  • The number of collected stacks is proportional to the number of active transactions.
  • If the collection amount increased in a specific section, it can be seen that there was a service delay or a sudden increase in inflow.

This can also be seen in the following diagram:

• Stack Chart

  The rates of individual top stacks appear on the chart.

Unique stack

As a result of calculation based on the hash value of the entire stack trace, the analysis information is provided with the percentage of the same call rates in all steps. Top stack provides the call rates between steps.

Unique stack provides the data based on accurate calls in the entire stack trace. Useful data for call relationship details is provided. For example, you can identify stack traces with higher backlog rates.

By reviewing the call step details, you can determine whether there is any abnormal module on the call path.

Active stack

Transactions in progress are called "active transactions." The stacks regularly dumped by active transactions are called Active stack.

Note

The WhaTap agent dumps Active stacks for active transactions every 10 seconds (optional) and sends them to the server.
active_stack_second=10

If Active stack is selected, you can check the collected active stacks on the chart. With statistical data for every 5 minutes, the Active transaction count is displayed with the bar chart and the TPS with the line chart.

If you click the bar, the active transaction information for the clicked period appears, and if you click the information, you can see the active stack of the transaction.

Note

For more information about active transaction, see the following.

Collecting the compact active stacks

Because the active stack regularly performs thread dumps, it can add lots of overheads to the agent if implemented incorrectly. WhaTap has various options to collect active stacks while minimizing the agent load.

Tip

In Sitemap > Thread List/Dump, from the thread list, the overhead can be determined by checking the CPU time of the WhaTap-ActiveStackDump thread.

Active Stack Example

.NET

DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64,Int64,Int64,Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr,IntPtr,IntPtr,Int32)
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr,IntPtr,IntPtr,Int32)
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr,System.Web.RequestNotificationStatus ByRef)
System.Threading.ThreadHelper.ThreadStart()
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,System.Threading.ContextCallback,System.Object)
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,System.Threading.ContextCallback,System.Object,Boolean)
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,System.Threading.ContextCallback,System.Object,Boolean)
System.Net.TimerThread.ThreadProc()
System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[],Int32,Boolean)
System.Threading.Thread.Sleep(Int32)

Optimized data collection

• Dump the stack only for the thread that is performing transactions.

• You can adjust the active stack dump time interval.

  active_stack_second=10

• It is limited to the maximum line of the active stack. It collects 50 lines from the top line by default.

  trace_active_callstack_depth=50

• Each hashed line of the active stack is collected. Text is collected only once.

• The maximum number of active stacks collected at a time is limited.

  active_stack_count=100

Active Stack for Background Threads

Basically, the active stack means the stack of the thread in which transactions are being processed. You may need to analyze the stack for some background threads as well. At this time, the active stack for the background thread can be obtained using the option. It is available since the version 1.6.2.

• If the value of async_stack_enabled is set to true, the function is enabled.

  async_stack_enabled=false

• The stack dump interval depends on the foreground active stack setting.

  active_stack_second=10

• When setting the target thread name, use * to specify the string pattern.

  async_thread_match=http*,abc*

  The stack dump target is determined by the thread name. Using comma (,), you can specify multiple matches.

• If the top method of the stack is a class or method registered in async_thread_parking, the system considers that the thread is parked and does not generate any dump.

  async_thread_parking_class=sun.misc.Unsafe
  async_thread_parking_method=park