User:Giuseppe Lavagetto/How to debug HHVM

We have one HHVM appserver with high load and some thread constantly using up 100% of the CPU, and we want to figure out what's going on.

Here is a recap of my standard procedure, for posterity and for stopping exercising my muscle memory

• Depool the appserver, some of the things we will do are quite disruptive, and also we need to weed out effects of incoming traffic from hanging threads.
• Identify all hanging threads
• Attach to the master process

gdb -p $hhvm_main_pid

• Check out the thread numbers and corresponding PIDs, enter the thread stack, and dump the request headers

(gdb) info threads 
... # Find the thread n/system PID correspondence
(gdb) thread <N>
(gdb) set print elements 0
(gdb) p (*(HPHP::FastCGITransport*)g_context.m_node.m_p.m_transport).m_requestParams
# This returns a hash containing all request parameters

• Check what's happening in PHP terms using perf record/perf report on the thread

  # record call graph attaching to pid PID
  perf record -g -p $PID -F 1 sleep 5
  chown root /tmp/perf*.map
  perf report
  

  this could give you an indication of where - if by any chance - in the php code we're spending most time
• If you still don't have enough information, you can gather more from doing a full core dump and attaching gdb to it

  hhvm-dump-debug --full
  # this will create the .bt file, with the full gdb backtrace, and a core file that can be examined later.