Case Analysis of SWT Restart

Keywords: Android Java git

Strongly Recommend Articles: Welcome to Collection
Android Dry Goods Sharing

Read for five minutes, ten o'clock a day, and study with you for life. Here's Android, a programmer.

This article mainly introduces some knowledge points in Android development. By reading this article, you will gain the following contents:

1.MTK AEE Log Analysis Tool
2.AEE Log Analysis Process
3.system_server Card Death Case Analysis and Solution

This paper mainly focuses on the analysis and solution of Exception Type: system_server_watchdog, system_server card death.

MTK AEE Log Analysis Tool

MTK AEE Log acquisition method:

Focus on Wechat Public Number: Programmer Android
Reply to aee to get the tools for parsing restart db log.

II. AEE Log Analysis Process

1. Use AEE tools to parse dbg files.

2. Analysis of parsed exp_main and other files

The exp_main file records the log print information when a restart occurs.

Some restart exception Log information is as follows:

$** *** *** *** *** *** *** *** Fatal *** *** *** *** *** *** *** **$
Build Info: 'alps-mp-o1.mp7:alps-mp-o1.mp7:mt6765:S01,ACE/AS0618/AS0618:8.1.0/O11019/1548123508:user/release-keys'
Flavor Info: 'None'
Exception Log Time:[Thu Mar 14 14:00:03 CST 2019] [38684.729626]

Exception Class: SWT
Exception Type: system_server_watchdog

Current Executing Process: 
system_server


Trigger time:[2019-03-14 14:00:03.711844] pid:1029

Backtrace: 
Process: system_server
Subject: Blocked in handler on ActivityManager (ActivityManager)
Build: ACE/AS0618/AS0618:8.1.0/O11019/1548123508:user/release-keys

exp_main file parsing
From the overview of Log general information at the beginning, we can see the time and type of reboot, the process of triggering reboot and the place of PID and system Blocked.

Restart Log with exp_main and trace analysis
Log analysis is as follows:

// 1. Reboot trigger time and PID 
Trigger time:[2019-03-14 14:00:03.711844] pid:1029
// 2.Blocked process
Backtrace: 
Process: system_server
Subject: Blocked in handler on ActivityManager (ActivityManager)
// 3. View Trace Information Based on PID
----- pid 1029 at 2019-03-14 13:59:58 -----
Cmd line: system_server

... ...
// 4. View Blocked's information based on Backtrace
"ActivityManager" prio=5 tid=11 Blocked

... ...
// 5.tid=11 Waiting for tid=106 Threads to Release Resource Locks
  - waiting to lock <0x090691f3> (a android.util.ArrayMap) held by thread 106  
... ...

// 6. Check tid = 106 locks
"backup" prio=5 tid=106 Waiting
... ... 

  at java.lang.Object.wait(Native method)
  - waiting on <0x06a44c62> (a com.android.server.am.ContentProviderRecord)
// 7. Deadlock Locks
  at com.android.server.am.ActivityManagerService.getContentProviderImpl(ActivityManagerService.java:12127)
  - locked <0x06a44c62> (a com.android.server.am.ContentProviderRecord)
  
  ...  ...

"Binder:1029_8" prio=5 tid=107 Blocked
// 8. tid=107 in log is aggravated by 106 Blocked's further problem
  at com.android.server.notification.RankingHelper.getRecord(RankingHelper.java:258)
  - waiting to lock <0x090691f3> (a android.util.ArrayMap) held by thread 106
  
$** *** *** *** *** *** *** *** Fatal *** *** *** *** *** *** *** **$

Complete log s are available on the Public Number

3.system_server Card Death Case Analysis and Solution

Log to find the cause of death, solve this problem.
The ActivityManagerService class needs to be modified.

The code path is as follows: alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java

Solutions:
Avoid the provider lock triggering the SWT restart mechanism of MTK 60s for a long time. Set a timeout time and release the resource lock when it exceeds the time to avoid this problem.

The diff modification scheme is as follows:

--- a/[alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c)

+++ b/[alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444)

@@ [-545,7](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c#l545) [+545,9](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444#l545) @@ public class ActivityManagerService extends IActivityManager.Stub

     // How long we wait for an attached process to publish its content providers

     // before we decide it must be hung.

     static final int CONTENT_PROVIDER_PUBLISH_TIMEOUT = 10*1000;

-

+    // How long we wait for provider to be notify before we decide it may be hung.

+    static final int CONTENT_PROVIDER_WAIT_TIMEOUT = 20*1000;

+       

     // How long we wait for a launched process to attach to the activity manager

     // before we decide it's never going to come up for real, when the process was

     // started with a wrapper for instrumentation (such as Valgrind) because it

@@ [-1745,6](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c#l1745) [+1747,7](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444#l1747) @@ public class ActivityManagerService extends IActivityManager.Stub

     static final int PUSH_TEMP_WHITELIST_UI_MSG = 68;

     static final int SERVICE_FOREGROUND_CRASH_MSG = 69;

     static final int DISPATCH_OOM_ADJ_OBSERVER_MSG = 70;

+        static final int CONTENT_PROVIDER_WAIT_TIMEOUT_MSG = 71;

     static final int START_USER_SWITCH_FG_MSG = 712;

     static final int NOTIFY_VR_KEYGUARD_MSG = 74;

@@ [-2108,6](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c#l2108) [+2111,12](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444#l2111) @@ public class ActivityManagerService extends IActivityManager.Stub

                 synchronized (ActivityManagerService.this) {

                     mActivityStarter.doPendingActivityLaunchesLocked(true);

                 }

+            } break;

+                       case CONTENT_PROVIDER_WAIT_TIMEOUT_MSG: {

+                ContentProviderRecord cpr = (ContentProviderRecord)msg.obj;

+                synchronized (ActivityManagerService.this) {

+                    processContentProviderWaitTimedOutLocked(cpr);

+                }

             } break;

             case KILL_APPLICATION_MSG: {

                 synchronized (ActivityManagerService.this) {

@@ [-7029,7](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c#l7029) [+7038,31](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444#l7038) @@ public class ActivityManagerService extends IActivityManager.Stub

         cleanupAppInLaunchingProvidersLocked(app, true);

         removeProcessLocked(app, false, true, "timeout publishing content providers");

     }

+       

+    @GuardedBy("this")

+    private final void processContentProviderWaitTimedOutLocked(ContentProviderRecord cpr) {

+        try {

+            if (mLaunchingProviders.contains(cpr)) {

+                if (DEBUG_MU) Slog.v(TAG_MU,

+                    "Remove from mLaunchingProviders, " + cpr

+                    + " launchingApp=" + cpr.launchingApp);

+                mLaunchingProviders.remove(cpr);

+            }

+            if (DEBUG_MU) Slog.v(TAG_MU,

+                "RemoveMessages CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, " + cpr

+                + " launchingApp=" + cpr.launchingApp);

+            mHandler.removeMessages(CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, cpr);

+            synchronized (cpr) {

+                cpr.notifyAll();

+                cpr.launchingApp = null;

+            }

+        } catch (Exception e) {

+            if (DEBUG_MU) Slog.v(TAG_MU,

+                "processContentProviderWaitTimedOutLocked exception, " + e);

+        }

+    }

+       

     private final void processStartTimedOutLocked(ProcessRecord app) {

         final int pid = app.pid;

         boolean gone = false;

@@ [-12124,11](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=d5e2e1411f3698a829e997d402c7482ec277fa8c;hb=d5e2e1411f3698a829e997d402c7482ec277fa8c#l12124) [+12157,33](http://192.168.11.104/gitweb/?p=alps-mp-o1.mp1-V1.git;a=blob;f=alps/frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java;h=03208f78a2bf3167b4f0790019326e3939cc5444;hb=03208f78a2bf3167b4f0790019326e3939cc5444#l12157) @@ public class ActivityManagerService extends IActivityManager.Stub

                     if (conn != null) {

                         conn.waiting = true;

                     }

+                                       // add 20s wait timeout,avoid 

+                    if (!mHandler.hasMessages(CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, cpr)) {

+                        if (DEBUG_MU) Slog.v(TAG_MU,

+                            "SendMessageDelayed CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, " + cpr

+                            + " launchingApp=" + cpr.launchingApp);

+                        Message msg = mHandler.obtainMessage(CONTENT_PROVIDER_WAIT_TIMEOUT_MSG);

+                        msg.obj = cpr;

+                        mHandler.sendMessageDelayed(msg, CONTENT_PROVIDER_WAIT_TIMEOUT);

+                    } else {

+                        if (DEBUG_MU) Slog.v(TAG_MU,

+                            "There is another waiting to start provider " + cpr

+                            + " launchingApp=" + cpr.launchingApp

+                            + ", not send CONTENT_PROVIDER_WAIT_TIMEOUT_MSG again");

+                    }

+                                       

                     cpr.wait();

                 } catch (InterruptedException ex) {

                 } finally {

                     if (conn != null) {

                         conn.waiting = false;

+                    }

+                                       // remove wait time out message

+                    if (mHandler.hasMessages(CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, cpr)) {

+                        if (DEBUG_MU) Slog.v(TAG_MU,

+                            "After wait removeMessages CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, "

+                            + cpr + " launchingApp=" + cpr.launchingApp);

+                            mHandler.removeMessages(CONTENT_PROVIDER_WAIT_TIMEOUT_MSG, cpr);

                     }

                 }

             }

So far, this article is over. If there are any mistakes, you are welcome to make suggestions and corrections. At the same time look forward to your attention, thank you for reading, thank you!

Posted by discomatt on Tue, 03 Sep 2019 02:23:20 -0700