/DataStore introduction/
Jetpack DataStore is an improved new data storage solution that allows the use of protocol buffers to store key value pairs or typed objects.
DataStore stores data in an asynchronous and consistent transaction mode, which overcomes some shortcomings of shared preferences (hereinafter collectively referred to as SP).
DataStore is implemented based on Kotlin collaboration and Flow, and can migrate SP data to replace sp.
DataStore provides two different implementations: Preferences DataStore and proto DataStore, where Preferences DataStore is used to store key value pairs; Proto DataStore is used to store typed objects. Corresponding usage examples will be given later.
/SharedPreferences disadvantages/
Before the emergence of DataStore, the storage mode we used most was undoubtedly SP, which was simple, easy to use and widely praised. However, google defines SP as lightweight storage. If less data is stored, there is no problem in use. When more data needs to be stored, SP may cause the following problems:
1. When the SP loads data for the first time, it needs to load it in full. When the amount of data is large, it may block the UI thread and cause a jam
2. SP read / write files are not type safe, and there is no mechanism to send error signals, and there is a lack of transactional API
3. The commit() / apply() operation may cause ANR problems:
commit() is a synchronous submission, which will directly execute IO operations in the UI main thread. When the write operation takes a long time, the UI thread will be blocked, resulting in ANR; Although apply() is committed asynchronously, when writing to the disk asynchronously, if the onStop() method in Activity / Service is executed, it will also wait for the SP to write synchronously. If the waiting time is too long, it will also cause ANR problems. Let's expand on apply():
Finally, the apply() function is executed in SharedPreferencesImpl#EditorImpl.java:
public final CountDownLatch writtenToDiskLatch = new CountDownLatch(1);
public void apply() {
final MemoryCommitResult mcr = commitToMemory();
final Runnable awaitCommit = new Runnable() {
public void run() {
try {
mcr.writtenToDiskLatch.await();
} catch (InterruptedException ignored) {
}
}
};
//Before 8.0
QueuedWork.add(awaitCommit);
//After 8.0
QueuedWork.addFinisher(awaitCommit);
//Perform disk write operations asynchronously
SharedPreferencesImpl.this.enqueueDiskWrite(mcr, postWriteRunnable);
//... others
}
Construct a Runnable task named awaitCommit and add it to the QueuedWork. The CountDownLatch.await() method is directly called inside the task, that is, the wait operation is directly executed in the UI thread. It depends on when the task is executed in the QueuedWork.
There are differences in the implementation of QueuedWork class between Android versions above 8.0 and below 8.0:
QueuedWork.java before 8.0:
public class QueuedWork {
private static final ConcurrentLinkedQueue<Runnable> sPendingWorkFinishers =
new ConcurrentLinkedQueue<Runnable>();
public static void add(Runnable finisher) {
sPendingWorkFinishers.add(finisher);
}
public static void waitToFinish() {
Runnable toFinish;
// Take the task from the queue: if the task is empty, the loop will jump out and the UI thread can continue to execute;
//On the contrary, if the task is not empty, take out the task and execute it. The actually executed CountDownLatch.await(), that is, the UI thread will block the wait
while ((toFinish = sPendingWorkFinishers.poll()) != null) {
toFinish.run();
}
}
//... others
}
QueuedWork.java after 8.0:
public class QueuedWork {
private static final LinkedList<Runnable> sFinishers = new LinkedList<>();
public static void waitToFinish() {
Handler handler = getHandler();
StrictMode.ThreadPolicy oldPolicy = StrictMode.allowThreadDiskWrites();
try {
//After 8.0 optimization, it will actively try to write to the disk
processPendingWork();
} finally {
StrictMode.setThreadPolicy(oldPolicy);
}
try {
while (true) {
Runnable finisher;
synchronized (sLock) {
//Remove task from queue
finisher = sFinishers.poll();
}
//If the task is empty, the loop will jump out and the UI thread can continue to execute
if (finisher == null) {
break;
}
//If the task is not empty, execute CountDownLatch.await(), that is, the UI thread will block waiting
finisher.run();
}
} finally {
sCanDelay = true;
}
}
}
It can be seen that no matter before or after 8.0, waitto finish() will try to get the task from the Runnable task queue. If any, it will be taken out and executed directly to see where waitto finish() is called:
ActivityThread.java
private void handleStopActivity(IBinder token, boolean show, int configChanges, int seq) {
//... others
QueuedWork.waitToFinish();
}
private void handleStopService(IBinder token) {
//... others
QueuedWork.waitToFinish();
}
Some code details are omitted. It can be seen that the waitto finish() method will be called in the handleStopActivity and handleStopService methods in the ActivityThread, that is, in the onStop() of the Activity and the onStop() of the Service, the execution will continue after the write task is completed.
Therefore, although apply() writes to the disk asynchronously, if it executes onStop() of Activity/Service at this time, it may still block the UI thread and cause ANR.
Voice over: the ANR problem caused by the use of SP can be optimized by some Hook means, such as byte publishing Today's headline ANR Optimization Practice Series - * * * * bid farewell to SharedPreference waiting( https://mp.weixin.qq.com/s/kfF83UmsGM5w43rDCH544g ). The SPS used in our project are also optimized according to this method, and the optimized effect is still significant. Therefore, the project has not migrated the SPS (such as migrating to MMKV or DataStore), but it does not affect us to learn new storage postures.
/DataStore usage/
DataStore benefits:
- DataStore processes data updates based on transaction mode.
- DataStore accesses data based on Kotlin Flow. By default, it operates asynchronously in Dispatchers.IO to avoid blocking UI threads, and can handle exceptions when reading data.
- Methods of applying () and commit() to store data are not provided.
- Support SP one-time automatic migration to DataStore.
Preferences DataStore
Add dependency
implementation 'androidx.datastore:datastore-preferences:1.0.0'
Building Preferences DataStore
val Context.bookDataStorePf: DataStore<Preferences> by preferencesDataStore(
// File name
name = "pf_datastore")
Through the above code, we have successfully created Preferences DataStore, where preferencesDataStore() is a top-level function, including the following parameters:
- Name: the name of the file that created the Preferences DataStore.
- corruptionHandler: if the DataStore fails to deserialize the data when trying to read the data, it will throw an android.DataStore.core.corruptionexception, and the corruptionHandler will be executed at this time.
- produceMigrations: SP generates and migrates to Preferences DataStore. ApplicationContext is passed as a parameter to these callbacks, and the migration runs before any access to the data.
- Scope: scope of the collaboration. The default IO operation is executed in the Dispatchers.IO thread.
After the above code is executed, a file named pf will be created under / data/data / project package name / files /_ The datastore files are as follows:
You can see that the suffix is not xml, but. preferences_pb. Here's a point to note: you can't write the above initialization code into the Activity. Otherwise, when you repeatedly enter the Activity and use the Preferences DataStore, you will try to create a. Preferences with the same name_ Pb file, because it has been created once before, when an attempt to create a file with the same name is detected, an exception will be thrown directly: java.lang.IllegalStateException: There are multiple DataStores active for the same file: xxx.You should either maintain your DataStore as a singleton or confirm that there is no two DataStore's active on the same file (by confirming that the scope is cancelled). The error reporting class is under Android X / datastore / core / singleprocessdatastore of Android x.datastore: datastore core: 1.0.0:
internal val activeFiles = mutableSetOf<String>()
file.absolutePath.let {
synchronized(activeFilesLock) {
check(!activeFiles.contains(it)) {
"There are multiple DataStores active for the same file: $file. You should " +
"either maintain your DataStore as a singleton or confirm that there is " +
"no two DataStore's active on the same file (by confirming that the scope" +
" is cancelled)."
}
activeFiles.add(it)
}
}
Where file is the file generated through File(applicationContext.filesDir, "datastore/$fileName"), that is, the file address of Preferences DataStore to be operated in the disk. activeFiles saves the generated file path in memory. If it is judged that the file already exists in activeFiles, throw an exception directly, that is, duplicate creation is not allowed.
-
Save data
First declare an entity class BookModel:
data class BookModel(
var name: String = "",
var price: Float = 0f,
var type: Type = Type.ENGLISH
)
enum class Type {
MATH,
CHINESE,
ENGLISH
}
Perform storage operations in BookRepo.kt:
const val KEY_BOOK_NAME = "key_book_name"
const val KEY_BOOK_PRICE = "key_book_price"
const val KEY_BOOK_TYPE = "key_book_type"
//Preferences. Key < T > type
object PreferenceKeys {
val P_KEY_BOOK_NAME = stringPreferencesKey(KEY_BOOK_NAME)
val P_KEY_BOOK_PRICE = floatPreferencesKey(KEY_BOOK_PRICE)
val P_KEY_BOOK_TYPE = stringPreferencesKey(KEY_BOOK_TYPE)
}
/**
* Preferences DataStore Save data
*/
suspend fun saveBookPf(book: BookModel) {
context.bookDataStorePf.edit { preferences ->
preferences[PreferenceKeys.P_KEY_BOOK_NAME] = book.name
preferences[PreferenceKeys.P_KEY_BOOK_PRICE] = book.price
preferences[PreferenceKeys.P_KEY_BOOK_TYPE] = book.type.name
}
}
In Activity:
lifecycleScope.launch {
val book = BookModel(
name = "Hello Preferences DataStore",
price = (1..10).random().toFloat(), //Here, the price will change every click. In order to show that the UI layer can monitor data changes at any time
type = Type.MATH )
mBookRepo.savePfData(book)
}
Store through the bookdatastorepf.edit (transform: suspend (mutablepreferences) - > unit) suspend function, which accepts the transform block and can update the status in the DataStore in a transactional manner.
-
Fetch data
/**
* Preferences DataStore When fetching data, you can perform a series of processing on Flow data
*/
val bookPfFlow: Flow<BookModel> = context.bookDataStorePf.data.catch { exception ->
// dataStore.data throws an IOException when an error is encountered when reading data
if (exception is IOException) {
emit(emptyPreferences())
} else {
throw exception
}
}.map { preferences ->
//The corresponding key is preferences. Key < T >
val bookName = preferences[PreferenceKeys.P_KEY_BOOK_NAME] ?: ""
val bookPrice = preferences[PreferenceKeys.P_KEY_BOOK_PRICE] ?: 0f
val bookType = Type.valueOf(preferences[PreferenceKeys.P_KEY_BOOK_TYPE] ?: Type.MATH.name)
return@map BookModel(bookName, bookPrice, bookType)
}
In Activity:
lifecycleScope.launch {
mBookViewModel.bookPfFlow.collect {
mTvContentPf.text = it.toString()
}
}
Flow < bookmodel > is returned through bookDataStorePf.data, so a series of data processing can be carried out through flow in the future. When reading data from a file, if an error occurs, the system will throw IOExceptions. You can use the catch() operator before map() and issue emptyPreferences() when the exception thrown is IOException. If there are other types of exceptions, re throw the exception.
Note: when accessing data in Preferences DataStore, the Key is of preferences.Key < T > type, and t can only store Int, Long, Float, Double, Boolean, String and set < String > types. This is limited to the getValueProto() method of Android / datastore / preferences / core / preferencesserializer class participating in sequencing:
private fun getValueProto(value: Any): Value {
return when (value) {
is Boolean -> Value.newBuilder().setBoolean(value).build()
is Float -> Value.newBuilder().setFloat(value).build()
is Double -> Value.newBuilder().setDouble(value).build()
is Int -> Value.newBuilder().setInteger(value).build()
is Long -> Value.newBuilder().setLong(value).build()
is String -> Value.newBuilder().setString(value).build()
is Set<*> ->
@Suppress("UNCHECKED_CAST")
Value.newBuilder().setStringSet(
StringSet.newBuilder().addAllStrings(value as Set<String>)
).build()
//If it is not the above type, an exception will be thrown directly
else -> throw IllegalStateException(
"PreferencesSerializer does not support type: ${value.javaClass.name}"
)
}
}
You can see that in the last else logic, if it is not the above type, exceptions will be thrown directly. Because the Key is of the type of preferences.Key < T >, the system packages a layer for us by default, which is located in android.datastore.preferences.core.preferenceskeys.kt:
public fun intPreferencesKey(name: String): Preferences.Key<Int> = Preferences.Key(name)
public fun doublePreferencesKey(name: String): Preferences.Key<Double> = Preferences.Key(name)
public fun stringPreferencesKey(name: String): Preferences.Key<String> = Preferences.Key(name)
public fun booleanPreferencesKey(name: String): Preferences.Key<Boolean> = Preferences.Key(name)
public fun floatPreferencesKey(name: String): Preferences.Key<Float> = Preferences.Key(name)
public fun longPreferencesKey(name: String): Preferences.Key<Long> = Preferences.Key(name)
public fun stringSetPreferencesKey(name: String): Preferences.Key<Set<String>> =
Preferences.Key(name)
Because the above declarations are in the top-level function, they can be used directly. For example, if we want to declare a String type preferences. Key < T >, we can declare it directly as follows:
val P_KEY_NAME: Preferences.Key<String> = stringPreferencesKey("key")
-
SP migration to Preferences DataStore
If you want to migrate an SP, you only need to add the produceMigrations parameter in the Preferences DataStore construction phase (the meaning of this parameter has been described in the creation phase) as follows:
//SharedPreference file name
const val BOOK_PREFERENCES_NAME = "book_preferences"
val Context.bookDataStorePf: DataStore<Preferences> by preferencesDataStore(
name = "pf_datastore", //DataStore file name
//Migrate SP to Preference DataStore
produceMigrations = { context ->
listOf(SharedPreferencesMigration(context, BOOK_PREFERENCES_NAME))
}
)
In this way, when the build is completed, the content in the SP will also be migrated to the Preferences DataStore. Note that the migration is one-time, that is, after the migration, the SP file will be deleted, as follows:
Proto DataStore
One disadvantage of SP and Preferences DataStore is that it is unable to define the schema and ensure that the correct data type is used when the access key is used. Proto DataStore can utilize Protocol Buffers( https://developers.google.com/protocol-buffers/ )Define a schema to solve this problem. Protobuf protocol buffer is a mechanism to serialize structured data. By using the protocol, Proto DataStore can know the type of storage and provide the type without using keys.
- Add dependency
1. Add protocol buffer plug-in and Proto DataStore dependency. In order to use Proto DataStore and make protocol buffer generate code for our architecture, protobuf plug-in needs to be introduced into build.gradle:
plugins {
...
id "com.google.protobuf" version "0.8.17"
}
android {
//............. other configurations
sourceSets {
main {
java.srcDirs = ['src/main/java']
proto {
//Specify the proto source file address
srcDir 'src/main/protobuf'
include '**/*.protobuf'
}
}
}
//proto buffer protocol buffer related configurations are used for DataStore
protobuf {
protoc {
//For the protocol version, see: https://repo1.maven.org/maven2/com/google/protobuf/protoc/
artifact = "com.google.protobuf:protoc:3.18.0"
}
// Generates the java Protobuf-lite code for the Protobufs in this project. See
// https://github.com/google/protobuf-gradle-plugin#customizing-protobuf-compilation
// for more information.
generateProtoTasks {
all().each { task ->
task.builtins {
java {
option 'lite'
}
}
}
}
//Modify the location of the generated java class. The default is $buildDir/generated/source/proto
generatedFilesBaseDir = "$projectDir/src/main/generated"
}
}
dependencies {
api 'androidx.datastore:datastore:1.0.0'
api "com.google.protobuf:protobuf-javalite:3.18.0"
...
}
There seem to be a lot of libraries that need to be configured or imported. You can consider putting these configurations into a single module.
2. Defining and using protobuf objects
Just define the way of data structure once, and the compiler will generate source code to easily write and read structured data. We declare the path of the proto source code address in sourceSets {} of the configuration dependency in src/main/protobuf. All proto files should be in the declared path:
Contents of Book.proto file:
//The protobuf version is specified. If it is not specified, proto2 is used by default. It must be specified in the first line
syntax = "proto3";
//option: optional field
//java_package: Specifies the package name of the Java class generated by the proto file
option java_package = "org.ninetripods.mq.study";
//java_outer_classname: Specifies the name of the Java class generated by the proto file
option java_outer_classname = "BookProto";
enum Type {
MATH = 0;
CHINESE = 1;
ENGLISH = 2;
}
message Book {
string name = 1; //title
float price = 2; //Price
Type type = 3; //type
}
After the above code is written, execute build - > rebuild project, and the corresponding Java code will be generated under the path configured by generatedFilesBaseDir, as follows:
3. Create serializer
The serializer defines how to access the data types we define in the proto file. If there is no data on disk, the serializer also defines a default return value. As follows, we create a serializer named BookSerializer:
object BookSerializer : Serializer<BookProto.Book> {
override val defaultValue: BookProto.Book = BookProto.Book.getDefaultInstance()
override suspend fun readFrom(input: InputStream): BookProto.Book {
try {
return BookProto.Book.parseFrom(input)
} catch (exception: InvalidProtocolBufferException) {
throw CorruptionException("Cannot read proto.", exception)
}
}
override suspend fun writeTo(t: BookProto.Book, output: OutputStream) {
t.writeTo(output)
}
}
Among them, BookProto.Book is the code generated through the protocol buffer. If the BookProto.Book object or related methods cannot be found, you can clean up and Rebuild the project to ensure that the protocol buffer generates objects.
-
Building Proto DataStore
//Building Proto DataStore
val Context.bookDataStorePt: DataStore<BookProto.Book> by dataStore(
fileName = "BookProto.pb",
serializer = BookSerializer)
dataStore is a top-level function. The parameters you can pass in are as follows:
- fileName: the name of the file that created the Proto DataStore.
- Serializer: the serializer serializer defines how to access formatted data.
- corruptionHandler: if the DataStore fails to deserialize the data when trying to read the data and throws an android.DataStore.core.corruptionexception, call corruptionHandler.
- produceMigrations: executed when SP migrates to Proto DataStore. ApplicationContext is passed as a parameter to these callbacks, and the migration runs before any access to the data
- Scope: scope of the collaboration. The default IO operation is executed in the Dispatchers.IO thread.
After the above code is executed, a file named BookProto.pb will be created under / data/data / project package name / files / as follows:
Save data
lifecycleScope.launch {
//Building BookProto.Book
val bookInfo = BookProto.Book.getDefaultInstance().toBuilder()
.setName("Hello Proto DataStore")
.setPrice(20f)
.setType(BookProto.Type.ENGLISH)
.build()
bookDataStorePt.updateData { bookInfo }
}
Proto DataStore provides a suspend function DataStore.updateData() to store data. When the storage is completed, the coroutine is also executed.
Fetch data
/**
* Proto DataStore Fetch data
*/
val bookProtoFlow: Flow<BookProto.Book> = context.bookDataStorePt.data
.catch { exception ->
if (exception is IOException) {
emit(BookProto.Book.getDefaultInstance())
} else {
throw exception
}
}
//In Activity
lifecycleScope.launch {
mBookViewModel.bookProtoFlow.collect {
mTvContentPt.text = it.toString()
}
}
Proto DataStore fetches data in the same way as Preferences DataStore and will not be repeated.
-
SP migration to Proto DataStore
//Building Proto DataStore
val Context.bookDataStorePt: DataStore<BookProto.Book> by dataStore(
fileName = "BookProto.pb",
serializer = BookSerializer,
//Migrating SP S to Proto DataStore
produceMigrations = { context ->
listOf(
androidx.datastore.migrations.SharedPreferencesMigration(
context,
BOOK_PREFERENCES_NAME
) { sharedPrefs: SharedPreferencesView, currentData: BookProto.Book ->
//Remove data from SP
val bookName: String = sharedPrefs.getString(KEY_BOOK_NAME, "") ?: ""
val bookPrice: Float = sharedPrefs.getFloat(KEY_BOOK_PRICE, 0f)
val typeStr = sharedPrefs.getString(KEY_BOOK_TYPE, BookProto.Type.MATH.name)
val bookType: BookProto.Type =
BookProto.Type.valueOf(typeStr ?: BookProto.Type.MATH.name)
//Store the data in the SP in the Proto DataStore
currentData.toBuilder()
.setName(bookName)
.setPrice(bookPrice)
.setType(bookType)
.build()
}
)
}
)
Proto DataStore defines the SharedPreferencesMigration class. The following two parameters are specified in migrate:
- SharedPreferences view: can be used to retrieve data from SharedPreferences
- BookProto.Book: current data
Similarly, if productmigrations is passed in during creation, the SP files will be migrated to Proto DataStore, and the SP files will be deleted after migration.
It should also be noted here that the SharedPreferencesMigration class is used in both Preferences DataStore and Proto DataStore during migration, but the package name corresponding to this class is different in these two places. For example, the package name path of Proto DataStore is android.datastore.migrations.SharedPreferencesMigration. When they are written in a file, Note that one of them should use the full path.
/Summary/
Directly compare the SP given by the official with DataStore:
Pay attention to me and share knowledge every day~