Chapter 2 Database Environments
Whether you are using Direct Persistence Layer (DPL) or the underlying API, you must use a database environment. A database environment encapsulates one or more databases. By allowing a single in-memory cache for each database contained in the environment, this encapsulation provides your threads with effective access to the database. This encapsulation also allows you to group operations performed by multiple databases within a single transaction (see Berkeley DB, Java Edition Getting Started with Transaction Processing guide for more information).
If you are using a basic API, you usually use a database environment to create and open databases (closing individual databases with a single database handle). You can also use the environment to delete and rename databases. For transactional applications, you can use this environment to start transactions. For non-transactional applications, you can use the environment to synchronize memory caches to disk.
If you are using DPL, all these things are still in progress, but DPL will handle them for you. Under DPL, you'll make it clear that the most common thing to use an environment is to get transaction handles.
Whatever API you use, you can also use the database environment for management and configuration activities related to database log files and in-memory caching. See Managing Berkeley DB Java Edition Applications for more information.
To learn how to use environments in transactionally protected applications, see Berkeley DB, Java Edition Getting Started with Transaction Processing guide.
2.1 Opening Database Environments (Opening Database Environment)
Open a database environment by instantiating an Environment object. You must give the constructor the name of the disk directory where the environment is located. The directory location must exist or it will fail to open.
By default, if the environment does not exist, the environment will not be created for you. If you want to create an environment, set the creation property to true. For example:
import java.io.File; import java.util.IllegalFormatCodePointException; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; /** * Open the environment * Open the database environment * * */ public class Study001_Open_Single_Environment { //Configure the database environment file path, private final static String BDB_001_ENV_HOME_FILE_PATH="bdb_001_env_home"; private final static File BDB_001_ENV_HOME_File=new File(BDB_001_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub //Create environment object references Environment myDbEnvironment = null; try { //Create an environment configuration object reference and instantiate an environment configuration object EnvironmentConfig envConfig = new EnvironmentConfig(); //Settings allow creation if they do not exist envConfig.setAllowCreate(true); //Create if the file path does not exist if (!BDB_001_ENV_HOME_File.exists()) { BDB_001_ENV_HOME_File.mkdirs(); } //Open a database environment by instantiating an Environment object. myDbEnvironment = new Environment(BDB_001_ENV_HOME_File, envConfig); //(Setting up database environment path, loading environment configuration object) //Print configuration files System.out.println("------Beginning of database configuration file---------"); System.out.println(myDbEnvironment.getConfig()); System.out.println("-----End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here dbe.printStackTrace(); System.err.println(dbe.toString()); } } }
Opening the environment usually results in starting some background threads. JE uses these threads for log file cleaning and some management tasks. However, these threads are opened only once in each process, so if you open the same environment multiple times in the same process, there will be no performance impact on the application. In addition, background threads (except evictor threads) do not start if the environment is opened read-only.
Please note that turning on your environment will lead to a normal recovery. This will cause your database to enter a consistent state related to the changed data found in your log file. For more information, see the database and log files.
2.1.1 Multiple Environment
Most JE applications require only one database environment, because any number of databases can be created in a single environment, and the total size of the data in the environment is unlimited. That is to say, your application can open and use as many environments as possible, because you can manage disk and memory. In addition, you can instantiate multiple Environment objects for the same physical environment.
The main reason for multiple environments is that applications must manage multiple unique data sets. By placing each data set in a separate environment, applications can gain real advantages in data manageability and application performance. By placing each data set in a unique environment, a set of separate log files will be created and stored in a separate directory, so you can manipulate the log files for each data set individually. That is to say, you can:
Individual backup, restore or delete a single data set by copying or deleting files in its environment.
Balancing loads between machines by moving files from one machine to another from a single data set.
I/O performance is improved by placing each data set on a separate physical disk.
It is very effective to delete a single data set by deleting the environment's log files. This is more efficient than deleting individual database records, and more efficient than deleting databases. This may be a real benefit if you manage large temporary datasets that must be frequently deleted.
Note the drawbacks of using multiple environments. In particular, it is important to understand that a single transaction cannot contain changes made in multiple environments. If you need to perform a set of operations in multiple data sets in an atomic manner (using a single transaction), use a single environment and use other methods to differentiate data sets.
For example, an application running hosted services for multiple clients might want to separate the data sets of each client. You can do this in multiple environments, but you can do atomic operations on all data sets. If you need to encapsulate multiple data sets in a single transaction, consider other ways to separate data sets.
For example, you can use a unique key range in a single database to differentiate each data set. Or you can create an auxiliary key that identifies the data set. Or you can use different databases for each data set. All of these methods allow you to maintain multiple different data sets in a single environment, but obviously each data set adds a layer of complexity to your code, rather than just using a unique environment for each data set.
2.1.2 Multiple Environment Subdirectories
You can distribute the JE environment in multiple subdirectories. This allows you to improve data throughput by distributing disk I/O to multiple disks or file systems. The environment subdirectory resides in the environment home directory and is consecutively named data001 / through dataNNN /, where NNN is the number of subdirectories to be used. Typically, each dataNNN / name is a symbolic link to the actual directory residing on a separate file system or disk. Or, each subdirectory can be a mount point for the file system that resides on different disk drives.
You can control the number of subdirectories to be used through the je. log. nData Directories property in the je.properties file. This value must be set before the environment is opened, and the subdirectory must already exist at this time. The value set for this property cannot be changed during the lifetime of the environment, or an exception is thrown when trying to open the environment.
The default value of je. log. nData Directories is 0, which means that no subdirectories are in use in the environment. A value greater than 0 denotes the number of subdirectories to be used, and the number of subdirectories must exist before the environment is opened.
For example, if you set je. log. nData Directories to 3, the environment home directory must contain three subdirectories named data001, data002 and data003 when you first open the environment (and then each environment). This will cause your JE log files (*. jdb files) to be evenly distributed among the three subdirectories. Finally, if you change the value of je. log. nData Directories without completely deleting the environment, the application throws an exception when you open the environment.
2.1.3 Configuring a Shared Cache for Multiple Environments (Configuring shared caches for multiple environments)
By default, each different JE environment has a separate dedicated in-memory cache. If a single JVM process opens multiple environments simultaneously, it is strongly recommended that all such environments be configured to use shared caching. Shared caches use memory more efficiently than separate private caches.
For example, suppose you open five environments in a process and have a total of 500 MB of memory available for caching. Using dedicated caches, you can configure each cache to 100 MB. If one environment has a larger active dataset than the other, it will not be able to take advantage of unused memory in the cache of the other environment. By using shared caching, multiple open environments will make better use of memory, because caching LRU algorithms apply to all information in all environments of shared caching.
To configure the environment to use shared caching, set EnvironmentConfig.setSharedCache () to true. This must be set for each environment in the process in which you want to use shared caching. For example:
import java.io.File; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; public class Study002_SharedCache_Multi_Environment { //Configure the database environment file path, private final static String BDB_002_01_ENV_HOME_FILE_PATH="bdb_002_01_env_home"; private final static String BDB_002_02_ENV_HOME_FILE_PATH="bdb_002_02_env_home"; private final static File BDB_002_01_ENV_HOME_File=new File(BDB_002_01_ENV_HOME_FILE_PATH); private final static File BDB_002_02_ENV_HOME_File=new File(BDB_002_02_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub Environment myEnv1 = null; Environment myEnv2 = null; try { EnvironmentConfig envConfig = new EnvironmentConfig(); //Allow reconstruction envConfig.setAllowCreate(true); /** * To configure the environment to use shared caching, set EnvironmentConfig.setSharedCache () to true. * This must be set for each environment in the process in which you want to use shared caching. * */ envConfig.setSharedCache(true); //Create an environment file path if it does not exist if(!BDB_002_01_ENV_HOME_File.exists()) { BDB_002_01_ENV_HOME_File.mkdirs(); } if(!BDB_002_02_ENV_HOME_File.exists()) { BDB_002_02_ENV_HOME_File.mkdirs(); } myEnv1 = new Environment(BDB_002_01_ENV_HOME_File, envConfig); myEnv2 = new Environment(BDB_002_02_ENV_HOME_File, envConfig); System.out.println("------Beginning of database configuration file---------"); System.out.println(myEnv1.getConfig()); System.out.println(myEnv2.getConfig()); System.out.println("------End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here } } }
2.2 Closing Database Environments
Close your environment by calling the Environment.close () method. This method executes checkpoints, so there is no need to explicitly execute synchronization or checkpoints before calling it. For information about checkpoints, see Berkeley DB, Java Edition "Introduction to Transaction Processing". For information on synchronization, see Database Modification and Synchronization.
import java.io.File; import java.util.IllegalFormatCodePointException; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; /** * Open the environment Open the database environment * */ public class Study001_Open_Single_Environment { // Configure the database environment file path, private final static String BDB_001_ENV_HOME_FILE_PATH = "bdb_001_env_home"; private final static File BDB_001_ENV_HOME_File = new File(BDB_001_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub // Create environment object references Environment myDbEnvironment = null; try { // Create an environment configuration object reference and instantiate an environment configuration object EnvironmentConfig envConfig = new EnvironmentConfig(); // Settings allow creation if they do not exist envConfig.setAllowCreate(true); // Create if the file path does not exist if (!BDB_001_ENV_HOME_File.exists()) { BDB_001_ENV_HOME_File.mkdirs(); } // Open a database environment by instantiating an Environment object. myDbEnvironment = new Environment(BDB_001_ENV_HOME_File, envConfig); // (Setting up database environment path, loading environment configuration object) // Print configuration files System.out.println("------Beginning of database configuration file---------"); System.out.println(myDbEnvironment.getConfig()); System.out.println("-----End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here dbe.printStackTrace(); System.err.println(dbe.toString()); } // Close the database environment if (myDbEnvironment != null) { myDbEnvironment.close(); } } }
If you are using DPL, you can only close your environment after all other store activities have been completed and all Stores currently open in that environment have been closed. If you are using the basic API, you can only close your environment after all other database activities have been completed and you have closed any database currently open in that environment.
The environment may shut down before the JE cleaner thread completes its work. This happens if you execute a large number of deletions immediately before closing the environment. As a result, your log file may be much larger than you expected, because cleaner threads have no chance to complete its work.
For more information about vacuum cleaner threads, please refer to vacuum cleaner threads.
If you want to ensure that the cleaner is running before cleaning up the environment, call Environment.cleanLog () before calling Environment.close ():
Closing the last environment handle in the application results in releasing all internal data structures and stopping background threads. If there are any open databases, the JE will complain before closing them. At this point, any ongoing transactions will be suspended. Any public cursors are also closed at this time. However, it is recommended that you close all cursor handles immediately after using them to ensure concurrency and release resources, such as page locking.
2.3 Environment Properties
- You can use the EnvironmentConfig class to set properties for the environment.
- You can also use EnvironmentMutableConfig to set properties for specific Environment instances.
2.3.1 The EnvironmentConfig Class
The EnvironmentConfig class provides you with a large number of fields and methods. Describing all these adjustment parameters is beyond the scope of this manual. However, you may want to set some properties. They are described here.
Note that for each property you can usually set, there is a corresponding getter method. In addition, you can always use the Environment.getConfig () method to retrieve the EnvironmentConfig object used by the environment.
You can set environment configuration parameters in the EnvironmentConfig class by using the following methods:
- EnvironmentConfig.setAllowCreate()
If true, the database environment is created when opened. If false, then if the environment does not exist, the environment opens fail. This property is meaningless if the database environment already exists. The default is false.
- EnvironmentConfig.setReadOnly()
If true, all databases open in this environment must be read-only. If you are writing a multi-process application, all processes except one must set this value to true. The default is false.
You can also set this property using the je.env.isReadOnly parameter in the env_home / je.properties file.
- EnvironmentConfig.setTransactional()
If true, configure the database environment to support transactions. The default is false.
You can also set this property using the je.env.isTransactional parameter in the env_home / je.properties file.
import java.io.File; import java.util.IllegalFormatCodePointException; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; /** * Open the environment Open the database environment * */ public class Study001_Open_Single_Environment { // Configure the database environment file path, private final static String BDB_001_ENV_HOME_FILE_PATH = "bdb_001_env_home"; private final static File BDB_001_ENV_HOME_File = new File(BDB_001_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub // Create environment object references Environment myDbEnvironment = null; try { // Create an environment configuration object reference and instantiate an environment configuration object EnvironmentConfig envConfig = new EnvironmentConfig(); // If true, the database environment is created when opened. //If false, then if the environment does not exist, the environment opens fail. //This property is meaningless if the database environment already exists. The default is false. envConfig.setAllowCreate(true); //If true, configure the database environment to support transactions. The default is false. envConfig.setTransactional(true); // Create if the file path does not exist if (!BDB_001_ENV_HOME_File.exists()) { BDB_001_ENV_HOME_File.mkdirs(); } // Open a database environment by instantiating an Environment object. myDbEnvironment = new Environment(BDB_001_ENV_HOME_File, envConfig); // (Setting up database environment path, loading environment configuration object) // Print configuration files System.out.println("------Beginning of database configuration file---------"); System.out.println(myDbEnvironment.getConfig()); System.out.println("-----End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here dbe.printStackTrace(); System.err.println(dbe.toString()); } // Close the database environment if (myDbEnvironment != null) { myDbEnvironment.cleanLog(); // Clean the log before closing myDbEnvironment.close(); } } }
2.3.2 EnvironmentMutableConfig
EnvironmentMutableConfig manages properties that can be reset after building an EnvironmentObject. In addition, EnvironmentConfig extends EnvironmentMutable Config, so you can set these variable attributes as needed during environmental construction.
The EnvironmentMutableConfig class allows you to set the following properties:
- setCachePercent()
Determine the percentage of JVM memory available for JE caching. For more information, see Adjusting Cache Size.
- setCacheSize()
Determine the total amount of memory available for database caching. For more information, see Adjusting Cache Size.
- setTxnNoSync()
Determines whether the change record created due to transaction commit is written to the backup log file on disk. Value true causes data not to be refreshed to disk. See Berkeley DB, Java Edition Getting Started with Transaction Processing Guide.
- setTxnWriteNoSync()
Determine whether the log is refreshed at transaction commit (but the log is still written). By setting this value to true, you may achieve better performance than by committing refresh logs, but you will gain better performance by losing some transaction persistence guarantees.
There is also a corresponding getter method (getTxnNoSync ()). Furthermore, you can always retrieve the EnvironmentMutableConfig object of the environment using the Environment.getMutableConfig () method.
import java.io.File; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; import com.sleepycat.je.EnvironmentMutableConfig; /** * * */ public class Study003_EnvironmentMutableConfig { // Configure the database environment file path, private final static String BDB_003_ENV_HOME_FILE_PATH = "bdb_003_env_home"; private final static File BDB_003_ENV_HOME_File = new File(BDB_003_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub // Create environment object references Environment myDbEnvironment = null; try { // Create an environment configuration object reference and instantiate an environment configuration object EnvironmentConfig envConfig = new EnvironmentConfig(); // If true, the database environment is created when opened. // If false, then if the environment does not exist, the environment opens fail. // This property is meaningless if the database environment already exists. The default is false. envConfig.setAllowCreate(true); // If true, configure the database environment to support transactions. The default is false. envConfig.setTransactional(true); // Create an environment configuration object reference and instantiate an environment multi-configuration object EnvironmentMutableConfig envMultableConfig = new EnvironmentMutableConfig(); // Determines whether the change record created due to transaction commit is written to the backup log file on disk. Value true causes data not to be refreshed to disk. envMultableConfig.setTxnNoSyncVoid(false); // Determine whether the log is refreshed at transaction commit (but the log is still written). // By setting this value to true, you may achieve better performance than by committing refresh logs, but you will gain better performance by losing some transaction persistence guarantees. envMultableConfig.setTxnWriteNoSyncVoid(false); // Create if the file path does not exist if (!BDB_003_ENV_HOME_File.exists()) { BDB_003_ENV_HOME_File.mkdirs(); } // Open a database environment by instantiating an Environment object. myDbEnvironment = new Environment(BDB_003_ENV_HOME_File, envConfig); // (Setting up the database environment path) myDbEnvironment.setMutableConfig(envMultableConfig); // Print configuration files System.out.println("------Beginning of database configuration file---------"); System.out.println(myDbEnvironment.getConfig()); System.out.println("-----End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here dbe.printStackTrace(); System.err.println(dbe.toString()); } // Close the database environment if (myDbEnvironment != null) { myDbEnvironment.cleanLog(); // Clean the log before closing myDbEnvironment.close(); } } }
2.4 Environment Statistics
JE provides a lot of information about how your environment works. Most of the information involved figures only related to JE developers, so the description of these statistics is beyond the scope of this manual.
However, a very important statistic (especially for long-running applications) is EnvironmentStats.getNCacheMiss (). This statistics returns the total number of requests for database objects that are not available in the cache. This number is important for application administrators trying to determine the correct size of caches in memory. See Adjusting Cache Size for details.
To get this statistics from your environment, call Environment.getStats () to return the EnvironmentStats object. Then you can call the EnvironmentStats.getNCacheMiss () method. For example:
import java.io.File; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; import com.sleepycat.je.EnvironmentMutableConfig; /** * * */ public class Study003_EnvironmentMutableConfig { // Configure the database environment file path, private final static String BDB_003_ENV_HOME_FILE_PATH = "bdb_003_env_home"; private final static File BDB_003_ENV_HOME_File = new File(BDB_003_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub // Create environment object references Environment myDbEnvironment = null; try { // Create an environment configuration object reference and instantiate an environment configuration object EnvironmentConfig envConfig = new EnvironmentConfig(); // If true, the database environment is created when opened. // If false, then if the environment does not exist, the environment opens fail. // This property is meaningless if the database environment already exists. The default is false. envConfig.setAllowCreate(true); // If true, configure the database environment to support transactions. The default is false. envConfig.setTransactional(true); // Create an environment configuration object reference and instantiate an environment multi-configuration object EnvironmentMutableConfig envMultableConfig = new EnvironmentMutableConfig(); // Determines whether the change record created due to transaction commit is written to the backup log file on disk. Value true causes data not to be refreshed to disk. envMultableConfig.setTxnNoSyncVoid(false); // Determine whether the log is refreshed at transaction commit (but the log is still written). // By setting this value to true, you may achieve better performance than by committing refresh logs, but you will gain better performance by losing some transaction persistence guarantees. envMultableConfig.setTxnWriteNoSyncVoid(false); // Create if the file path does not exist if (!BDB_003_ENV_HOME_File.exists()) { BDB_003_ENV_HOME_File.mkdirs(); } // Open a database environment by instantiating an Environment object. myDbEnvironment = new Environment(BDB_003_ENV_HOME_File, envConfig); // (Setting up the database environment path) myDbEnvironment.setMutableConfig(envMultableConfig); //Environment.getStats () can only get statistics from the application process //This statistics returns the total number of requests for database objects that are not available in the cache. long cacheMisses = myDbEnvironment.getStats(null).getNCacheMiss(); System.out.println(cacheMisses); // Print configuration files System.out.println("------Beginning of database configuration file---------"); System.out.println(myDbEnvironment.getConfig()); System.out.println("-----End of database configuration file---------"); } catch (DatabaseException dbe) { // Exception handling goes here dbe.printStackTrace(); System.err.println(dbe.toString()); } // Close the database environment if (myDbEnvironment != null) { myDbEnvironment.cleanLog(); // Clean the log before closing myDbEnvironment.close(); } } }
Note that Environment.getStats () can only get statistics from the process of the application. In order for the application administrator to get this statistics, you must use JMX to retrieve the statistics (see JConsole and JMX support), or you must print it to check (for example, record values every minute).
Keep in mind that what really matters for the size of the cache is the change of this value over time, not the actual value itself. So you might consider providing increments from this statistic to the next (an increment of 0 is ideal, and a large increment is an indication of too little caching).
2.5 Database Environment Management Example
This example provides a complete class that can open and close an environment. In subsequent examples of this book, it has been extended to open and close environments and databases. We did this to make the sample code simpler and easier to manage. You can find this course in the following places:
JE_HOME/examples/je/gettingStarted/MyDbEnv.java
JE_HOME is where the JE distribution is placed.
2.5.1 Example 2.1 Database Environment Management Class
import java.io.File; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; public class Study004_DatabaseEnvironmentManagement { private Environment myEnv; // We use class constructors to instantiate the EnvironmentConfig object, which is used to configure our environment when it is opened. public Study004_DatabaseEnvironmentManagement() { } public void setup(File envHome, boolean readOnly) throws DatabaseException { // Instantiate an environment configuration object EnvironmentConfig myEnvConfig = new EnvironmentConfig(); // Configuration Read-Only myEnvConfig.setReadOnly(readOnly); // If true, the database environment is created when opened. // If false, then if the environment does not exist, the environment opens fail. // This property is meaningless if the database environment already exists. The default is false. myEnvConfig.setAllowCreate(!readOnly); // If true, configure the database environment to support transactions. The default is false. myEnvConfig.setTransactional(!readOnly); // Instantiate the Environment. This opens it and also possibly // creates it. myEnv = new Environment(envHome, myEnvConfig); } // Getter methods // Next we provide a getter method that allows us to retrieve the environment directly. This is required for the examples that follow this guide. public Environment getEnv() { return myEnv; } // Close the environment // finally, we need a way to shut down our environment. We encapsulate this operation in a try block so that it can be used properly in final statements. public void close() { if (myEnv != null) { try { myEnv.cleanLog(); myEnv.close(); } catch (DatabaseException dbe) { System.err.println("Error closing environment" + dbe.toString()); } } } }
Code call
import java.io.File; import com.sleepycat.je.DatabaseException; public class Study004_DatabaseEnvironmentManagementTest { // Configure the database environment file path, private final static String BDB_004_ENV_HOME_FILE_PATH = "bdb_004_env_home"; private final static File BDB_004_ENV_HOME_File = new File(BDB_004_ENV_HOME_FILE_PATH); public static void main(String[] args) { // TODO Auto-generated method stub Study004_DatabaseEnvironmentManagement study004_DatabaseEnvironmentManagement = new Study004_DatabaseEnvironmentManagement(); //Create the database environment file path if it does not exist if(!BDB_004_ENV_HOME_File.exists()) { BDB_004_ENV_HOME_File.mkdirs(); } try { study004_DatabaseEnvironmentManagement.setup(BDB_004_ENV_HOME_File, false); //Print configuration information System.out.println("-----------------------Print Environment Config start---------------------------------------"); System.out.println(study004_DatabaseEnvironmentManagement.getEnv().getConfig()); System.out.println("-----------------------Print Environment Config end--------------------------------------"); } catch (DatabaseException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { study004_DatabaseEnvironmentManagement.close(); } } }