Backup Tool

From Sense/Net Wiki
Jump to: navigation, search
  •  
  •  
  •  
  •  
  • 100%
  • 6.0.5
  • Enterprise
  • Community
  • Planned

Overview

Backup tool
Backup tool enables you to back up the Lucene index, the database and the webfolder without stopping the webserver. It needs a complete and running Sense/Net web because it will be used for saving the index. Backups can be created by executing the console application called Backup.exe.

Details

There are three levels of backup: Index, Database and Full. The level specification is required and can be given as a command line argument:

  • -INDEX: Creates a backup of the Lucene index only and puts it into the database.
  • -DATABASE: Creates a backup of the Lucene index in the database first, and then creates a backup of the complete database into the specified backup directory.
  • -FULL: Creates a backup of the Lucene index in the database first, and then creates a backup of both the complete database and the web folder into the specified backup directory. The web folder is compressed to a single .zip file.

Lucene index backup

Every tool that connects to the Sense/Net Content Repository and uses Lucene index has a starting sequence that executes according to the following:

  • if it does not have an index directory, or the directory is empty, it restores the last index backup from the database and executes all index changes that have happened since the last backup.
  • if it does have an index, but it is older than the last backup stored in the database, it will erase the lucene directory and restore the index from db as discussed above.
  • if it does have an index, and there is no newer index backup in the db, it will keep the local index as is and only execute indexing tasks for changes that have happened since the last shutdown of the current tool.

Restoring the last index backup from the database and executing all changes that have happened since the last backup can take a lot of time - especially if there are frequent writes into the repository since the last backup (or last shutdown). In most cases console tools need to start up in this way but it is undesired in the case of the startup of the Backup tool. The backup tool does not exceute any special actions in the Content Repository, it only needs a fresh index to store it in the database - and it is more performant to use the index of a currently running web instance instead of restoring the last backup from the database and replaying all index changes since the time of the last backup to obtain a fresh index. To avoid unnecessary procedures the Backup.exe does not use the Lucene index but sends a message to a running repository across the cluster channel. This way backing up the index is an "in process" method - it always requires a running portal, and it is the running portal that actually creates the backup of the index and not the console tool. When the running application receives the backup request, it acknowledges the request and copies the content the of index directory to an index backup directory, compresses it and stores it into the database. After that, it responds to Backup.exe with a signal across the cluster channel. Backup.exe waits for this response and if it is received in the given timeframe it continues (creating db backup and web folder backup if backup level is DATABASE or FULL), otherwise stops running. The index writing is freezed while the index is being copied, nevertheless the portal can run continuously and the index of the web instance will not be out-of-sync after the backup has finished.

Index backup can communicate with a running Repository instance (hosted by a Sense/Net web) but cannot start it. Make sure the Sense/Net web is running before starting the Backup.exe!

To avoid starting more than one synchronous backups a unique name has to be configured for identifying the running application that will respond to the backup requests and make the backup. The key of unique name is IndexBackupCreatorId in the configurations. The backup tool and the running application must be on the same computer. Also, ClusterChannelProvider and MsmqChannelQueueName have to be configured in the web.config of the repository instance and in the configuration of Backup.exe in order that the backup requests are delivered (see Communication between appdomains for details). The acknowledgement timeout and backup timeframe can also be configured but are not compulsory to give. These are given in seconds and the default is 5 and 20, respectively.

This is the default configuration for the index backup in the Backup.exe.config:

<appSettings>
  ...
  <add key="ClusterChannelProvider" value="SenseNet.Communication.Messaging.MsmqChannelProvider, SenseNet.Storage"/>
  <add key="MsmqChannelQueueName" value=".\private$\ryan"/>
  <add key="IndexBackupStartedTimeOut" value="5"/>
  <add key="IndexBackupFinishedTimeOut" value="20"/>
  <add key="EnableOuterSearchEngine" value="false"/>
  <add key="IndexBackupCreatorId" value="CustomId"/>

The index backup directory is configured in the web.config of the reciever application. The default is: "..\LuceneIndex_backup". To configure a web instance as a reciever for backup requests the following settings have to be configured in the web.config:

<appSettings>
  ...
  <add key="IndexDirectoryBackupPath" value="..\LuceneIndex_backup" />
  <add key="IndexBackupCreatorId" value="CustomId"/>
  ...
  <add key="ClusterChannelProvider" value="SenseNet.Communication.Messaging.MsmqChannelProvider, SenseNet.Storage"/>
  <add key="MsmqChannelQueueName" value=".\private$\ryan"/>

Configure IndexBackupCreatorId for a single web instance and the backup tool only! Configuring multiple web instances as backup request recievers is unnecessary, not performant, and altogether not recommended!

Check the ClusterChannelProvider and the MsmqChannelQueueName settings in the web.config and the Backup.exe.config! If MSMQ is not configured correctly, index backup will not work at all!

Backup strategy for huge index

As the index backup is stored in the database itself, there is a recommended index size limit for using this backup tool. If the compressed Lucene index folder is bigger than 500MB, you should consider making manual backups instead. In that case please use the following backup strategy:

  • choose one of the running web sites and shut it down
  • backup the Lucene index folder manually (you will find it in the App_Data folder of the web site)
  • restart the web site
  • create a backup from the database

If you follow the steps above, you will have the database and index backup in your hands that you can use to restore the system at any time. Do not worry about the few minutes time gap between the creation of the two backups: when you restore these backups, the system will correct the index files using the fresher information stored in the database if needed.

When you want to restore the backup, follow these steps:

  • shut down every web site
  • delete all Lucene index folders (in every App_Data folder)
  • restore the database
  • copy the Lucene index backup to all the web site App_Data folders
  • restart web sites

Index backup on live sites

The backup mechanism of Sense/Net ensures that the index is in a consistent state when it is compressed and stored in the database. To make this available the web application creating the index backup freezes every saving and indexing procedure while the index is being copied to its temporary backup location. This could take several seconds and means that processing of requests may be delayed while the index is being copied. After the index copy is done every request (even the delayed ones) are processed as normally. If the copying of the index exceeds a certain limit, the saving/indexing requests will fail with the following error message:

Operation timed out, indexing is paused.

The default timeout for the index to be copied is 60 seconds. This can be configured in the web.config of the backup creator web application:

    <!-- Indexing pause timeout in seconds. Default is 60. -->
    <!--<add key="IndexingPausedTimeout" value="60"/>-->

If an error occurs during backup, the backup fails and no consecutive requests are served correctly, the system must be restarted.

Database backup

Database backup works with SQL scripts provided by the current database provider. The backup file will be placed into the filesystem (see below). Database provider must be configured. The default is the built-in SqlProvider:

<appSettings>
  ...
  <add key="DataProvider" value="SenseNet.ContentRepository.Storage.Data.SqlClient.SqlProvider" />
  ...
</appSettings>

Since the Lucene index saving always precedes the database backup, thus the database backup always contains the latest index. This method ensures the fastest startup of any newly restored website.

Full backup

Backup using Full backup level is carried out according to the following:

  • saving the Lucene index
  • zipping the web folder
  • backing up the database

The webfolder saving is reasonable to preserve any changes or custom settings in the web folder (including the web.config). The webfolder backup is a compressed file and it is placed next to the database backup file. This level requires the -WEB command followed by the webfolder's full path. There might be some items that should be excluded from the webfolder backup. These items are listed in the configuration of the Backup.exe. The default is:

<appSettings>
  ...
  <add key="WebDirectoryExclusions" value="LuceneIndex, LuceneIndex_backup, App_Data, obj, Root, WebSite.csproj, WebSite.csproj.user, WebSite.csproj.vspscc" />
  ...
</appSettings>

The items are separated by comma. The exclusion list automatically includes the backup main directory if it is placed in the web directory.

If a file and a directory has the same name, both will be excluded if put into the exclusion list.

Please note that only folders or files in the web root directory can be excluded. Subfolder exclusion is not supported.

The Backups folder

The main backup directory name is Backups next to the Backup.exe if it is not specified with the -TARGET parameter.

Every backup has a directory under the main backup directory with special name. Backup directory content depends on the backup level:

  • INDEX: directory won't be created
  • DATABASE: database backup file
  • FULL: database backup file and a webfolder zip file

The backup directory, the database backup file and the web zip file have a special name in the following format: {prefix}{time}

  • prefix: Configurable appSettings value
    • backup directory: appSettings key: BackupDirectoryPrefix. Default is SnBackup-
    • database file: appSettings key: DatabaseBackupPrefix. Default is SenseNetContentRepository-
    • web zip file: appSettings key: WebZipPrefix. Default is WebSite-
  • time: backup date and time in this format: yyyyMMdd-HHmmss

Automatically cleaning the backup directories

There is a way to delete unnecessary backups. This can be done using the -KEEPDAYS command followed by the number of days. After the current backup is finished, every backup directory will be deleted whose creation date is older than T - days, where days is the passed parameter.

The time is counted from 0:00:00. For example if the KEEPDAYS value is 1 and you are starting a backup now, after it is finished all backups are available that are created today and yesterday.

Because the -INDEX level backup does not use the file system, the cleaning is not executed in this level.

Checking parameters

If the -CONFIRM parameter is used, the Backup tool displays the interpreted parameters and waits for user confirmation before executing the tasks. After the answer is Y, the backup will run normally. If the pressed key is the N, the tasks won't be executed.

Example

  • Index backup only
 Backup.exe -INDEX
  • Index and database backup without web directory
 Backup.exe -DATABASE
 Backup.exe -DATABASE -TARGET C:\...\SenseNet\WebSite\Backups
  • Index, database and web directory backup
 Backup.exe -FULL -WEB C:\...\SenseNet\WebSite
 Backup.exe -FULL -WEB C:\...\SenseNet\WebSite -TARGET C:\...\SenseNet\WebSite\Backups
  • Cleaning
 Backup.exe -DATABASE -TARGET C:\...\SenseNet\WebSite\Backups -KEEPDAYS 2
  • Index, database and web directory backup with cleaning and confirmation
 Backup.exe -FULL -WEB C:\...\SenseNet\WebSite -TARGET C:\...\SenseNet\WebSite\Backups -KEEPDAYS 2 -CONFIRM

In this case the Backup.exe shows the following screen:

Confirmed backup

Related links

References

There are no related external articles for this article.