MongoDB blob provider

From Sense/Net Wiki
Jump to: navigation, search
  •  
  •  
  •  
  •  
  • 10%
  • 6.5.5
  • Enterprise
  • Community
  • Planned

sensenet 7: this article has a more recent version on the new community site.

Overview

By default Sense/Net ECM stores binaries (files) in the database. In enterprise environments it is not a suitable solution for various reasons (licensing and storage among them), so we offer an option to store binaries in an external Blob provider instead of the main Content Repository database. This article is about an external blob provider we offer for our Enterprise customers. It stores binaries in MongoDB so that operators have more flexibility when installing and maintaining Sense/Net ECM. We deliver our Enterprise customer an Enterprise Mongo Package that contains all the binaries needed to make it work.

Details

Although MongoDB is capable of storing metadata (usually that is what it is used for), we decided to use it as our binary storage in this blob provider. We chose it because it is more scalable and reliable than a simple file system (SMB) solution and is able to work better in a distributed environment.

Installation and maintenance

You should follow the instructions below to install and maintain this enterprise feature.

MongoDB installation

We tested this component with MongoDB 3.2. You need to be familiar with this third party product to install, set up and administer it. You can learn more of this version at the MongoDB 3.2 homepage.

MongoDB database installation

Sense/Net ECM can connect to an existing MongoDB server. If the specified database defined in the connection does not exist, MongoDB creates it when used for the first time. Even though the automatically created database is usable, the necessary indexes (for the database to be performant) are missing. The indexes can be created anytime. We recommend that you prepare the database and indexes before first use.

Datbase and index creation

The preferred MongoDB database administration tool is the MongoDB console. The necessary operations are the following.

Start a Powershell console or a CMD shell. Go to the installed server's bin directory and start MongoDB console.

CD [mongodb install dir]\bin
.\mongod.exe

Check whether the database exists with the "show dbs" command.

show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

If not, enter the database and create the "Blobs" collection. The collection name cannot be changed. The database will be created automatically if it does not exist.

use [databasename]
db.createCollection("Blobs")

Check whether the indexes exist. Use the "getIndexes" function of the "Blobs" collection.

db.Blobs.getIndexes()

In case of a new database a single index is created automatically. The output is something like this (see the "key" that is "_id" in this case):

[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "[databasename].Blobs"
    }
]

If the indexes do not exist, create them.

db.Blobs.createIndex( { FileIdentifier: 1, ChunkIndex: 1 }, { name: "FileIdentifier_ChunkIndex" })
db.Blobs.createIndex( { FileIdentifier: 1 }, { name: "FileIdentifier" })

After that the database and indexes are successfully prepared and usable.

Configuration

You need to change the configuration file for your content repository and for tools as well in order to use MongoDB for blob storage.

For a first step, check the components. The following files must be copied from your Enterprise Mongo Driver package into [yourwebsite]\bin and [yourwebsite]\tools:

  • MongoDB.Bson.dll
  • MongoDB.Driver.Core.dll
  • MongoDB.Driver.dll
  • SenseNet.MongoDbBlobStorage.dll

It is important that you keep all the configurations in sync for the different components in Sense/Net ECM - otherwise you won't be able to save or read binaries correctly.

Sense/Net web application

The following file is affected: web.config To configure the Repository, you must do: Configure the connectionstring. (Give a valid connectionstring like this: mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]] For further reading: https://docs.mongodb.com/manual/reference/connection-string/) The only further constraint on it that the database name is always mandatory.

<connectionStrings>
   <add name="SenseNet.MongoDbBlobDatabase" connectionString="mongodb://[hostname]:[port]/[database name]" />
   ...

Configure the BinaryChunkSize that must be multiple of the MongoDbBlobDatabaseChunkSize (see below).

<sensenet>
   <dataHandling>
      <add key="BinaryChunkSize" value="5160960" />
      ...

Configure the blob provider:

<appSettings>
   <add key="BlobProvider" value="SenseNet.ContentRepository.Storage.Data.MongoDbBlobStorage.MongoDbBlobProvider"/>|
   <add key="MinimumSizeForBlobProviderKB" value="10000"/>
   <add key="MongoDbBlobDatabaseChunkSize" value="516096"/>
   ...

Take care of the the BinaryChunkSize and MongoDbBlobDatabaseChunkSize values: BinaryChunkSize can be divided without a remainder with MongoDbBlobDatabaseChunkSize.

BinaryChunkSize must be multiple of MongoDbBlobDatabaseChunkSize otherwise an exception is thrown

Tools

Configure the tools if you use them. Modify all relevant configurations in the tools directory. The tool is relevant if the it reads or writes any repository content. The following files are affected: Import.exe.config, Export.exe.config, SyncPortal2AD.exe.config, SnAdminRuntime.exe.config. On each tool configuration file you must do as it was bid in the previous section.

Sharding

The MongoDB Blob Provider can be used in sharded environment. Of course if you use sharding you have to add the host and port of a routing server (mongos) to the connection string instead of a single data server (mongod). On the first binary content operation the provider creates a collection named Blobs in the given database with a FileIdentifier guid field in it. FileIdentifier is a generated unique identifier, so it is strongly recommended to use it as the shard key. The MongoDB Blob Provider was tested in a MongoDB 3.2 sharded cluster environment with two distributed shards in it. For more about sharding please read the vendor's documentation.

Backup

Migration

Currently there is no migration tool that moves existing binaries from the main database to MongoDB. However after you install this provider, the system will save new files to it and old files will also be moved as you save them.

Related links

References

There are no external references for this article.