This project has moved. For the latest updates, please go here.

How to properly dispose of / close / release the store when using EntityContext?

Feb 12, 2015 at 2:28 PM
Edited Feb 12, 2015 at 2:28 PM
Calling Dispose() on MyEntityContext releases the lock on the data.bs file. That is, unless something was changed and SaveChanges was called. When an entity is added, 2 additional locks are put on the data.bs file. Overall (adding, editind, deleting entities) up to 5 locks can be set on that file, and MyEntityContext.Dispose() releases only one of them.

The situation is the same for the store of PersistenceType.Rewrite and AppendOnly. I tried calling BrightstarService.Shutdown(). I tried overriding the context and calling Cleanup();

Nothing works and I can't find anything about this in the documentation.

Reason for asking: I'd like to consolidate the store when my application is exiting:
var client = BrightstarService.GetClient(connectionString);
var consolidateJob = client.ConsolidateStore(storename);
  while (!(consolidateJob.JobCompletedOk || consolidateJob.JobCompletedWithErrors))
{
        System.Threading.Thread.Sleep(500);
        consolidateJob = client.GetJobInfo(storename, consolidateJob.JobId);
         Console.WriteLine(consolidateJob.StatusMessage);
}
The job completes without failing if the data wan't updated. If it was, then the job waits for some time for the lock on the file to be released, and then finally throws System.IO.IOException.

What am I doing wrong?
Coordinator
Feb 12, 2015 at 7:45 PM
From what you describe, it doesn't sound like you are doing anything wrong. The consolidate job should be able to run even if there have been updates on the store.
I'll have a look into the file locking issues, though I must say that I haven't seen this problem in using B* myself, and we have unit tests that create, populate and then consolidate a store in a single unit test.

Are you sure that the file that it is failing to lock is data.bs ? If you have a stack trace for the error that would be helpful. Do you have anything else running that might be locking the store ? This would include Polaris and the BrightstarService executable if you have it installed as a Windows service for example, but it might also include the Windows indexing service or perhaps even a virus scanner).

If you have a simple piece of code that consistently reproduces the error on your system, can you post that here - it helps to make sure that we are really testing the same thing!
Feb 13, 2015 at 6:08 AM
I'm using BirightstarDB v1.9 for .net 4.0. I installed it in my C# project using Nuget. Nothing else Brightstar-related is installed.

The consolidation job returns with stack trace:
BrightstarDB Error: 400 : Error Processing Transaction System.IO.IOException: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces.
   w System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   w System.IO.__Error.WinIOError()
   w System.IO.FileInfo.MoveTo(String destFileName)
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.<>c__DisplayClassa.<RenameFile>b__8()
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.WrapSharingViolations(WrapSharingViolationsCallback action, WrapSharingViolationsExceptionsCallback exceptionsCallback, Int32 retryCount, Int32 waitTime)
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.RenameFile(String storeConsolidateFile, String storeDataFile)
   w BrightstarDB.Storage.BPlusTreeStore.BPlusTreeStoreManager.ActivateConsolidationStore(String storeLocation)
   w BrightstarDB.Storage.BPlusTreeStore.Store.Consolidate(Guid jobId)
   w BrightstarDB.Server.StoreWorker.Consolidate(Guid jobId)
   w BrightstarDB.Server.ConsolidateJob.Run()
   w BrightstarDB.Server.StoreWorker.ProcessJobs(Object state)
BrightstarDB Error: 400 : Error Processing Transaction System.IO.IOException: Proces nie może uzyskać dostępu do pliku, ponieważ jest on używany przez inny proces.
   w System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   w System.IO.__Error.WinIOError()
   w System.IO.FileInfo.MoveTo(String destFileName)
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.<>c__DisplayClassa.<RenameFile>b__8()
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.WrapSharingViolations(WrapSharingViolationsCallback action, WrapSharingViolationsExceptionsCallback exceptionsCallback, Int32 retryCount, Int32 waitTime)
   w BrightstarDB.Storage.Persistence.FilePersistenceManager.RenameFile(String storeConsolidateFile, String storeDataFile)
   w BrightstarDB.Storage.BPlusTreeStore.BPlusTreeStoreManager.ActivateConsolidationStore(String storeLocation)
   w BrightstarDB.Storage.BPlusTreeStore.Store.Consolidate(Guid jobId)
   w BrightstarDB.Server.StoreWorker.Consolidate(Guid jobId)
   w BA first chance exception of type 'System.NullReferenceException' occurred in BrightstarDB.dll
rightstarDB.Server.ConsolidateJob.Run()
   w BrightstarDB.Server.StoreWorker.ProcessJobs(Object state)
    DateTime=2015-02-13T06:51:59.0314458Z
Job Error
System.NullReferenceException: Odwołanie do obiektu nie zostało ustawione na wystąpienie obiektu.
   w BrightstarDB.Dto.ExceptionDetailObject.BuildString(StringBuilder sb, Int32 indentLevel)
   w BrightstarDB.Dto.ExceptionDetailObject.ToString()
   w System.String.Concat(Object arg0, Object arg1)
   w VIEWMODEL.AppState.PerformCleanup() w C:\IMPARTS\VIEWMODEL\AppState.cs:wiersz 193
The error is in Polish, it says: "process cannot access the file because it is being used by another process". And there's that NullReferenceException at the end...

At first I had the same thought you did, that maybe something else was locking the file, so I used handle.exe to check. And it's my app that holds all the locks, and the number of locks goes up after calling SaveChanges on the entity context.

Also, once in a blue moon the consolidation job completes without errors even if updates were made.

I'll try to reproduce this some time later today and then I'll post the code.
Feb 13, 2015 at 12:27 PM
Edited Mar 3, 2015 at 7:18 AM
I replicated the error in another project.

As I said in my previous post, I'm using nuget. Versions of packages (I updated them today):
<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="BrightstarDB" version="1.9.1.0" targetFramework="net40" />
  <package id="BrightstarDBLibs" version="1.9.1.0" targetFramework="net40" />
  <package id="dotNetRDF" version="1.0.7.3471" targetFramework="net40" />
  <package id="HtmlAgilityPack" version="1.4.9" targetFramework="net40" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net40" />
  <package id="VDS.Common" version="1.4.0" targetFramework="net40" />
</packages>
The first time, when the database is created, the consolidation job runs without errors. Problems start after that.

The code to replicate the error (it's a wpf app):

Model:
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using BrightstarDB.EntityFramework;

namespace graftest
{
    [Entity]
    public interface IMElement
    {
        [Key]
        string ID { get; }

        [Required(ErrorMessage = "Key is required")]
        string Key { get; set; }

        string Details { get; set; }

        ICollection<IMChild> Children { get; set; }
    }
}

namespace graftest
{
    [Entity]
    public interface IMChild
    {
        [Key]
        string ID { get; }

        [Required(ErrorMessage = "Key is required")]
        string Key { get; set; }

        string Details { get; set; }

        [InverseProperty("Children")]
        IMElement Parent { get; set; }
    }
}
...
using System;
using System.Linq;
using System.Threading;
using BrightstarDB.Client;

namespace graftest
{
    public class ContextManager
    {
        private const string storeName = "IMSTORE";
        private const string connectionString = "type=embedded;storesdirectory=.\\;storename=IMSTORE";
        public MyEntityContext CTX { get; set; }

        #region SINGLETON PROPERTY Single

        private static ContextManager _Single = null;

        public static ContextManager Single
        {
            get
            {
                return _Single = _Single ?? new ContextManager();
            }
        }

        #endregion

        // -----------------------------------------------

        #region PRIVATE CONSTRUCTOR

        private ContextManager()
        {
            BrightstarDB.Logging.EnableConsoleOutput(true);
            BrightstarDB.Logging.EnableProfiling(true);
            BrightstarDB.Logging.BrightstarTraceSource.Switch.Level = System.Diagnostics.SourceLevels.Error;
            BrightstarDB.Configuration.PersistenceType = BrightstarDB.Storage.PersistenceType.Rewrite;
            BrightstarDB.Configuration.EnableOptimisticLocking = false;
        }

        #endregion

        // -----------------------------------------------

        public void LoadData()
        {
            this.CreateContext();

            // add some random data...
            for (int i = 0; i < 10; i++)
            {
                IMElement elem = this.CTX.MElements.Create();
                elem.Key = DateTime.Now.ToLongTimeString();
                elem.Details = ""+i;

                this.CTX.MElements.Add(elem);

                for (int j = 0; j < 10; j++)
                {
                    IMChild child = this.CTX.MChilds.Create();
                    child.Key = DateTime.Now.ToLongTimeString();
                    child.Details = "" + j + " " + i;
                    this.CTX.MChilds.Add(child);

                    elem.Children.Add(child);
                }

                this.CTX.SaveChanges();
            }
        }

        private void CreateContext()
        {
            if (this.CTX == null)
            {
                try
                {
                    var client = BrightstarService.GetClient(connectionString);
                    if (!client.DoesStoreExist(storeName))
                    {
                        Console.WriteLine("Store doesn't extst. Create it.");
                        client.CreateStore(storeName, BrightstarDB.Storage.PersistenceType.Rewrite);
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                }

                this.CTX = new MyEntityContext(connectionString);
            }
        }

        internal void UpdateData()
        {
            var cds = this.CTX.MChilds.Where(a => a.Details.Contains(" 1"));
            foreach (var c in cds)
            {
                c.Details = c.Details + "u";

            }
            this.CTX.SaveChanges();
        }

        public void ConsolidateStore()
        {
            this.CTX.Dispose();
            this.CTX = null;
            BrightstarService.Shutdown(true);

            Thread.Sleep(1000);

            try
            {
                var client = BrightstarService.GetClient(connectionString);


                if (client.DoesStoreExist(storeName))
                {
                    Console.WriteLine("Starting consolidation job\n");
                    var consolidateJob = client.ConsolidateStore(storeName);
                    while (!(consolidateJob.JobCompletedOk || consolidateJob.JobCompletedWithErrors))
                    {
                        System.Threading.Thread.Sleep(500);
                        consolidateJob = client.GetJobInfo(storeName, consolidateJob.JobId);

                        Console.WriteLine(consolidateJob.StatusMessage);
                    }
                    Console.WriteLine("\nEnd of consolidation job");

                    if (consolidateJob.JobCompletedWithErrors)
                        Console.WriteLine("consolidateJob error: " + consolidateJob.ExceptionInfo);
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            BrightstarService.Shutdown(true);
        }

    }
}
...
using System.Windows;

namespace graftest
{
    /// <summary>
    /// Interaction logic for App.xaml
    /// </summary>
    public partial class App : Application
    {
        protected override void OnExit(ExitEventArgs e)
        {
            ContextManager.Single.ConsolidateStore();
            base.OnExit(e);
        }
    }
}

using System.Threading;
using System.Windows;

namespace graftest
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
            this.Loaded += new RoutedEventHandler(MainWindow_Loaded);
        }

        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            ContextManager.Single.LoadData();
            Thread.Sleep(1000);
            ContextManager.Single.UpdateData();
        }
    }
}
Mar 20, 2015 at 12:13 PM
It seems that caching was the problem. If I temporarily set BrightstarDB.Configuration.EnableQueryCache = false before consolidating it works fine.
Coordinator
Mar 20, 2015 at 1:16 PM
Thanks for letting us know - I'll take a look and see if I can work out what the caching code is locking.
Mar 23, 2015 at 10:36 AM
Edited Mar 23, 2015 at 10:38 AM
It seems my happiness was premature - while setting BrightstarDB.Configuration.EnableQueryCache = false makes consolidating execute fine in most cases, it turned out there was another thing in my app that caused the same problem.

In my application, I bind my views to ListCollectionView created for ObservableCollection of BrightstarDB entities. Then, if I edit an item and then exit the application without moving CurrentItem of my ListCollectionView to another item, then consolidation throws an exception. It is very consistent and happens every time this condition is met. I don't know what causes this behaviour (I plan to investigate). My temporary solution is:
 protected override void OnClosing(System.ComponentModel.CancelEventArgs e)
 { 
       // OnClosing must be overwritten, it doesn't work if I use the Closing event handler
        this.MyViewModel.GoToNext.Execute(null); //<-- here this.MyCollectionView.MoveCurrentToNext() is called
        base.OnClosing(e);
 }
In conclusion, setting BrightstarDB.Configuration.EnableQueryCache = false and moving my current item to next before closing the window ultimately resolved my problem.
Coordinator
Mar 23, 2015 at 11:56 AM
I used your code to reproduce the issue. It is pretty hard to figure out if/where a file handle is being left open. One thing I have noticed is that if the store already exists when the app starts up, then the consolidation works (at least in my test runs); it is only when the code tries to create the store for the first time that the consolidation fails. Do you see the same thing?

I'm not sure if the problem with the ObservableCollection is related - it sounds like it might be a separate issue from the consolidate (but obviously both these things affect your application's shutdown).

I'm going to keep digging into the consolidation issue without trying to replicate the ListViewCollection issue yet, but if you have some simple code for that you could share that would be great!

Cheers

Kal
Mar 23, 2015 at 1:30 PM
  1. when store is newly created, then consolidation fails no matter what (99% of the time anyway)
  2. when store already exists, consolidation fails if BrightstarDB.Configuration.EnableQueryCache = true and entities are ADDED, REMOVED or UPDATED; when cache is disabled before consolidating, consolidation will run OK in this scenario
  3. consolidation fails (even if cache is disabled) when existing item is UPDATED and ListCollectionView.CurrentItem is not moved before running the consolidation code (in my app add/remove moves the CurrentItem anyway); the error logged by BrightstarDB is exactly the same as in 1 and 2 situation, meaning the file is being locked
Coordinator
Mar 23, 2015 at 3:47 PM
I think I found the cause and have committed a fix for it.

The problem was that when a store gets updated we replace the old "read store" (really an object that manages store read accesses) with a new one. The old one isn't disposed at this point as it may be being used by other threads (such as a long running export) that we don't want to force to halt and restart just because a new commit has occurred. Instead we just remove the hard reference from the manager object and let the GC dispose of the old read store once all users of it have released their references.

This is all fine until you actually really do want all file handles to be closed (like when you do a consolidate) - because GC may not have yet disposed the old read store instances. This dependency on GC to run explains why the error was not consistently occurring (sometimes you will just get lucky). I'm not sure if it explains your issue #3, but it could do - again it could just be that the extra operation is enough to push the GC into getting rid of the old read store objects.

I've changed the code so that the manager object puts a weakref to the old read store object into a list and when you come to do that final "real" close (on store shutdown or just before a consolidate operation), it goes through the list and calls the Dispose() method on any objects that are still live. This fixes the issue I reproduced using your test code - at least it works for me :) - it will go into the 1.10 release which should be any day now - or you can build yourself from the code on the develop branch in GitHub.