This project has moved and is read-only. For the latest updates, please go here.

"Random" Exceptions

May 10, 2016 at 2:42 AM
Edited May 10, 2016 at 2:58 AM
I am running b* in a ASP.Net 4.5 context. The store is embedded and rewrite.
I am seeing random exceptions and I cannot quite pinpoint the cause.

The application will run for an hour maybe three and then boom.
I will capture a couple different scenarios and then maybe I will get a bigger picture.

Here is the first exception of NullReferenceException:
   at BrightstarDB.Storage.BPlusTreeStore.Store.MakeTriple(Tuple`4 data)
   at System.Linq.Enumerable.<>c__DisplayClass12`3.<CombineSelectors>b__11(TSource x)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at VDS.RDF.Query.Algebra.LazyBgp.StreamingEvaluate(SparqlEvaluationContext context, Int32 pattern, Boolean& halt) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\LazyBgp.cs:line 283
   at VDS.RDF.Query.Algebra.LazyBgp.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\LazyBgp.cs:line 165
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessBgp(IBgp bgp, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 543
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 415
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.Select.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\Project.cs:line 85
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSelect(Select select, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 745
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 487
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.Slice.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\Slice.cs:line 108
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSlice(Slice slice, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 778
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 499
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.SubQuery.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\SubQuery.cs:line 87
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSubQuery(SubQuery subquery, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 790
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 503
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.LeftJoin.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\AlgebraJoinClasses.cs:line 270
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessLeftJoin(ILeftJoin leftJoin, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 653
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 455
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(IRdfHandler rdfHandler, ISparqlResultsHandler resultsHandler, SparqlQuery query) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 184
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(SparqlQuery query) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 108
   at BrightstarDB.Query.SparqlQueryHandler.ExecuteSparql(SparqlQuery query, IStore store, TextWriter resultsWriter)
Its context:
public static IThing GetThing(string thingId)
        {
            IsThingIdFormatValid(thingId, true);

            IThing thing = null;

            using (var ctx = new KPEntityContext(KPEntityContext.DefaultConnectionString))
            {
                thing = ctx.Things.FirstOrDefault(t => t.Identifier.Equals(thingId));

                if (thing == null)
                {
                    logger.Warn("GetThing: Thing '{0}' was not found.", thingId);
                    throw new ThingNotFoundException(string.Format("'{0}' thing not found.", thingId), null);
                }
            }

            return thing;
        }
Not Flagged     10496   1   Worker Thread   <No Name>       Normal
Not Flagged     12092   8   Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged     16292   6   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Server.StoreWorker.ProcessJobs    Normal
Not Flagged     7908    0   Worker Thread   GC Finalizer Thread     Normal
Not Flagged     5812    9   Worker Thread   [Thread Destroyed]      Normal
Not Flagged     28128   10  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged >   10416   12  Worker Thread   Worker Thread   dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery    Normal
Not Flagged     144 13  Worker Thread   Worker Thread   System.Web.dll!System.Web.HttpApplication.AssignContext Normal
Not Flagged     8836    15  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged     4072    16  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged     23696   17  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged     23744   18  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Things.LearnFromOpc    Normal
Not Flagged     20908   19  Worker Thread   Worker Thread   System.ComponentModel.Composition.dll!System.ComponentModel.Composition.Hosting.TypeCatalog.GetCandidateParts   Normal
Not Flagged     21868   20  Worker Thread   <No Name>   System.dll!System.Net.TimerThread.ThreadProc    Normal
Not Flagged     37308   23  Worker Thread   Worker Thread       Normal
May 10, 2016 at 2:49 AM
An attempt was made to move the file pointer before the beginning of the file.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.SeekCore(Int64 offset, SeekOrigin origin)
   at System.IO.FileStream.Seek(Int64 offset, SeekOrigin origin)
   at BrightstarDB.Storage.Persistence.FilePage..ctor(Stream stream, UInt64 pageId, Int32 pageSize)
   at BrightstarDB.Storage.Persistence.AppendOnlyFilePageStore.Retrieve(UInt64 pageId, BrightstarProfiler profiler)
   at BrightstarDB.Storage.BPlusTreeStore.ResourceTable.GetResource(UInt64 pageId, Byte segment, BrightstarProfiler profiler)
   at BrightstarDB.Storage.BPlusTreeStore.ResourceIndex.LongUriResource.get_Value()
   at BrightstarDB.Storage.BPlusTreeStore.ResourceIndex.LongUriResource.Matches(String resourceValue, Boolean isLiteral, UInt64 dataTypeId, UInt64 langCodeId)
   at BrightstarDB.Storage.BPlusTreeStore.ResourceIndex.ResourceIndex.FindResourceInBTree(String resourceValue, Boolean isLiteral, UInt64 dataTypeId, UInt64 langCodeId, UInt32 hashCode)
   at BrightstarDB.Storage.BPlusTreeStore.ResourceIndex.ResourceIndex.GetResourceId(String resourceValue, Boolean isLiteral, String dataType, String langCode, Boolean addToCache, BrightstarProfiler profiler)
   at BrightstarDB.Storage.BPlusTreeStore.Store.FindResourceId(String resourceValue, Boolean isLiteral, String dataType, String langCode)
   at BrightstarDB.Storage.BPlusTreeStore.Store.Match(String subject, String predicate, String obj, Boolean isLiteral, String dataType, String langCode, IEnumerable`1 graphs)
   at BrightstarDB.Query.StoreSparqlDataset.GetTriplesWithPredicateObject(INode pred, INode obj)
   at VDS.RDF.Query.Patterns.TriplePattern.GetTriples(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Patterns\TriplePattern.cs:line 425
   at VDS.RDF.Query.Algebra.LazyBgp.StreamingEvaluate(SparqlEvaluationContext context, Int32 pattern, Boolean& halt) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\LazyBgp.cs:line 256
   at VDS.RDF.Query.Algebra.LazyBgp.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\LazyBgp.cs:line 165
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessBgp(IBgp bgp, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 543
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 415
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.Select.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\Project.cs:line 85
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSelect(Select select, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 745
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 487
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.Slice.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\Slice.cs:line 108
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSlice(Slice slice, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 778
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 499
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.SubQuery.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\SubQuery.cs:line 87
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessSubQuery(SubQuery subquery, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 790
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 503
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.Algebra.LeftJoin.Evaluate(SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\Algebra\AlgebraJoinClasses.cs:line 270
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessLeftJoin(ILeftJoin leftJoin, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 653
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(ISparqlAlgebra algebra, SparqlEvaluationContext context) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 455
   at VDS.RDF.Query.SparqlEvaluationContext.Evaluate(ISparqlAlgebra algebra) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\SparqlEvaluationContext.cs:line 403
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(IRdfHandler rdfHandler, ISparqlResultsHandler resultsHandler, SparqlQuery query) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 184
   at VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(SparqlQuery query) in e:\mercurial\dotnetrdf\Libraries\core\net40\Query\LeviathanQueryProcessor.cs:line 108
   at BrightstarDB.Query.SparqlQueryHandler.ExecuteSparql(SparqlQuery query, IStore store, TextWriter resultsWriter)
public static IConfiguration Configuration
        {
            get
            {
                lock (_locker)
                {
                    IConfiguration configuration = null;

                    using (var ctx = new KPEntityContext(KPEntityContext.DefaultConnectionString))
                    {
                        configuration = ctx.Configurations.FirstOrDefault(); //TODO: figure out why fails during certain times
                    }

                    return configuration;
                }
            }
        }
Not Flagged     39776   1   Worker Thread   <No Name>       Normal
Not Flagged     27836   5   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Server.StoreWorker.ProcessJobs    Normal
Not Flagged >   21992   6   Worker Thread   Worker Thread   dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery    Normal
Not Flagged     10112   0   Worker Thread   GC Finalizer Thread     Normal
May 10, 2016 at 1:52 PM
Edited May 10, 2016 at 2:20 PM
I turned on b* console logging, here is a little more info:

NullReferenceException
BrightstarDB Information: 0 : Query kp_data ASK  { <http://www.brightstardb.com/.well-known/genid/f2cdf847-52ce-4297-87ba-b5bad88c1373> a <http://mr.iiot.com/kepwareProxy/physicalEntity>. }
BrightstarDB Information: 0 : Query kp_data ASK  { <http://www.brightstardb.com/.well-known/genid/28158ba6-a333-4672-b921-f47fc5ee249b> a <http://mr.iiot.com/kepwareProxy/virtualEntity>. }
BrightstarDB Information: 0 : GetResourceStatements http://www.brightstardb.com/.well-known/genid/f2cdf847-52ce-4297-87ba-b5bad88c1373
BrightstarDB Information: 0 : GetResourceStatements http://www.brightstardb.com/.well-known/genid/28158ba6-a333-4672-b921-f47fc5ee249b
BrightstarDB Information: 0 : ProcessTransaction
BrightstarDB Information: 0 : Queueing Job Id b1c9ed73-687e-40d4-87d0-cbd9dce876ac
BrightstarDB Information: 0 : Queued Job Id b1c9ed73-687e-40d4-87d0-cbd9dce876ac
BrightstarDB Information: 0 : Job found b1c9ed73-687e-40d4-87d0-cbd9dce876ac
BrightstarDB Information: 0 : GuardedUpdateTransaction b1c9ed73-687e-40d4-87d0-cbd9dce876ac - processing preconditions
BrightstarDB Information: 0 : GuardedUpdateTransaction b1c9ed73-687e-40d4-87d0-cbd9dce876ac - processing deletes
BrightstarDB Information: 0 : Delete Triple http://www.brightstardb.com/.well-known/genid/f2cdf847-52ce-4297-87ba-b5bad88c1373 http://mr.iiot.com/kepwareProxy/representedBy http://www.brightstardb.com/.well-known/genid/28158ba6-a333-4672-b921-f47fc5ee249b False   http://www.brightstardb.com/.well-known/model/wildcard
BrightstarDB Error: 411 : Unexpected exception caught in GuardedUpdateTransaction (b1c9ed73-687e-40d4-87d0-cbd9dce876ac): System.NullReferenceException: Object reference not set to an instance of an object.
   at BrightstarDB.Rdf.NTriplesParser.Parse(TextReader reader, ITripleSink sink, String defaultGraphUri)
   at BrightstarDB.Server.GuardedUpdateTransaction.Run()
BrightstarDB Error: 400 : Error Processing Transaction System.NullReferenceException: Object reference not set to an instance of an object.
   at BrightstarDB.Rdf.NTriplesParser.Parse(TextReader reader, ITripleSink sink, String defaultGraphUri)
   at BrightstarDB.Server.GuardedUpdateTransaction.Run()
   at BrightstarDB.Server.StoreWorker.ProcessJobs(Object state)
A first chance exception of type 'BrightstarDB.Client.BrightstarClientException' occurred in BrightstarDB.dll
Not Flagged     15244   1   Worker Thread   <No Name>       Normal
Not Flagged     7948    7   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Server.StoreWorker.ProcessJobs    Normal
Not Flagged     13644   8   Worker Thread   Worker Thread   System.dll!System.Net.Sockets.Socket.Receive    Normal
Not Flagged     45836   0   Worker Thread   GC Finalizer Thread     Normal
Not Flagged     24440   9   Worker Thread   Worker Thread   System.dll!System.Net.Sockets.Socket.Receive    Normal
Not Flagged     18332   10  Worker Thread   <No Name>       Normal
Not Flagged     27900   11  Worker Thread   Worker Thread   System.dll!System.Net.Sockets.Socket.Receive    Normal
Not Flagged     8880    14  Worker Thread   Worker Thread       Normal
Not Flagged     25028   15  Worker Thread   Worker Thread   System.dll!System.Net.Sockets.Socket.Receive    Normal
Not Flagged     34920   0   Worker Thread   Worker Thread       Normal
Not Flagged     20336   5   Worker Thread   Worker Thread       Normal
Not Flagged >   23180   20  Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Client.EmbeddedDataObjectStore.DoSaveChanges  Normal
Not Flagged     5560    21  Worker Thread   Worker Thread       Normal
Not Flagged     15952   22  Worker Thread   <No Name>   System.dll!System.Net.TimerThread.ThreadProc    Normal
Not Flagged     43144   0   Worker Thread   Worker Thread       Normal
Not Flagged     46168   23  Worker Thread   Worker Thread       Normal
May 10, 2016 at 2:22 PM
StackOverflowException

Stack:
    [External Code] 
    dotNetRDF.dll!VDS.RDF.Query.Patterns.TriplePattern.FindResults(VDS.RDF.Query.SparqlEvaluationContext context, System.Collections.Generic.IEnumerable<VDS.RDF.Triple> ts) Line 562   C#
    dotNetRDF.dll!VDS.RDF.Query.Patterns.TriplePattern.Evaluate(VDS.RDF.Query.SparqlEvaluationContext context) Line 271 C#
    dotNetRDF.dll!VDS.RDF.Query.Algebra.Bgp.Evaluate(VDS.RDF.Query.SparqlEvaluationContext context) Line 112    C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessBgp(VDS.RDF.Query.Algebra.IBgp bgp, VDS.RDF.Query.SparqlEvaluationContext context) Line 543  C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(VDS.RDF.Query.Algebra.ISparqlAlgebra algebra, VDS.RDF.Query.SparqlEvaluationContext context) Line 415    C#
    dotNetRDF.dll!VDS.RDF.Query.SparqlEvaluationContext.Evaluate(VDS.RDF.Query.Algebra.ISparqlAlgebra algebra) Line 403 C#
    dotNetRDF.dll!VDS.RDF.Query.Algebra.LeftJoin.Evaluate(VDS.RDF.Query.SparqlEvaluationContext context) Line 286   C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessLeftJoin(VDS.RDF.Query.Algebra.ILeftJoin leftJoin, VDS.RDF.Query.SparqlEvaluationContext context) Line 653   C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessAlgebra(VDS.RDF.Query.Algebra.ISparqlAlgebra algebra, VDS.RDF.Query.SparqlEvaluationContext context) Line 455    C#
    dotNetRDF.dll!VDS.RDF.Query.SparqlEvaluationContext.Evaluate(VDS.RDF.Query.Algebra.ISparqlAlgebra algebra) Line 403 C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(VDS.RDF.IRdfHandler rdfHandler, VDS.RDF.ISparqlResultsHandler resultsHandler, VDS.RDF.Query.SparqlQuery query) Line 169    C#
    dotNetRDF.dll!VDS.RDF.Query.LeviathanQueryProcessor.ProcessQuery(VDS.RDF.Query.SparqlQuery query) Line 108  C#
    [External Code] 
>   KepwareProxy.Data.dll!KP.ServiceEndpoints.GetServiceEndpoint(string ipAddress) Line 74  C#
    KepwareProxy.Engine.dll!KP.scripting.ActionScriptingEngine.Dispatch(string userHostAddress, KP.data.dto.IOpcItemsPublished opcItems) Line 30    C#
    KepwareProxy.Engine.dll!KP.IIOTService.Dispatch(string fullScriptPath, string userHostAddress, KP.data.dto.IOpcItemsPublished opcItems) Line 30 C#
    KepwareProxy.WebApi.dll!KP.web.Controllers.httpServerController.IngestMessage(System.Threading.CancellationToken ct) Line 40    C#
    [Resuming Async Method] 
    [External Code] 
Threads:
Not Flagged     4056    1   Worker Thread   <No Name>       Normal
Not Flagged     19952   5   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Server.StoreWorker.ProcessJobs    Normal
Not Flagged     34368   6   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Storage.BPlusTreeStore.RelatedResourceIndex.RelatedResourceIndex.GetPredicateIndex    Normal
Not Flagged     16892   0   Worker Thread   GC Finalizer Thread     Normal
Not Flagged     9616    8   Worker Thread   Worker Thread       Normal
Not Flagged >   39800   9   Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Storage.Persistence.LruPageCache.MakeCacheKey Normal
Not Flagged     17144   15  Worker Thread   Worker Thread       Normal
Not Flagged     7204    12  Worker Thread   Worker Thread       Normal
Not Flagged     32240   13  Worker Thread   Worker Thread   KepwareProxy.Data.dll!KP.Configurations.Configuration.get   Normal
Not Flagged     30800   14  Worker Thread   <No Name>   System.dll!System.Net.TimerThread.ThreadProc    Normal
Not Flagged     18344   0   Worker Thread   Worker Thread       Normal
Not Flagged     2632    16  Worker Thread   Worker Thread       Normal
Not Flagged     14808   18  Worker Thread   Worker Thread       Normal
Not Flagged     12148   19  Worker Thread   Worker Thread       Normal
Not Flagged     42512   20  Worker Thread   Worker Thread   BrightstarDB.dll!BrightstarDB.Storage.BPlusTreeStore.RelatedResourceIndex.RelatedResourceIndex.GetPredicateIndex    Normal
Not Flagged     27828   0   Worker Thread   Worker Thread       Normal
May 10, 2016 at 2:36 PM
Hi Chris,

Thanks for the information and stack traces.
Can you confirm if the database is scrambled or if it is just something in memory. For example, if you stop the server, restart and then export all of the data from the store, does the export job run through to completion without error ? I have occasionally hit similar weird exceptions but always found that restarting the server clears the error That's not ideal I know and it also makes it really hard to debug the root cause; but I'd be happier knowing that your data is still safe on the disk! I suspect there maybe some corner-case bug in the page caching that is somehow corrupting the page data in memory.

One other thing, are you running concurrent readers and/or writers when the bug shows up or is your application just a single threaded read loop ?

Cheers

Kal
May 10, 2016 at 3:26 PM
I am installing VS2015 and setting up the 'Core' project so that I can take a look in your code.

A restart clears it up, until it shows up again.
The ASP.NET app does have concurrent readers and writers, but from a couple of the exceptions I only see a single thread accessing the db.

/c
May 10, 2016 at 3:34 PM
Yes, I fear it is the same mandlebug that I've had problems tracking down. Any help / insights you can provide would be really most welcome!

Cheers

Kal
May 11, 2016 at 8:12 PM
I have upgraded my project to VS2015, 4.5.2, built BrightstarDB from source and referenced to all projects in solution.

List of packages:
<packages>
  <package id="Antlr" version="3.5.0.2" targetFramework="net452" />
  <package id="bootstrap" version="3.0.0" targetFramework="net452" />
  <package id="dotNetRDF" version="1.0.9.3683" targetFramework="net452" />
  <package id="HtmlAgilityPack" version="1.4.9" targetFramework="net452" />
  <package id="jQuery" version="2.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.Mvc" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.Razor" version="3.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.Web.Optimization" version="1.1.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Core" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.HelpPage" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.WebHost" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebPages" version="3.2.3" targetFramework="net452" />
  <package id="Microsoft.jQuery.Unobtrusive.Ajax" version="3.2.3" targetFramework="net452" />
  <package id="Microsoft.Web.Infrastructure" version="1.0.0.0" targetFramework="net452" />
  <package id="Modernizr" version="2.8.3" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="8.0.2" targetFramework="net452" />
  <package id="NLog" version="4.3.3" targetFramework="net452" />
  <package id="NLog.Config" version="4.3.3" targetFramework="net452" />
  <package id="NLog.Schema" version="4.3.0" targetFramework="net452" />
  <package id="Respond" version="1.4.2" targetFramework="net452" />
  <package id="VDS.Common" version="1.6.0" targetFramework="net452" />
  <package id="WebGrease" version="1.6.0" targetFramework="net452" />
</packages>
Application ran for about 2 hours and started throwing exceptions that it could not find resource id xxxx.
Still seeing lots of these: Exception thrown: 'System.ObjectDisposedException' in mscorlib.dll

Changes:
  • disabled ALL writers
  • concurrent readers ONLY
  • removed a throw from within a using b* context statement
Will report back in 24 hours.
May 11, 2016 at 8:58 PM
Hi Chris,

Now you have it built from source, one thing you can try is disabling the page cache. If the problem doesn't show up when the page caching is turned off then that could point to it as the culprit (although I suppose it could also mean that there is some race condition that only gets triggered when you don't have to hit the disk for every page).

The easiest way to do that is to create a no-op implementation of the BrightstarDB.Storage.Persistence.IPageCache interface and then change BrightstarDB.Storage.Persistence.PageCache so that an instance of your no-op page cache is returned instead of the LRU one.

The System.ObjectDisposedException is a bit concerning.

If you have some simple solution that you can share which I can use to replicate the problem that would be a great help!

Cheers

Kal
May 12, 2016 at 6:07 PM
Ok, still running with no problems with concurrent reads and no writes.

Tomorrow I will try disabling the page cache and breaking it again.

At some point I will have to develop a sample solution.... probably sooner than later, since I have 10 more days to resolve this issue.

/c
May 14, 2016 at 2:11 AM
Edited May 14, 2016 at 2:18 AM
Kal, just to make sure before I proceed. Is this what you had in mind ?

Because with this implementation, the creation of the store fails and I get even more exceptions than with LRU impl.
internal class NoopPageCache : IPageCache
    {
        public int FreePages
        {
            get
            {
                return 0;
            }
        }

        public event PostEvictionDelegate AfterEvict
        {
            add { }
            remove { }
        }

        public event PreEvictionDelegate BeforeEvict
        {
            add { }
            remove { }
        }

        public event EvictionCompletedDelegate EvictionCompleted
        {
            add { }
            remove { }
        }

        public void Clear()
        {
            
        }

        public void Clear(string partition)
        {
            
        }

        public void InsertOrUpdate(string partition, IPageCacheItem page)
        {
            
        }

        public IPageCacheItem Lookup(string partition, ulong pageId)
        {
            return null;
        }
    }
May 14, 2016 at 7:33 AM
Hi Chris,

Yes that looks like the sort of thing I mean. One issue though will be that no cache will obviously slow down page access and so might lead to inconclusive results. If you have enough memory on the machine you might consider a dead simple cache implementation using either a concurrent dictionary or a regular dictionary with a lock. Either way you can omit all of the cache expiration stuff if you are able to.load the entire db to memory (let's say if dbsize < memory/2).

I'm at a conference this weekend so won't get the chance to try this out myself I'm afraid , sorry! Please let me know how it goes and I hope to be able to do some real digging on this next week.

Cheers

Kal
May 16, 2016 at 12:53 PM
Edited May 16, 2016 at 4:48 PM
Hi Kal, hope you had a good weekend.

I do want to get into your codebase more but it will have to wait after we can stabilize the current condition. And if messing with page cache is the solution then I will need some guidance.

ObjectDisposedException source

I created a small web app that reproduces most of what I have been experiencing.
project
  1. Create c:\bstar
  2. Start it up
  3. Hammer: GET http://yourhost:yourport/api/api1
  4. You will end up throwing soon
State:
Looks like concurrent readers.
OUTPUT
StoreWorker.ProcessJobs
T7 bstar
T7 user
T8 bstar
T8 user

Image

/c
May 18, 2016 at 5:16 PM
Edited May 18, 2016 at 5:17 PM
Hi Chris,

Thanks for this - it's been really useful to have a test case that I can hammer on while trying to debug the issue.

I've made some progress on this today and I think that I have got a solution. I'll report back tomorrow after I've done a bit more testing and run some regression tests to make sure that my "fix" hasn't broken anything else!

Cheers

Kal
May 19, 2016 at 11:49 AM
Hi again,

I've pushed fixes to the develop branch in this commit: https://github.com/BrightstarDB/BrightstarDB/commit/e5f124b238e76f0231d1c4c0051531d7b246b53b

I rewrote your test app into a simple console-based app for easier testing and then left it to hammer itself for a couple of hours without the concurrency issues showing up. In terms of changes made I think there were actually two bugs, one was to do with concurrent access to the read stream - a potential race condition where threads overlap on the seek and read operations; the other was to do with caching which is complicated in the re-write store because each cached page actually holds both the "real" data and the shadow copy but only provides access to one of those and this can cause problems when a reads and writes overlap as the writer changes the data exposed to the reader - this was solved by separating out the cache partition used by the writer so that it is never possible for it to be accessed by a reader (which is a slight performance hit but better that than incorrect results).

I also added a bit more debug and checking code - primarily this is just in the Debug build, but I did add some post-load sanity checking that is in there in both Release and Debug builds.

Anyway, as the code is now all in the develop branch on GitHub and I would really appreciate it if you can give it a spin with your app and let me know how it goes!

Cheers

Kal
May 23, 2016 at 2:19 AM
Thanks Kal. Sorry for the late reply. Been busy rolling out ERP at the day job.

I will begin testing tomorrow!

/chris
May 23, 2016 at 1:22 PM
Kal.

It is running well this morning! I will keep it running for a couple days.

One question I had is how do I correctly implement store consolidation? Right now I am consolidating on application startup and then every 24hours. The consolidation job on start up usually errors out or causes other exceptions...

Thanks
Chris
May 23, 2016 at 1:30 PM
Damn, spoke to soon.

I was letting the application run, getting hammered from a couple different directions and at the same time I was deleting objects through the UI.

Exception

Output

Threads

What do you need from me?

/c
Jun 9, 2016 at 12:25 AM
Kal:

were you able to replicate?

/chris
Jun 9, 2016 at 9:16 AM
Hi Chris,

Sorry - I missed your previous post on this one somehow.

I'll have a look to see if I can replicate - I'm assuming that the hammering you were doing was more or less the same as the sample code you provided before ?

Also, when you say you are using the UI to delete objects, is the UI connecting to the service (i.e. using an http connection?) or directly to the store files (i.e. using an embedded connection)?
Jun 11, 2016 at 3:21 PM
Same way, yes. UI is connecting directly to an embedded connection.

The other part of the question is : when is a good time to consolidate store? should I bring the application down for maintenance?

Thanks
Chris
Jun 13, 2016 at 9:09 AM
Hi Chris,

BrightstarDB doesn't do any file-level locking - the assumption is that only a single process is accessing the database at any given time (this is partly for performance but really mostly just to keep things simple). So it is not recommended to have two embedded connections to the same database open at the same time. The server is using an embedded connection of course, plus then Polaris is making a second embedded connection to the store. It will work fine if one of the connections is only doing reads, but if you try and do writes from two embedded connections at the same time you are likely to hit errors like the one you reported.

In terms of consolidating the store, a consolidation job can run at any time - it will block writers, but readers can work concurrently with the consolidation process.

Cheers

Kal
Jun 15, 2016 at 1:54 AM
Kal,


Sorry, by UI I meant my own UI (not Polaris), so it is the same process doing writes, reads and deletes. Your fix https://github.com/BrightstarDB/BrightstarDB/commit/e5f124b238e76f0231d1c4c0051531d7b246b53b did resolve reads and writes, but the last error report is when there are reads + writes + deletes.

I have not been very successful with the consolidation process... but lets leave that for another day.

Do you have the console app that you are using to test, maybe I can reproduce the delete errors in there?

Chris

PS. Codeplex email notifications have not been working very well lately.
Jul 1, 2016 at 10:22 AM
Hi Chris,

Sorry for the long delay in replying.

I've added a repository on GitHub that contains the console app that I was using to test. Perhaps you could take a look and suggest how to modify it to replicate your concurrency problems ?

The repository is https://github.com/BrightstarDB/ConcurrencyIssues

Cheers

Kal
Aug 9, 2016 at 7:05 PM
I have not forgotten about this. I better get to this sooner than later.

/c