This project has moved and is read-only. For the latest updates, please go here.

Storesize ?

Jul 15, 2014 at 2:40 PM
Hello,

is the size of the store big or OK ?

I have a db with 3 fields (Id, Name, Birthday) and 1500000 records. The data.bs has 1174 MB and the transactions.bs has 592 MB.

Why is the transactions.bs so big, I think the data.bs is the BTree ?

Thanks.
Jul 20, 2014 at 4:48 PM
Transactions.bs is the transaction history for the store. It is only a history file. It can be replayed to regenerate the store and it would probably be the basis for performing replication between BrightstarDB nodes when we get to that. However it is not used in normal single-server operations. As I mentioned before this is something that I'm considering allowing you to disable for stores where you want to minimize disk space used. This would have a small impact on performance (avoiding a couple of extra disk writes), but would obviously be a big saving on space.
Jul 20, 2014 at 6:20 PM
Hello,

I have seen, the store is not only the B+Tree, it's a container for Graph, PrefixManager, ResourceIndex, SubjectRelatedResourceIndex and ObjectRelatedResourceIndex.
Would it not be better, to separate Files.

I'm afraid if the store has more then 1TB (or more) and can no longer be handled.

May be it is better, to spit to store is separate files !?

Regards
Jul 21, 2014 at 3:04 PM
Has every page in the store 4 KByte ?
Jul 21, 2014 at 8:18 PM
Edited Jul 21, 2014 at 8:22 PM
I have done some tests.

In this test, I have added one entity with 2 attributes.
            string matStr = "22771";
            mat = new Material(session)
            {
                Nummer = matStr,
                Kurzbezeichnung = "Logic " + matStr
            };
            session.SaveChanges();
I have logged the FilePages and removed the SetData for PageID 93, too many writes.
log_20140721 0740.txt (304 hits)
    
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 340
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 320
    ...
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 4 - Pos : 0

    BrightstarDB Information: 0 : FilePage SetData : 95 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 340
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 320
    ...
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 40
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 4 - Pos : 0
    
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 360
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 340
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 320
    ...
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 100
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 94 - len : 4 - Pos : 0

    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 360
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 340
    ...
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 220
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 20 - Pos : 200
    BrightstarDB Information: 0 : FilePage SetData : 95 - len : 4 - Pos : 0

    BrightstarDB Information: 0 : FilePage SetData : 96 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 160
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 140
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 120
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 100
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 20 - Pos : 40
    BrightstarDB Information: 0 : FilePage SetData : 96 - len : 4 - Pos : 0
    
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 160
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 140
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 120
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 100
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 20 - Pos : 40
    BrightstarDB Information: 0 : FilePage SetData : 97 - len : 4 - Pos : 0
    
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 160
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 140
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 120
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 100
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 20 - Pos : 40
    BrightstarDB Information: 0 : FilePage SetData : 98 - len : 4 - Pos : 0

    BrightstarDB Information: 0 : FilePage SetData : 99 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 160
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 140
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 120
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 100
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 80
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 60
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 40
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 20 - Pos : 20
    BrightstarDB Information: 0 : FilePage SetData : 99 - len : 4 - Pos : 0
    
    BrightstarDB Information: 0 : FilePage SetData : 100 - len : -1 - Pos : 0
    
    BrightstarDB Information: 0 : FilePage SetData : 101 - len : -1 - Pos : 0

    BrightstarDB Information: 0 : FilePage SetData : 102 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 102 - len : 8 - Pos : 1948
    BrightstarDB Information: 0 : FilePage SetData : 102 - len : 8 - Pos : 1956
    BrightstarDB Information: 0 : FilePage SetData : 102 - len : 8 - Pos : 1964
    
    BrightstarDB Information: 0 : FilePage SetData : 103 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 103 - len : 8 - Pos : 1948
    BrightstarDB Information: 0 : FilePage SetData : 103 - len : 8 - Pos : 1956
    BrightstarDB Information: 0 : FilePage SetData : 103 - len : 8 - Pos : 1964

    BrightstarDB Information: 0 : FilePage SetData : 104 - len : -1 - Pos : 0
    BrightstarDB Information: 0 : FilePage SetData : 104 - len : -1 - Pos : 128
    

log_20140721 0740.txt (18 hits)
    BrightstarDB Information: 0 : FilePage Write : 93 - Size : 4096 - Pos : 376832
    BrightstarDB Information: 0 : FilePage Write : 94 - Size : 4096 - Pos : 380928
    BrightstarDB Information: 0 : FilePage Write : 95 - Size : 4096 - Pos : 385024
    BrightstarDB Information: 0 : FilePage Write : 94 - Size : 4096 - Pos : 380928
    BrightstarDB Information: 0 : FilePage Write : 95 - Size : 4096 - Pos : 385024
    BrightstarDB Information: 0 : FilePage Write : 93 - Size : 4096 - Pos : 376832
    BrightstarDB Information: 0 : FilePage Write : 96 - Size : 4096 - Pos : 389120
    BrightstarDB Information: 0 : FilePage Write : 97 - Size : 4096 - Pos : 393216
    BrightstarDB Information: 0 : FilePage Write : 93 - Size : 4096 - Pos : 376832
    BrightstarDB Information: 0 : FilePage Write : 98 - Size : 4096 - Pos : 397312
    BrightstarDB Information: 0 : FilePage Write : 99 - Size : 4096 - Pos : 401408
    BrightstarDB Information: 0 : FilePage Write : 102 - Size : 4096 - Pos : 413696
    BrightstarDB Information: 0 : FilePage Write : 102 - Size : 4096 - Pos : 413696
    BrightstarDB Information: 0 : FilePage Write : 103 - Size : 4096 - Pos : 417792
    BrightstarDB Information: 0 : FilePage Write : 103 - Size : 4096 - Pos : 417792
    BrightstarDB Information: 0 : FilePage Write : 100 - Size : 4096 - Pos : 405504
    BrightstarDB Information: 0 : FilePage Write : 101 - Size : 4096 - Pos : 409600
    BrightstarDB Information: 0 : FilePage Write : 104 - Size : 4096 - Pos : 421888
The store grew by 48KB, exact the 12 FilePages !

Regards.
Jul 29, 2014 at 1:59 PM
CyborgDE wrote:
Hello,

I have seen, the store is not only the B+Tree, it's a container for Graph, PrefixManager, ResourceIndex, SubjectRelatedResourceIndex and ObjectRelatedResourceIndex.
Would it not be better, to separate Files.

I'm afraid if the store has more then 1TB (or more) and can no longer be handled.

May be it is better, to spit to store is separate files !?

Regards
Definitely it would have been better if we had started off that way. Unfortunately we didn't do that. I think in the long term that will need to change, but it would be a change to store format which I want to minimize to doing only for major releases - so I would want to combine thinking about this with also thinking about other forms of indexes and on-disk storage, including the ideas you have proposed. I think it would be better to take a step back from all these issues and look at it from the high-level again as part of design for a 2.0 release.

Cheers

Kal
Jul 29, 2014 at 2:02 PM
CyborgDE wrote:
Has every page in the store 4 KByte ?
By default yes - though the code has been written to allow that to be changed I think at the moment the only way to change it is in the code itself and then the datastore would only be compatible with that version of the code. This again is a thing I would like to change - it would be nice to be able to configure the page size for the stores - especially for rewriteable stores.
Jul 29, 2014 at 2:09 PM
CyborgDE wrote:
I have done some tests.

In this test, I have added one entity with 2 attributes.
            string matStr = "22771";
            mat = new Material(session)
            {
                Nummer = matStr,
                Kurzbezeichnung = "Logic " + matStr
            };
            session.SaveChanges();
I have logged the FilePages and removed the SetData for PageID 93, too many writes.

<snip/>
The store grew by 48KB, exact the 12 FilePages !

Regards.
This is with an append-only store ? Every node in the BTree needs to be rewritten and your code is actually inserting three triples (one for the object type, one for each property) and each triple is inserted into the two index BTrees, so 12 pages sounds about right.

93 writes sounds bad, but this is because you are actually tracking each SetData call, and SetData is the way we do the low-level manipulation of the BTree node's arrays (we are operating directly on the byte[] of the page). There is an overhead here because currently SetData is the way we track that a page is changed and ready for queuing to the background writer thread - it would probably be better if we just notified the change when we are done updating the node. However just because we queue a page to the background writer 7 or 8 times doesn't necessarily mean that the disk is getting hit 7 or 8 times, the background writer - a page already queued to write can't be queued up a second time, it will always just write its latest state when it is processed by the background writer thread.