Managing incremental counters in mnesia DBMS?

Question

I have realised that mnesia doesnot support auto-increment feature as does MySQL or other RDBMS do.The counters talked about in mnesia documentation are not really well explained. forexample i have found sofar one function in the entire documentation which manipulates counters

mnesia:dirty_update_counter({Tab::atom(),Key::any()}, Val::positive_integer())

So, this has disturbed me for a time coz it works with records of type

{TabName, Key, Integer}

This is also unclear and possibly because no erlang book or mnesia documentation provides an example to explain it.This has forced me to implement my own counter manipulation APIs.Since i wanted to be able to access and manage my records using counters, i had to include a field called 'counter' in my records, then at the point of adding the record in the table which is intended to have counters, i do it like this:

#recordname{field1 = Val1,...,counter = auto_increment(?THIS_TABLE)}

The position of the counter fields doesnot matter. The API is implemented like this below:

%% @doc this function is called whenever u are writing a new record in the table
%% by giving its result to the counter field in your record.
%% @end
%%
%% @spec auto_increment(TableName::atom()) -> integer() | exit(Reason)

auto_increment(TableName)->
    case lists:member(counter,table_info(TableName,attributes)) of
        false -> erlang:exit({counter,field,not_found,in_table,TableName});
        true -> table_info(TableName,size) + 1
    end.

table_info(Tab,Item)->
    F = fun({X,Y}) -> mnesia:table_info(X,Y) end,
    mnesia:activity(transaction,F,[{Tab,Item}],mnesia_frag).

To explain this, if the counter field isnot an attribute of the table i forcibly let the process that is trying to execute this code to exit with a reason, so if the programmers call this within a try ...catch, or a case (catch ...) of body , they would easily see what is wrong. Alternatively, i could ask whether this code fragment is being executed within a transaction by using mnesia:is_transaction() and if this returns true, i call mnesia:abort/1, if false, i could just exit with reason.Also, i use mnesia_frag in the mnesia activity function because this implementation will work regardless of the fragmentation properties of a table.If i use the mnesia:transaction(Fun), fragmented tables will become inconsistent because this call will only access the initial fragment (the base table).
Now, when a record is deleted from the table with counters, we need to re-arrange the order in the table. This operation is expensive as it requires iteration over the whole table.Because if they delete a record whose counter = 5, the one with counter = 6 must become counter = 5 and so on following the pattern. All records whose counters were greater than the one deleted, must be decremented. So by passing the counter value deleted and the TableName, its possible to iterate over the table using
mnesia:foldl/3 or mnesia:foldr/3 , the difference between these two comes in only with ordered table types
Here is the function which takes care of this:

auto_decrement(Counter_deleted,TableName)->
    Attrs = table_info(TableName,attributes),
    case lists:member(counter,Attrs) of
        false -> erlang:exit({counter,field,not_found,in_table,TableName});
        true -> 
            Counter_position = position(counter,Attrs) + 1,         
            Iterator =  fun(Rec,_) when element(Counter_position,Rec) > Counter_deleted -> 
                            Count = element(Counter_position,Rec),
                            New_rec = erlang:setelement(Counter_position,Rec,Count - 1),
                            mnesia:write(TableName,New_rec,read),
                            [];
                        (_,_) -> []
                        end,
            Find = fun({Fun,Table}) -> mnesia:foldl(Fun, [],Table) end,
            mnesia:activity(transaction,Find,[{Iterator,TableName}],mnesia_frag)
    end.

You notice that i have code which helps me find the position of the counter field dynamically from the record. The code which helps me do this is shown below:

position(_,[]) -> -1;
position(Value,List)->
    find(lists:member(Value,List),Value,List,1).

find(false,_,_,_) -> -1;
find(true,V,[V|_],N)-> N;
find(true,V,[_|X],N)->
    find(V,X,N + 1).

find(V,[V|_],N)-> N;
find(V,[_|X],N) -> find(V,X,N + 1).

This is so because this module must not know any of the programmers' records to help him with counters.Hence in order to access the value of the counter from the record using tuple manipulation functions such as element(N::integer(),Tuple::tuple()), i have to calculate its position in the tuple representation of the record, dynamically.

These two functions have worked for me and are still working till auto_increment
is implemented in mnesia. 

For example, using qlc (query list comprehension) to query tables with dynamic constraints, consider these pieces of code below:

select(Q)->
    F = fun(QH) -> qlc:e(QH) end,
    mnesia:activity(transaction,F,[Q],mnesia_frag).

read_by_custom_validation(Validation_fun,From_table)->
    select(qlc:q([X || X <- mnesia:table(From_table),Validation_fun(X) == true])).

%% Applying the two functions....
find_records_with_counter(From_this,To_that) when 
       is_integer(From_this),is_integer(To_that),To_that > From_this ->
    F = fun(#recordName{counter = N}) when N >= From_this,N =< To_That -> true;
        (_) -> false
    end,
    read_by_custom_validation(F,TableName).

In a stock management system, this is working...

(stock_project@127.0.0.1)6> stock:get_items_in_range(1,4).
[#item{item_id = "D694",name = "cement",
       time_stamp = {"30/12/2010","11:29:10 am"},
       min_stock = 500,units = "bags",unit_cost = 20000,
       state = available,last_modified = undefined,
       category = "building material",counter = 1},
 #item{item_id = "131B",name = "nails",
       time_stamp = {"30/12/2010","11:29:10 am"},
       min_stock = 20000,units = "kgs",unit_cost = 1000,
       state = available,last_modified = undefined,
       category = "building material",counter = 2},
 #item{item_id = "FDD9",name = "iron sheets",
       time_stamp = {"30/12/2010","11:29:10 am"},
       min_stock = 20,units = "bars",unit_cost = 50000,
       state = available,last_modified = undefined,
       category = "building material",counter = 3},
 #item{item_id = "09D4",name = "paint",
       time_stamp = {"30/12/2010","11:29:10 am"},
       min_stock = 30000,units = "tins",unit_cost = 5000,
       state = available,last_modified = undefined,
       category = "building material",counter = 4}]
(stock_project@127.0.0.1)7>

This is working for me. Please advise me on how else i should take care of counters.Or you could tell me how u r handling them over that side.

Answer 1

Is a counter the best solution for your problem? I expect that a highly distributed system is probably better off not having them and hence they have been left out.

1. If you want a count of how many elements are in the table. Have a separate table of record {counts, TableName, Count} and use the dirty_update_counter operation in the add and remove transactions of your storage tables.
2. If you want to retrieve data in a specific order (the order they were added for instance) look at using an OrderedBy solution shown here.
3. If you want fast foreign key lookups, I would look at playing around with the type of your keys (binary values for item_id maybe faster) and test if you get any significant improvement.

Also, I don't understand why you have to update the counter on every record when a table entry is deleted. I didn't think MySQL or any other RDBMS did this.