Erlang Data Storage Modules Tutorial by Flavio Ishii
Erlang Data Storage Modules
•Modules: •Erlang Terms Storage (ETS) •Disk Erlang Term Storage (DETS) •Mnesia Distributed DBMS • Data Storage Design
Flavio Ishii, Sept. 17, 2009
Tuples
•A Tuple groups items into one entity/ term.
•Used in Erlang data storage modules.
Flavio Ishii, Sept. 17, 2009
Tuples > N = { “Flavio Ishii” }. > S = [“cycling”,"ultimate frisbee" ]. > P = { person, { 1, N, S } }. > { person, Properties } = P. > Properties. > { Id, Fullname, Sports } = Properties. > X = lists:append( Sports, [“basketball”] ). > X. Flavio Ishii, Sept. 17, 2009
ETS • Erlang Term Storage • Constant access time (excl. ordered_set) • Table Options: • access rights: private | public | *protected • types: *set | ordered_set | bag | duplicate_bag • transfer ownership: {heir, Pid, HeirData}, or give_away/3
• Various methods to query the table: mnesia function (pattern matching) or qlc * default
Flavio Ishii, Sept. 17, 2009
ETS Table Access Rights • private - Only owner process can read/write • public - All processes with table id can read/write • protected - Only owner process can write and any process with table id can read.
Flavio Ishii, Sept. 17, 2009
ETS Table Types • set - unique {key, value}, inserts may overwrite. • [ {b,2}, {a,5} ] • ordered_set - key ordered and unique tuple • [ {a,5}, {b,2} ] • bag - duplicate key allowed • [ {a,5}, {b,2}, {b,4} ] • duplicate_bag - duplicate tuple allowed • [ {a,5}, {b,2}, {b,2} ] • hash tables: set, bag, and duplicate_bag • balanced binary tree: ordered_set Flavio Ishii, Sept. 17, 2009
ETS > Tab = ets:new(user, []). > ets:insert(Tab, [{1,flavio},{2,ralph},{3,melissa},{4,bob}]). > ets:info(Tab). > ets:match(Tab,'$1'). > ets:match(Tab,{'$1',bob}). > ets:match_object(Tab,{'$1',bob}). > ets:select(Tab,[{{'$1',bob},[],['$$']}]). > ets:select(Tab,[{{'$1',bob},[],['$_']}]). > qlc:eval( qlc:q([{Y} || {X,Y} <- ets:table(Tab), (X > 2) and (X < 4)])).
Flavio Ishii, Sept. 17, 2009
DETS • • • • •
Stores persistent data in disk/file (disk seeks) Process needs to open the file. Table may be shared by local processes. No ordered_set table type. Must be properly closed when process is terminated. • Checks for consistency on startup after crash.
Flavio Ishii, Sept. 17, 2009
DETS > dets:open_file(user, []). > dets:insert(user, [{1,flavio},{2,ralph},{3,melissa}, {4,bob}]). > dets:info(user). > dets:match(user,'$1'). > dets:match(user,{'$1',bob}). > dets:match_object(user,{'$1',bob}). > dets:select(user,[{{'$1',bob},[],['$$']}]). > dets:select(user,[{{'$1',bob},[],['$_']}]). > qlc:eval( qlc:q([{Y} || {X,Y} <- dets:table(user), (X > 2) and (X < 4)])).
Flavio Ishii, Sept. 17, 2009
Mnesia • • • •
Erlang’s distributed DBMS ETS & DETS core Location Transparency Fault Tolerance via Table Replication &/or Fragmentation across nodes • Transactions, Locking, & Dirty Operations • Schema Manipulation at runtime • ACID Properties
Flavio Ishii, Sept. 17, 2009
Mnesia ACID Properties • Atomicity - succeed on all or no nodes • Consistency - ensure consistent state after crash • Isolation - isolate manipulations on same record • Durability - changes are committed to disk.
Flavio Ishii, Sept. 17, 2009
Mnesia Table Options • • • • • • •
{type, Type} % set, ordered_set, bag {record_name, Name} {ram_copies, NodeList} % fastest {disc_copies, NodeList} % RAM and disc copies {disc_only_copies, NodeList} % slowest {attributes, AtomList} {index, IndexAtomList}
Flavio Ishii, Sept. 17, 2009
Terminal Time! • • • •
Create record structure Create schema and tables Insert records Query records
Flavio Ishii, Sept. 17, 2009
api.hrl % this file defines the records -record(counter,{id_name,value}). -record(user_details, {password,firstname,lastname,sports=[]}). -record(user,{id,username,user_details}).
Flavio Ishii, Sept. 17, 2009
Part 1/4 of api_db.erl % this file defines the records -module(api_db). -export([init_db/0, add_sport/2, mne_fun_query/1, qlc_query/1]). -include("api.hrl"). -include_lib("stdlib/include/qlc.hrl"). init_db() -> mnesia:create_schema([node()]), io:format("Mnesia schema created~n"), mnesia:start(), mnesia:change_table_copy_type(schema, node(), disc_copies), mnesia:create_table(counter, [{disc_copies,[node()]} , {attributes, record_info(fields, counter)}]), mnesia:create_table(user, [{disc_copies,[node()]}, {index, [username]} , {attributes, record_info(fields, user)}]), io:format("Mnesia tables created~n"), add_sample_data().
Flavio Ishii, Sept. 17, 2009
Part 2/4 of api_db.erl insert_user(UserRecord) -> case UserRecord#user.id =:= undefined of true -> NewUser = UserRecord#user{ id = mnesia:dirty_update_counter(counter, user_id, 1) }; false -> NewUser = UserRecord end, % mnesia:dirty_write(NewUser). or use a transaction... Fun = fun() -> mnesia:write(NewUser) end, mnesia:transaction(Fun). add_sample_data() -> Flavio = #user{username="flavio" , user_details=#user_details{ password="mypassword", firstname="Flavio", lastname="Ishii", sports=["biking","basketball"]}}, insert_user(Flavio), Bob = #user{username="bob", user_details=#user_details{ password="hispassword", firstname="Bob", lastname="The Builder", sports=["soccer"]}}, insert_user(Bob), io:format("Users added to table.~n").
Flavio Ishii, Sept. 17, 2009
Part 3/4 of api_db.erl % > api_db:add_sport("flavio","football"). add_sport(Un,NewSport) -> [User] = mne_fun_query({username,Un}), UserDetails = User#user.user_details, Sports = UserDetails#user_details.sports, NewList = lists:append(Sports,[NewSport]), NewUser = User#user{user_details=#user_details{sports=NewList}}, insert_user(NewUser). % This is one method of querying a user. % > api_db:qlc_query({username,"flavio"}). qlc_query({username,Username}) -> F = fun() -> qlc:e(qlc:q([U#user.user_details || U <- mnesia:table(user) , U#user.username =:= Username ])) end, mnesia:transaction(F).
Flavio Ishii, Sept. 17, 2009
Part 4/4 of api_db.erl % This may not be ideal but it demonstrates the use of Pattern Matching % > api_db:mne_fun_query({username,"flavio"}). mne_fun_query({username,Un}) -> MatchHead = #user{username='$1', _='_'}, Guard = [{'=:=','$1',Un}], Result = ['$_'], MatchSpec = [{MatchHead, Guard, Result}], mnesia:dirty_select(user, MatchSpec); % > api_db:mne_fun_query({sport,"soccer"}). mne_fun_query({sport,Sport}) -> UserDetails = #user_details{_='_',sports='$1'}, MatchHead = #user{user_details=UserDetails,_='_'}, Guard = [{'=:=','$1',[Sport]}], Result = ['$_'], MatchSpec = [{MatchHead, Guard, Result}], F = fun() -> mnesia:select(user, MatchSpec) end, mnesia:transaction(F); mne_fun_query(_) -> io:format("No match~n").
Flavio Ishii, Sept. 17, 2009
Head to Head Feature List ETS Persistent storage Complex search queries Distributed Replicated data storage Table Fragmentation Fault tolerance via replication Tight Erlang coupling Complex objects & relationships Dynamic reconfiguration Table indexing Distributed Transactions
X X
DETS X X X
X X
X X
X
X
Mnesia X X X X X X X X X X
Flavio Ishii, Sept. 17, 2009
Possible Design Permutations
Flavio Ishii, Sept. 17, 2009
More Permutations • Table access (i.e. public, protected)
• Table type (i.e. set, bag...) • Table locks • Distribution via replication and/or table fragmentation
Flavio Ishii, Sept. 17, 2009
Useful Mnesia Functions • • • • • • • • • • • • • •
change_table_access_mode/2 change_table_copy_type/3 backup/2 install_fallback/2 restore/2 dump_tables/1 lock/2 move_table_copy/3 async_dirty/2 sync_dirty/2 add_table_copy/3 transform_table/3 change_table_frag/2 activity/4 Flavio Ishii, Sept. 17, 2009
Resources • Programming Erlang, Joe Armstrong • ets, dets, qlc, mnesia manuals: • http://www.erlang.org/doc/man/ • Mnesia User’s Guide: • http://www.erlang.org/doc/apps/mnesia • Mnesia - A Distributed Robust DBMS for Telecommunications Applications, Håkan Mattsson, Hans Nilsson and Claes Wikström
Flavio Ishii, Sept. 17, 2009
Contact Info USASK - MADMUC Lab http://flavioishii.com @flavioishii
Flavio Ishii, Sept. 17, 2009