Erlang Data Storage Modules

Erlang Data Storage Modules Tutorial by Flavio Ishii

Erlang Data Storage Modules

•Modules: •Erlang Terms Storage (ETS) •Disk Erlang Term Storage (DETS) •Mnesia Distributed DBMS • Data Storage Design

Flavio Ishii, Sept. 17, 2009

Tuples

•A Tuple groups items into one entity/ term.

•Used in Erlang data storage modules.

Flavio Ishii, Sept. 17, 2009

Tuples > N = { “Flavio Ishii” }. > S = [“cycling”,"ultimate frisbee" ]. > P = { person, { 1, N, S } }. > { person, Properties } = P. > Properties. > { Id, Fullname, Sports } = Properties. > X = lists:append( Sports, [“basketball”] ). > X. Flavio Ishii, Sept. 17, 2009

ETS • Erlang Term Storage • Constant access time (excl. ordered_set) • Table Options: • access rights: private | public | *protected • types: *set | ordered_set | bag | duplicate_bag • transfer ownership: {heir, Pid, HeirData}, or give_away/3

• Various methods to query the table: mnesia function (pattern matching) or qlc * default

Flavio Ishii, Sept. 17, 2009

ETS Table Access Rights • private - Only owner process can read/write • public - All processes with table id can read/write • protected - Only owner process can write and any process with table id can read.

Flavio Ishii, Sept. 17, 2009

ETS Table Types • set - unique {key, value}, inserts may overwrite. • [ {b,2}, {a,5} ] • ordered_set - key ordered and unique tuple • [ {a,5}, {b,2} ] • bag - duplicate key allowed • [ {a,5}, {b,2}, {b,4} ] • duplicate_bag - duplicate tuple allowed • [ {a,5}, {b,2}, {b,2} ] • hash tables: set, bag, and duplicate_bag • balanced binary tree: ordered_set Flavio Ishii, Sept. 17, 2009

ETS > Tab = ets:new(user, []). > ets:insert(Tab, [{1,flavio},{2,ralph},{3,melissa},{4,bob}]). > ets:info(Tab). > ets:match(Tab,'$1'). > ets:match(Tab,{'$1',bob}). > ets:match_object(Tab,{'$1',bob}). > ets:select(Tab,[{{'$1',bob},[],['$$']}]). > ets:select(Tab,[{{'$1',bob},[],['$_']}]). > qlc:eval( qlc:q([{Y} || {X,Y} <- ets:table(Tab), (X > 2) and (X < 4)])).

Flavio Ishii, Sept. 17, 2009

DETS • • • • •

Stores persistent data in disk/file (disk seeks) Process needs to open the file. Table may be shared by local processes. No ordered_set table type. Must be properly closed when process is terminated. • Checks for consistency on startup after crash.

Flavio Ishii, Sept. 17, 2009

DETS > dets:open_file(user, []). > dets:insert(user, [{1,flavio},{2,ralph},{3,melissa}, {4,bob}]). > dets:info(user). > dets:match(user,'$1'). > dets:match(user,{'$1',bob}). > dets:match_object(user,{'$1',bob}). > dets:select(user,[{{'$1',bob},[],['$$']}]). > dets:select(user,[{{'$1',bob},[],['$_']}]). > qlc:eval( qlc:q([{Y} || {X,Y} <- dets:table(user), (X > 2) and (X < 4)])).

Flavio Ishii, Sept. 17, 2009

Mnesia • • • •

Erlang’s distributed DBMS ETS & DETS core Location Transparency Fault Tolerance via Table Replication &/or Fragmentation across nodes • Transactions, Locking, & Dirty Operations • Schema Manipulation at runtime • ACID Properties

Flavio Ishii, Sept. 17, 2009

Mnesia ACID Properties • Atomicity - succeed on all or no nodes • Consistency - ensure consistent state after crash • Isolation - isolate manipulations on same record • Durability - changes are committed to disk.

Flavio Ishii, Sept. 17, 2009

Mnesia Table Options • • • • • • •

{type, Type} % set, ordered_set, bag {record_name, Name} {ram_copies, NodeList} % fastest {disc_copies, NodeList} % RAM and disc copies {disc_only_copies, NodeList} % slowest {attributes, AtomList} {index, IndexAtomList}

Flavio Ishii, Sept. 17, 2009

Terminal Time! • • • •

Create record structure Create schema and tables Insert records Query records

Flavio Ishii, Sept. 17, 2009

api.hrl % this file defines the records -record(counter,{id_name,value}). -record(user_details, {password,firstname,lastname,sports=[]}). -record(user,{id,username,user_details}).

Flavio Ishii, Sept. 17, 2009

Part 1/4 of api_db.erl % this file defines the records -module(api_db). -export([init_db/0, add_sport/2, mne_fun_query/1, qlc_query/1]). -include("api.hrl"). -include_lib("stdlib/include/qlc.hrl"). init_db() -> mnesia:create_schema([node()]), io:format("Mnesia schema created~n"), mnesia:start(), mnesia:change_table_copy_type(schema, node(), disc_copies), mnesia:create_table(counter, [{disc_copies,[node()]} , {attributes, record_info(fields, counter)}]), mnesia:create_table(user, [{disc_copies,[node()]}, {index, [username]} , {attributes, record_info(fields, user)}]), io:format("Mnesia tables created~n"), add_sample_data().

Flavio Ishii, Sept. 17, 2009

Part 2/4 of api_db.erl insert_user(UserRecord) -> case UserRecord#user.id =:= undefined of true -> NewUser = UserRecord#user{ id = mnesia:dirty_update_counter(counter, user_id, 1) }; false -> NewUser = UserRecord end, % mnesia:dirty_write(NewUser). or use a transaction... Fun = fun() -> mnesia:write(NewUser) end, mnesia:transaction(Fun). add_sample_data() -> Flavio = #user{username="flavio" , user_details=#user_details{ password="mypassword", firstname="Flavio", lastname="Ishii", sports=["biking","basketball"]}}, insert_user(Flavio), Bob = #user{username="bob", user_details=#user_details{ password="hispassword", firstname="Bob", lastname="The Builder", sports=["soccer"]}}, insert_user(Bob), io:format("Users added to table.~n").

Flavio Ishii, Sept. 17, 2009

Part 3/4 of api_db.erl % > api_db:add_sport("flavio","football"). add_sport(Un,NewSport) -> [User] = mne_fun_query({username,Un}), UserDetails = User#user.user_details, Sports = UserDetails#user_details.sports, NewList = lists:append(Sports,[NewSport]), NewUser = User#user{user_details=#user_details{sports=NewList}}, insert_user(NewUser). % This is one method of querying a user. % > api_db:qlc_query({username,"flavio"}). qlc_query({username,Username}) -> F = fun() -> qlc:e(qlc:q([U#user.user_details || U <- mnesia:table(user) , U#user.username =:= Username ])) end, mnesia:transaction(F).

Flavio Ishii, Sept. 17, 2009

Part 4/4 of api_db.erl % This may not be ideal but it demonstrates the use of Pattern Matching % > api_db:mne_fun_query({username,"flavio"}). mne_fun_query({username,Un}) -> MatchHead = #user{username='$1', _='_'}, Guard = [{'=:=','$1',Un}], Result = ['$_'], MatchSpec = [{MatchHead, Guard, Result}], mnesia:dirty_select(user, MatchSpec); % > api_db:mne_fun_query({sport,"soccer"}). mne_fun_query({sport,Sport}) -> UserDetails = #user_details{_='_',sports='$1'}, MatchHead = #user{user_details=UserDetails,_='_'}, Guard = [{'=:=','$1',[Sport]}], Result = ['$_'], MatchSpec = [{MatchHead, Guard, Result}], F = fun() -> mnesia:select(user, MatchSpec) end, mnesia:transaction(F); mne_fun_query(_) -> io:format("No match~n").

Flavio Ishii, Sept. 17, 2009

Head to Head Feature List ETS Persistent storage Complex search queries Distributed Replicated data storage Table Fragmentation Fault tolerance via replication Tight Erlang coupling Complex objects & relationships Dynamic reconfiguration Table indexing Distributed Transactions

X X

DETS X X X

X X

X X

X

X

Mnesia X X X X X X X X X X

Flavio Ishii, Sept. 17, 2009

Possible Design Permutations

Flavio Ishii, Sept. 17, 2009

More Permutations • Table access (i.e. public, protected)

• Table type (i.e. set, bag...) • Table locks • Distribution via replication and/or table fragmentation

Flavio Ishii, Sept. 17, 2009

Useful Mnesia Functions • • • • • • • • • • • • • •

change_table_access_mode/2 change_table_copy_type/3 backup/2 install_fallback/2 restore/2 dump_tables/1 lock/2 move_table_copy/3 async_dirty/2 sync_dirty/2 add_table_copy/3 transform_table/3 change_table_frag/2 activity/4 Flavio Ishii, Sept. 17, 2009

Resources • Programming Erlang, Joe Armstrong • ets, dets, qlc, mnesia manuals: • http://www.erlang.org/doc/man/ • Mnesia User’s Guide: • http://www.erlang.org/doc/apps/mnesia • Mnesia - A Distributed Robust DBMS for Telecommunications Applications, Håkan Mattsson, Hans Nilsson and Claes Wikström

Flavio Ishii, Sept. 17, 2009

Contact Info USASK - MADMUC Lab http://flavioishii.com @flavioishii

Flavio Ishii, Sept. 17, 2009