Tables on disk

You can save the table to disk as a "serialized table" or "splayed table". It is easy to store complex data in serialized tables, but such tables are entirely getting into memory, and there is no way to work with such tables larger than RAM.

o)t: +`simple`complex!(1 2; (`a`b`c!!3; `c`d!(1 2; 3 4)));
o)`:./tmp/stbl set t;
o)tbl: get `:./tmp/stbl
simple complex
---------------------
1      `a`b`c!0 1 2
2      `c`d!(1 2;3 4)
o)

"Splayed table" in O means just the same as in Q/KDB - saving each field vector in a separate continuous file capable of mmapping it.

"Mmapping" allows to avoid loading all table contents in memory in one go, but to work with a table off the disk (by using OS memory-mapped files technique).

Only following field types are supported in splayed tables:

Saving table vаlues to disk is pretty simple. Just choose directory to keep your table files and execute something similar to:

o)a:(+:)`a`b`c!(1 2 3;1 2 3;1 2 3); f:`:/tmp/o_table/; f set a;
o)

Doing that creates /tmp/o_table directory filled with files named ".d", "a", "b" and "c".These files corresponding to fields and the ".d" file which contains symbol vector `a`b`c to keep the field order of the table.

Please pay attention that only fields mentioned in ".d" file are considered as table fields. Thus even if files are in the table directory, but not in the ".d" file, they will be ignored.

Use "get" function to load table contents:

o)f:`:/tmp/o_table/; get f
a b c
-----
1 1 1
2 2 2
3 3 3
o)

Modifying table on disk can be done in two ways:

Amending via a path symbol is useful e.g. for adding a new field.

o)// creating test table
o)a:(+:)`a`b`c!(1 2 3;1 2 3;1 2 3); f:`:/tmp/o_table2/; f set a;
o)// append two new fields
o)@[f;`e`d;:;(3#1;3#2)];
o)// modify .d file to record new fields
o).[`:/tmp/o_table2/.d;();,;`e`d];

Appending records via path symbol is ok.

o)a:(+:)`a`b`c!(1 2 3;1 2 3;1 2 3); f:`:/tmp/o_table3/; f set a;
o)// appending one record
o).[f;();,;(10;20;30)];
o)// appending two records
o).[f;();,;(10 10;20 20;30 30)];

Opening table in workspace works by binding global variable to table mmapped on disk.

o)// create new table on disk
o)a:(+:)`a`b`c!(1 2 3;1 2 3;1 2 3); f:`:/tmp/odb1/; f set a;
o)// load it to workspace. Now odb1 global variable contains a writable reference
o)load f;
o)// append new record
o).[`odb1;();,;(10;20;30)];
o)// ... or via short syntax
o)odb1,:(10;20;30);
o)// see contents
o)odb1
a  b  c
--------
1  1  1
2  2  2
3  3  3
10 20 30
10 20 30
o)// close table, saving pending changes
o)odb1::0;
o)