Version 1.3

28 Nov, 2010

FastFileDB 1.3 has been improved on several aspects. It is applied as an embedded database for the project Handy Application Maker II .Use DbVisualizer http://www.dbvis.com (database tools) to create new connection. (Refer below for setting). The final JDBC setting has been confirmed as below:

JDBC Driver: net.sf.fastfiledb.jdbc.Driver / net.sf.fastfiledb.sub.jdbc.Driver (light-weight)
JDBC URL: jdbc:fastfiledb:[database name];dir=[working directory];create;pool=1
User Name: adm
Password: adm

E.g. jdbc:fastfiledb:DefaultDB;dir=C:/data;create will create the database "DefaultDB" inside the folder "C:/data".

The light-weight driver provides faster startup time. It removes the database schema portion to gain speed.

These files can be found under the "lib" folder of FastFileDB.

The SQL statement parsing is developed by other open source project. Credit to:

Download FastFileDB 1.3 here

 

Version 1.2

28 Sep, 2010

FastFileDB 1.2 has been upgraded to alpha version (1.1 ver is a dev. ver), use DbVisualizer http://www.dbvis.com (tested tools) to create new connection. The JDBC URL has slightly changes:

JDBC Driver: net.sf.ffdb.jdbc.Driver
JDBC URL: jdbc:ffdb:[project name];dir=[working directory];create
User Name: adm
Password: adm

E.g. jdbc:ffdb:DefaultDB;dir=C:/data;create will create the database "DefaultDB" inside the folder "C:/data".

These files can be found under the "lib" folder of FastFileDB.

The SQL statement parsing is developed by other open source project. Credit to:

Download FastFileDB 1.2 here

 

Version 1.1

07 Jul, 2010

FastFileDB (FFDB) 1.1 has become a very light-weight database engine. It has its own custom-built JDBC. The GUI tool is still under developedment. Mean while the author recommends DbVisualizer (http://www.dbvis.com) - a database tools, as shown:

NOTE that FastFileDB 1.1 is still a development version (not yet reached alpha or beta).

JDBC Driver: net.sf.fastfiledb.JdbcDriver
JDBC URL: jdbc:ffdb:[project name]/[database name];createproject;createdatabase;encrypt;
User Name: adm
Password: adm

Download here

 

Version 1.0

FastFileDB (FFDB) is a light-weight file database. It saves records on file basis. Besides pure text database, it supports data type, data format, is data empty check and default field data for each field. Furthermore, it allows developer to set primary key and unique keys.

All data will be loaded into memory when FFDB started. The data size supported is based on physical memory size and JVM memory allocated. It is recommended for data size that is less than 100mb per record set. In FFDB pure text mode (turn off all features: data type, data format, primary key, etc), the loading time of a 100mb file is about 10 seconds. Turning on more features will consume more loading time. If all features are turned on, the performance will degrade 10 times or more, which means it may take minutes to load the same file.

Download here

 

Features

 

Code Guide

To simply create a table:

  1. Define column name:
    String[] colName = new String[] {"Product Description","Width","Height","Unit Price","On Sale?","Date produced","Time produced","Date time produced"};
    This is a 8-columns definition table, and it is the fastest mode (text-only-mode). Every column is treated as text.

  2. Init FlatFileDB
    FileDbTableModel fastFileDb = new FileDbTableModel();
    fastFileDb.create("TestFile.txt", true, false, FileDbTableModel.LIST_ARRAYLIST, false, colName, null, null, null, null, null, null);
    fastFileDb.reload();
    Or
    FileDbTableModel fastFileDb = new FileDbTableModel();
    fastFileDb.create("TestFile.txt", true, false, FileDbTableModel.LIST_ARRAYLIST, false, 8, null, null, null, null, null, null);
    fastFileDb.reload();
    to create a 8-columns table without specifying column name. NOTE that 'fastFileDb.reload();' must be explicitly called to start loading data. Otherwise, the table remains empty.

To enhance the table, by defining data type, data format, default value, and is column allows null value:

  1. Define data type (optional):
    byte[] dataType = new byte[] {FileDbDataType.STRING, FileDbDataType.INTEGER, FileDbDataType.INTEGER, FileDbDataType.DOUBLE, FileDbDataType.BOOLEAN, FileDbDataType.DATE, FileDbDataType.TIME, FileDbDataType.TIMESTAMP};
    Data input must match the data type specified. Otherwise it will be ignored and not updated. 11 data types supported can be found in FileDbDataType.

  2. Define data format (optional):
    String[] colFormat = new String[] {null, "000", "000", "#,###,##0.00", null, "dd/MM/yyyy", "hh:mm a", "dd/MM/yyyy hh:mm a"};
    Data will be formatted and accept formatted pattern as the input pattern. This process may take times to complete. Depends on total records and total size.

  3. Define default column value (optional):
    Object[] defaultValue = new Object[] {"New Product", new Integer(100), new Integer(100), new Double(0.00), "false", "2010-12-31", "00:00:00", "2010-12-31 00:00:00"};
    Default value will automatically assigned to column with empty data and does not allow null value.

  4. Define is column allows null value (option):
    boolean[] isBlank = new boolean[] {false, false, false, true, true, true, true, true};
    If a not null column is set empty, error occurs. If default value is set, then it will fill up the empty.

Primary key (optional):

  1. Set primary key:
    int[] primaryKeyIdx = new int[] {0};
    set the first column as the primary key. Columnn index start from zero to total columns minus one.

  2. Define more than one column as primary key;
    int[] primaryKeyIdx = new int[] {0, 1, 3};
    This statement defines first, second and fourth column as the primary key. NOTE: More keys more memory and hence, slower performance.

Unique key (optional):

  1. Similar to primary key beside it can have many groups defined:
    fastFileDb.addUniqueKey(new int[] {0});
    set the first column as the unique key. Similarly, columnn index start from zero to total columns minus one.

  2. Define more than one column as unique key;
    fastFileDb.addUniqueKey(new int[] {0, 1, 3};
    This statement defines first, second and fourth column as the one unique key set. NOTE: More keys more memory and hence, slower performance.

  3. Complex key definition by defining more than one set of unique keys;
    fastFileDb.addUniqueKey(new int[] {0};
    fastFileDb.addUniqueKey(new int[] {1};
    fastFileDb.addUniqueKey(new int[] {0, 1, 3};
    The above statement defines the first column as unique, second column is also unique, and the combination of first, second and fourth columns as one unique key set. The difference among these three are the third set is invalid only if the first, second and fourth column value appears duplicated. It does not check an invidual column for duplicated value, wheres the first and second unique key explicitly set as unique. Thus, duplicated value is invalid for each column.
    For the above case, the first and second columns cannot contain duplicated value whereas the fourth may contain duplicated value (provided that the combination of first and second and fourth does not repeat.)

 

Performance Tips

The text-only-mode is the fastest way to handle data. Uploading thousands of records within seconds and million of records wihin a minute. This mode treats all columns as text-based data. So input "abc" to an integer column is valid. No data type check.

The data type mode convert text data to specified data type. Every input will be validated before update. Invalid data type will not be updated, the original value is remained. The best thing is this mode does not hit the performance. It takes extra 1-10% of time compare to text-only-mode

Data format can slow the performance as every data will be formatted individually row-by-row, column-by-column. It consumes extra 10-80% of time depends on how large the data is and how many formatted column. Try to disabled this mode if you find your data loading is slow.

Primary key and unique key are same to data format besides it check for data duplication instead of formatting data. It needs 10-50% extra loading time.

All data is loaded into memory in object array form. So an extra 8/16Kb is needed for each row. This means that storing 150Mb physical size data will need 10 times of RAM size, which is 1.5Gb to store it. As a result, FastFileDB is recommended for file within 10Mb. It is originally designed to hold temporary data or lookup data.

Conclusion, when you find your table slow, try to minimise the number of formatted column and keys definition.

Download here