Firebird 2.5 has arrived

Superclassic architecture
Other new features in Firebird 2.5
What can we expect in Firebird 3.0?

Firebird 2.5 has arrived!

Support for Firebird is growing daily. Firebird is now being installed 2,000 times daily around the world, and that 365 days a year! It already boasts a million installations, and the release of Firebird 2.5 promises to increase that figure significantly.

Firebird 2.5 introduces a number of interesting new features and numerous improvements over previous Firebird versions. Firebird 2.5 is the first step into the Classic, Superserver and Embedded unification with codebase modernization, a common threadable architecture and a more efficient multi-threading support.

Threadability has been the primary goal: there has been a major MT cleanup; only threadable platforms are supported and the old synchronization logic has been updated with generic thread safety everywhere, starting from the API (both client and Embedded).

Firebird 2.5 achieves better scalability: no cooperative scheduling any more. The global mutex has gone, synchronization now goes to the lower levels. The server layer now offers worker thread pooling, asynchronous connection cleanup and unified listener code for Superserver and Classic server. Lock management provides non-blocking notifications, there are lock tables for each individual database.

This has a number of effects on the architectures supported:

Classic server: the lock manager no longer needs explicit semaphores and signals setup; the fb_lock_mgr binary is now deprecated on POSIX; services and automatic sweep are executed as threads (less processes spawned); the server layer can determine broken network links immediately and terminate the worker process; libfbembed.so is thread-safe.
Superserver: now offers better overall scalability and response times due to a lower-level synchronization inside the engine. SMP/multi-core hardware can be utilized efficiently when multiple databases are accessed simultaneously. fbclient.so/.dll is thread-safe.
Embedded server: now based on the Classic architecture, including all the benefits of the Superclassic: multiple processes can safely access the database, official utilities (GBAK, GFIX, ISQL etc.) can also be used. fbembed.dll is thread-safe.

Although SQL enhancements are not a primary objective of this release, for the first time, user management becomes accessible through SQL CREATE/ALTER/DROP USER statements and syntaxes for ALTER VIEW and CREATE OR ALTER VIEW have been implemented. PSQL improvements include the introduction of autonomous transactions ? and the ability to query another database via EXECUTE STATEMENT.

Superclassic architecture

Historical roots

The original ideas came from the Vulcan project, whose goal was to have a multi-threaded shared cache architecture, the side effect of which offered quite a different architecture, known as "no shared cache mode", implemented both by Jim Starkey and the guys at the SAS Institute. This original idea evolved into what we currently know as Superclassic architecture in Firebird.

What is Superclassic?

The key ideas behind this architecture allow multiple processes to access the same database files simultaneously and each of these processes may have multiple threads accessing this database. So it's quite a flexible configuration which follows a lot of different processes and threads. Superclassic inherits the key points of the Classic architecture, such as separate page cache and metedata cache per user connection. All these caches may reside in a single process or multiple processes, with pooled multiple worker threads. Database access can be safely shared among Classic, Superclassic and Embedded processes:

(Image copyright Dmitry Yemanov)

Firebird 2.5 has established a common threading platform for all incremations of Firebird for all its architectures. The common threading platform means that finally all kinds of Firebird are aware of threads linked to the corresponding system libraries. Finally the entire code base is thead-safe, both inside the engine itself and including client and embedded libraries. It can now also offer better concurrency over these threads. Thread support also opens abilities to provide asynchronous clean up of remote connections, for example, after a broken network link. It allows services or a sweep to run in the backgroud in a separate thread.

Even the Embedded engine is now based on the Superclassic architecture instead of the old Superserver architecture, where it locked the database file exclusively and didn't allow any other applications to work with it. So now not just your application but also other services, third-party tools and the like can access an embedded database and you can monitor the database whilst it's running, run queries, debug and so on and so on. One connection may be peformed directly from your application using the embedded library directly. At the same time another application may connect through the regular Classic server. And some other applications may establish multiple connections to the same database file using the Superclassic server. All process load, directly or indirectly, the embedded library, which acts as a core component of this architecture. The embedded library behaves the same way on all platforms, both on Windows and POSIX and it is absolutely thread-safe.

All architectures create separate lock tables for each database. So access to multiple databases on a single server, regardless of server type, runs independently.

Superclassic as an application

So far we have been talking about Superclassic as an architecture. But Superclassic is also an application; a kind of server installation setup. And in this regard the only difference between Classic and Superclassic is how it handles user connections. In Classic you have a dedicated process which listens for incoming requests. In Superclassic there is a single process, which both listens for incoming packets and threads. Superclassic maintains a thread pool. It doesn’t launch a new dedicated thread for every user connection. The threads are pooled so only those connections that are currently active - running queries, creating data etc. - will occupy a thread and the appropriate resources.

On Windows it’s the same executable as for the regular Classic, fb_inet_server. It has a new switch –m, which stands for multithreading. So just add the -a –m switches to the comment line to run the server as an application. To install Superclassic as a service you specify the same –m switch but to the service installation utility and you get the Superclassic installed as a service.

On Posix it was decided to introduce a new executable, fb_smp_server, which acts as a Superclassic hold process. It combines the listener and the actual engine code. It’s installed as a system daemon, similar to how you install the Superserver and it doesn’t require the listener provided by the system, which means it works similarly to fbserver with regard to attachment requirements and does not require [x]inetd.

Comparison Superclassic versus Classic

Superclassic uses less kernel resources (but not memory).
The scalability limits have been shifted.
It achieves better performance due to local calls inside the lock manager and other in-process optimizations (25% performance boost in TPC-C).
The server can be safely shut down as a whole.
There is now the possibility to enumerate attached databases/users.
The security database connection is cached, which means new connections can be handled faster.

Comparison Superclassic versus Superserver

Better scalability (the number of connections is not explicitly limited).
Better concurrency on SMP/multi-core hardware.
More responsive under high load.

Drawbacks of the Superclassic architecture

Somewhat ineffective memory usage.
High lock table contentions (page locks).
It requires careful tuning of the LM configuration.
A server crash affects all user attachments.
Does not make much sense on 32-bit systems.

Other new features in Firebird 2.5

Other new features and improvements in this release include:

Administrative enhancements

System audit tracing and user trace sessions via the Services API, making it possible to monitor and analyse everything going on in a database in real time.
New system role RDB$ADMIN in the ODS 11.2 database allows SYSDBA to transfer its privileges to another user on a per-database basis.
More information in the monitoring tables.
Asynchronous cancellation of connections.
Capability for ordinary users to monitor any of their own attachments as well as CURRENT_CONNECTION.

Other SQL language additions and enhancements

Regular expression support using the SIMILAR TO predicate.
ALTER COLUMN for computed columns.
Autonomous transactions within a PSQL module (stored procedure, trigger or dynamically executable PSQL block).
Enhanced access to stored procedures in view definitions.
Optional GRANTED BY or GRANTED AS for GRANT and REVOKE statements, enabling the grantor to be a user other than the CURRENT_USER (the default).
REVOKE ALL syntax to dispose of all privileges for a user or role at once.
Support for WHERE SOME_COL = ? OR ? IS NULL predications.
Removal of "reserved" status for all but a handful of keywords that are not reserved in the SQL standard.

Data-handling enhancements

New built-in functions for converting UUID CHAR(16) OCTETS strings to RFC4122-compliant format and vice versa.
Ability to pass 32-bit and 64-bit integers as hexadecimal in numeric literal and X-prefixed binary string literal formats.

API additions

Statements now return an SQL-2003 standard 5-alphanumeric SQLSTATE completion code.
New constant DSQL_unprepare available for use with isc_dsql_free_statement for efficient unpreparing of statements.

International language support

Default COLLATE clause for CREATE DATABASE.
Ability to change the default COLLATE for a used character set.
GBAK restore switches FIX_FSS_DATA and FIX_FSS_METADATA to restore legacy databases with UNICODE_FSS data and/or metadata correctly without resorting to scripts and manual techniques.
Accent-insensitive collation for Unicode.

Details of all new features can be referred to in the Firebird 2.5 Release Notes (currently beta version 2).

What can we expect in Firebird 3.0?

Firebird 3.0 represents the project's next milestone release and it currently gets the most of the development efforts. It's going to offer some important and highly anticipated features.

Firebird 3 is expected to have a single architecture and a single set of executables supporting all current architectures. In particular two basic moulds would be exclusive database access and shared database access.

Exclusive database access is basically what Superserver currently is. This is a single process opening the database and holding an exclusive lock on the database file. There are multiple threads running inside this process, accessing the database. There are multiple connections inside this process and all these connections share the same cache. So it’s basically what has been expected from Firebird for the last five or six years: a Superserver which is scalable and which has a shared page cache. But also there will be a shared database access which is mostly the same as Classic and Superclassic are now. We have multiple processes, each of these processes make up multiple threads; the synchronization between connections is external; it involves both the local call and the IPC interaction and there will be a choice how this shared database access works: you may choose something similar to the Superclassic – i.e. thread-based execution of your queries - or process-based like the common Classic if you need more reliability and crash resistance.

(Image copyright Dmitry Yemanov)

This simple picture shows just how it is going to happen … well at least how it is expected to happen! Basically Superserver and Superclassic are combined together to offer you an exclusive working mode with a single process which accesses the database and Superclassic and Classic architectures combined to provide shared database access with multiple processes.

Further key points include:

Revised architecture, including new authentication and configuration subsystems
Scalable (SMP/multi-core friendly) multi-threaded engine with the shared page cache
New ODS (version 12) with new system and monitoring tables
External (Java, C++, etc) stored procedures/triggers/functions
Reworked optimizer and new data access methods
Security enhancements
Various SQL extensions

The Firebird Foundation are currently aiming to release the first alpha version in Q2 2010.

Sources:
PPT presentation Bergamo 2008, Dmitry Yemanov
www.firebirdsql.org Webinar Superclassic: new architecture in Firebird, 2010-04-06

Further reading:
Get to know Firebird in 2 minutes
Firebird 2.5 Release Notes: Firebird 2.5 new features