Command-line utilities

Backup tools

Firebird 2 brings plenty of enhancements to backing up databases: a new utility for running on-line incremental backups and some improvements to gbak to avoid some of the traps that sometimes befall end-users.

New on-line incremental backup

N. Samofatov

Fast, on-line, page-level incremental backup facilities have been implemented. The backup engine comprises two parts:

NBak, the engine support module and
NBackup, the tool that does the actual backups.

`Nbak`

The functional responsibilities of NBak are:

to redirect writes to difference files when asked (ALTER DATABASE BEGIN BACKUP statement),
to produce a GUID for the database snapshot and write it into the database header before the ALTER DATABASE BEGIN BACKUP statement returns,
to merge differences into the database when asked (ALTER DATABASE END BACKUP statement),
to mark pages written by the engine with the current SCN [page scan] counter value for the database,
to increment SCN on each change of backup state.

The backup state cycle is:

 nbak_state_normal -> nbak_state_stalled -> nbak_state_merge -> nbak_state_normal

In normal state writes go directly to the main database files.
In stalled state writes go to the difference file only and the main files are read-only.
In merge state new pages are not allocated from difference files. Writes go to the main database files. Reads of mapped pages compare both page versions and return the version which is fresher, because we don't know if it is merged or not.

Note: This merge state logic has one quirky part. Both Microsoft and Linux define the contents of file growth as "undefined" i.e., garbage, and both zero-initialize them.

This is why we don't read mapped pages beyond the original end of the main database file and keep them current in difference file until the end of a merge. This is almost half of NBak fetch and write logic, tested by using modified PIO on existing files containing garbage.

`NBackup`

The functional responsibilities of NBackup are:

to provide a convenient way to issue ALTER DATABASE BEGIN/END BACKUP,
to fix up the database after file system copy (physically change nbak_state_diff to nbak_state_normal in the database header),
to create and restore incremental backups.

Incremental backups are multi-level. That means if you do a Level 2 backup every day and a Level 3 backup every hour, each Level 3 backup contains all pages changed from the beginning of the day till the hour when the Level 3 backup is made.

Backing up

Creating incremental backups has the following algorithm:

Issue ALTER DATABASE BEGIN BACKUP to redirect writes to the difference file.
Look up the SCN and GUID of the most recent backup at the previous level.
Stream database pages having SCN larger than was found at step 2 to the backup file.
Write the GUID of the previous-level backup to the header, to enable the consistency of the backup chain to be checked during restore.
Issue ALTER DATABASE END BACKUP.
Add a record of this backup operation to RDB$BACKUP_HISTORY. Record current level, SCN, snapshot GUID and some miscellaneous stuff for user consumption.

Restoring

Restore is simple: we reconstruct the physical database image for the chain of backup files, checking that the backup_guid of each file matches prev_guid of the next one, then fix it up (change its state in header to nbak_state_normal).

Usage

 nbackup <options>

Valid Options

-L <database>: Lock database for file system copy
-N <database>: Unlock previously locked database
-F <database>: Fixup database after file system copy
-B <level> <database> [<filename>]: Create incremental backup
-R <database> [<file0> [<file1>...]]: Restore incremental backup
-U <user>: User name
-P <password>: Password

Note:

<database> may specify a database alias.
incremental backups of multi-file databases are not supported yet.
"stdout" may be used as a value of <filename> for the -B option.

Improvements

(v.2.0.6) An improvement has been done for POSIX versions to address a problem whereby the full backup tool of the nBackup would hog I/O resources when backing up large databases, bringing production work to a standstill. Now, NBackup tries to read from the operating system cache before attempting to read from disk, thus reducing the I/O load substantially.

Note: The cost may be a 10 to 15 percent increase in the time taken to complete the full backup under high-load conditions.

Tracker reference CORE-2316.

User Manual

P. Vinkenoog

A user manual for NBak/NBackup has been prepared. It can be downloaded from the documentation area at the Firebird website: https://www.firebirdsql.org/pdfmanual/ - the file name is Firebird-nbackup.pdf.

`gbak` backup/porting/restore utility

Changed behaviours, new switches

V. Horsun

The new gbak switch

 -RECREATE_DATABASE [OVERWRITE]

is a separate switch designed to make harder for the unsuspecting to overwrite a database accidentally, as could occur easily with the shortened form of the old switch:

 -R[EPLACE_DATABASE]

In summary:

gbak -R (or gbak -r) now applies to the new -R[ECREATE_DATABASE] switch and will never overwrite an existing database if the O[VERWRITE] argument is absent.
The short form of the old gbak -R[EPLACE_DATABASE] is now -REP[LACE_DATABASE]. This switch does not accept the O[VERWRITE] argument.
The -REP[LACE_DATABASE] switch should be considered as deprecated, i.e. it will become unavailable in some future Firebird release.

This change means that, if you have any legacy batch or cron scripts that rely on gbak -r or gbak -R without modification, then the operation will except if the database exists.

If you want to retain the ability of your script to overwrite your database unconditionally, you will need to modify the command to use either the new switch with the OVERWRITE argument or the new short form for the old -REPLACE_DATABASE switch.

`gbak -V` and the `counter` parameter

During Firebird 1 development, an optional numeric <counter> argument was added to the -V[erbose] switch of gbak for both backup and restore. It was intended to allow you to specify a number and get a running count of rows processed as the row counter passed each interval of that number of rows. It caused undesirable side-effects and was removed before Firebird 1.0 was ever released. So, although it never happened, it was documented as "implemented" in the release notes and other places.

ISQL query utility

Work on ISQL has involved a lot of bug-fixing and the introduction of a few new, useful features. One trick to note is that CHAR and VARCHAR types defined in character set OCTETS (alias BINARY) now display in hex format. Currently, this feature cannot be toggled off.

New switches

The following command-line switches were added:

`-b[ail]` Bail out

D. Ivanov, C. Valderrama

Command line switch -b to instruct isql to bail out on error when used in non-interactive mode, returning an error code to the operating system.

When using scripts as input in the command line, it may be totally unappropriate to let isql continue executing a batch of commands after an error has happened. Therefore, the -b[ail] option will cause script execution to stop at the first error it detects. No further statements in the input script will be executed and isql will return an error code to the operating system.

Most cases have been covered, but if you find some error that is not recognized by isql, you should inform the project, as this is a feature in progress.
Currently there is no differentiation by error code---any non-zero return code should be interpreted as failure. Depending on other options (like -o, -m and -m2), isql will show the error message on screen or will send it to a file.

Some Features

Even if isql is executing nested scripts, it will cease all execution and will return to the operating system when it detects an error. Nested scripts happen when a script A is used as isql input but in turn A contains an INPUT command to load script B an so on. Isql doesn't check for direct or indirect recursion, thus if the programmer makes a mistake and script A loads itself or loads script B that in turn loads script A again, isql will run until it exhaust memory or an error is returned from the database, at whose point -bail if activated will stop all activity.
DML errors will be caught when being prepared or executed, depending on the type of error.
In many cases, isql will return the line number of a DML statement that fails during execution of a script. (More about error line numbers ...)
DDL errors will be caught when being prepared or executed by default, since isql uses AUTODDL ON by default. However, if AUTO DLL [AUTODDL?] is OFF, the server only complains when the script does an explicit COMMIT and this may involve several SQL statements.
The feature can be enabled/disabled interactively or from a script by means of the command

               SET BAIL [ON | OFF]

As is the case with other SET commands, simply using SET BAIL will toggle the state between activated and deactivated. Using SET will display the state of the switch among many others.

Even if BAIL is activated, it doesn't mean it will change isql behaviour. An additional requirement should be met: the session should be non-interactive. A non-interactive session happens when the user calls isql in batch mode, giving it a script as input.

Example

 isql -b -i my_fb.sql -o results.log -m -m2

Tip: However, if the user loads isql interactively and later executes a script with the input command, this is considered an interactive session even though isql knows it is executing a script.

Example

 isql
 Use CONNECT or CREATE DATABASE to specify a database
 SQL> set bail;
 SQL> input my_fb.sql;
 SQL> ^Z

Whatever contents the script has, it will be executed completely, errors and all, even if the BAIL option is enabled.

`-m2` to output stats and plans

C. Valderrama

This is a command-line option -m2 to send the statistics and plans to the same output file as the other output (via the -o[utput] switch).

When the user specifies that the output should be sent to a file, two possibilities have existed for years: either

at the command line, the switch -o followed by a file name is used,
the command OUTput followed by a file name is used, either in a batch session or in the interactive isql shell.

(In either case, simply passing the command OUTput is enough to have the output returned to the console). However, although error messages are shown in the console, they are not output to the file.

The -m command line switch was added, to meld (mix) the error messages with the normal output to wherever the output was being redirected.

This left still another case: statistics about operations (SET STATs command) and SQL plans as the server returns them. SET PLAN and SET PLANONLY commands have been treated as diagnostic messages and, as such, were always sent to the console.

What the -m2 command line switch does is to ensure that stats and plans information go to the same file the output has been redirected to.

Note: Neither -m nor -m2 has an interactive counterpart through a SET command. They are for use only as command-line isql options.

`-r2` to pass a case-sensitive role name

C. Valderrama

The sole objective of this parameter is to specify a case-sensitive role name.

The default switch for this parameter is -r. Roles provided in the command line are uppercased.
With -r2, the role is passed to the engine exactly as typed in the command line.

New commands

The following commands have been added or enhanced.

`SET HEAD[ing]` toggle

C. Valderrama

Some people consider it useful to be able to do a SELECT inside isql and have the output sent to a file, for additional processing later, especially if the number of columns makes isql display impracticable. However, isql by default prints column headers and, in this scenario, they are a nuisance.

Therefore, printing the column headers -- previously a fixed feature -- can now be enabled/disabled interactively or from a script by means of the

 SET HEADing [ON | OFF]

command in the isql shell. As is the case with other SET commands, simply using SET HEAD will toggle the state between activated and deactivated.

Note: There is no command line option to toggle headings off.

Using SET will display the state of SET HEAD, along with other switches that can be toggled on/off in the isql shell.

`SHOW SYSTEM` now shows predefined UDFs

The SHOW <object_type> command is meant to show user objects of that type. The SHOW SYSTEM commmand is meant to show system objects but, until now, it only showed system tables. Now it lists the predefined system UDFs incorporated into FB 2.

It may be enhanced to list system views if we create some of them in the future.

`SET SQLDA_DISPLAY ON/OFF`

A. dos Santos Fernandes

This SQLDA_DISPLAY command shows the input SQLDA parameters of INSERTs, and DELETEs. It was previously available only in DEBUG builds and has now been promoted to the public builds. It shows the information for raw SQLVARs. Each SQLVAR represents a field in the XSQLDA, the main structure used in the FB API to talk to clients transferring data into and out of the server.

Note: The state of this option is not included in the output when you type SET; in isql to see the current settings of most options.

`SET TRANSACTION` enhanced

C. Valderrama

The SET TRANSACTION statement has been enhanced so that, now, all TPB options are supported:

NO AUTO UNDO
IGNORE LIMBO
LOCK TIMEOUT <number>

Example

 SET TRANSACTION WAIT SNAPSHOT NO AUTO UNDO LOCK TIMEOUT 10

See also the document doc/sql.extensions/README.set_transaction.txt.

`SHOW DATABASE` now returns ODS version number

C. Valderrama

ODS (On-Disk Structure) version is now returned in the SHOW DATABASE command. (C. Valderrama)

Ability to show the line number where an error happened in a script

C. Valderrama

In previous versions, the only reasonable way to know where a script had caused an error was using the switch -e for echoing commands, -o to send the output to a file and -m to merge the error output to the same file. This way, you could observe the commands isql executed and the errors if they exist. The script continued executing to the end. The server only gives a line number related to the single command (statement) that it's executing, for some DSQL failures. For other errors, you only know the statement caused problems.

With the addition of -b for bail as described in (-b[ail] Bail out), the user is given the power to tell isql to stop executing scripts when an error happens, but you still need to echo the commands to the output file to discover which statement caused the failure.

Now, the ability to signal the script-related line number of a failure enables the user to go to the script directly and find the offending statement. When the server provides line and column information, you will be told the exact line of DML in the script that caused the problem. When the server only indicates a failure, you will be told the starting line of the statement that caused the failure, related to the whole script.

This feature works even if there are nested scripts, namely, if script SA includes script SB and SB causes a failure, the line number is related to SB. When SB is read completely, isql continues executing SA and then isql continues counting lines related to SA, since each file gets a separate line counter. A script SA includes SB when SA uses the INPUT command to load SB.

Lines are counted according to what the underlying IO layer considers separate lines. For ports using EDITLINE, a line is what readline() provides in a single call. The line length limit of 32767 bytes remains unchanged.

Enhanced command-line help

M. Kubecek

When unknown parameters are used, isql now shows all of the command-line parameters and their explanations instead of just a simple list of allowed switches.

 opt/firebird/bin] isql -?
 Unknown switch: ?
 usage: isql [options] [<database>]
      -a(all) extract metadata incl. legacy non-SQL tables
      -b(ail) bail on errors (set bail on)
      -c(ache) <num> number of cache buffers
      -ch(arset) <charset> connection charset (set names)
      -d(atabase) <database> database name to put in script creation
      -e(cho) echo commands (set echo on)
      -ex(tract) extract metadata
      -i(nput) <file> input file (set input)
      -m(erge) merge standard error
      -m2 merge diagnostic
      -n(oautocommit) no autocommit DDL (set autoddl off)
      -now(arnings) do not show warnings
      -o(utput) <file> output file (set output)
      -pag(elength) <size> page length
      -p(assword) <password> connection password
      -q(uiet) do not show the message "Use CONNECT..."
      -r(ole) <role> role name
      -r2 <role> role (uses quoted identifier)
      -sqldialect <dialect> SQL dialect (set sql dialect)
      -t(erminator) <term> command terminator (set term)
      -u(ser) <user> user name
      -x extract metadata
      -z show program and server version

ISQL bugs fixed

SF #910430 ISQL and database dialect, fixed by C. Valderrama, B. Rodriguez Somoza.
What was fixed When ISQL disconnected from a database, either by dropping it or by trying to connect to a non-existent database, it remembered the SQL dialect of the previous connection, which could lead to some inappropriate warning messages.

SF #223126 Misplaced collation when extracting metadadata with ISQL, fixed by B. Rodriguez Somoza.

SF #223513 Ambiguity between tables and views, fixed by B. Rodriguez Somoza.

SF #518349 ISQL SHOW mangles relationship, fixed by B. Rodriguez Somoza.

Unregistered bug Possible crashes with long terminators, fixed by C. Valderrama.

Unregistered bug Avoided several SQL> prompts when using the INPUT command interactively. Command-line Utilities 90 implemented by C. Valderrama.

Unregistered bugs Some memory leaks, fixed by C. Valderrama

`gsec` Authentication Manager

Changes to the gsec utility include:

`gsec` return code

C. Valderrama

gsec now returns an error code when used as a non-interactive utility. Zero indicates success; any other code indicates failure.

`gfix` server utility

Changes to the gfix utility include:

New shutdown states (modes)

N. Samofatov, D. Yemanov

The options for gfix -shut[down] have been extended to include two extra states or modes to govern the shutdown.

New Syntax Pattern

 gfix <command> [<state>] [<options>]
 <command> ::= {-shut | -online}
 <state> ::= {normal | multi | single | full}
 <options> ::= {-force <timeout> | -tran | -attach}

normal state: online database.
multi state: multi-user shutdown mode (the legacy one, unlimited attachments of SYSDBA/owner are allowed).
single state: single-user shutdown (only one attachment is allowed, used by the restore process).
full state: full/exclusive shutdown (no attachments are allowed).

Note: Multi is the default state for -shut, normal is the default state for -online.

The modes can be switched sequentially:

 normal <-> multi <-> single <-> full

Examples

 gfix -shut single -force 0
 gfix -shut full -force 0
 gfix -online single
 gfix -online

You cannot use -shut to bring a database one level more "online" and you cannot use -online to make a database more protected (an error will be thrown).

These are prohibited:

   gfix -shut single -force 0
   gfix -shut multi -force 0
 ::
   gfix -online
   gfix -online full
 ::
   gfix -shut -force 0
   gfix -online single

Timeout:

As before, the timeout is in seconds. In the case of the -attach and -tran timeouts, the timeout determines how long the engine will wait for any attached clients to complete their work and log off. The shutdown request should return the SQLCode -902 message shutfail (ISC code 335544557), Database shutdown unsuccessful if there are still active attachments when the timeout expires.

However, there is a known issue with the implementation of the new modes. A regression occurred, whereby the said message is returned but the engine does not revert the database to the online state, as it should. It affects all versions of Firebird up to and including v.2.0.5 and v.2.1.3, and all v.2.5 alphas, betas and release candidates.