Multi-Language Programmer’s Guide

Contents

Overview

YottaDB is a multi-language NoSQL database. The daemonless database engine resides in the address space of the process, and can be accessed from any supported language. Functions in the supported languages can call one another to the extent that such calling is permitted by the Supported language implementations.

As C is the lingua franca of programming, the C API provides access to the YottaDB engine from any language. As YottaDB adds standard APIs for other languages, additional sections will be added to the Programmers Guide.

Quick Start

Local Installation

  1. Install YottaDB.
  • Create a temporary directory and change to it, e.g.: mkdir /tmp/tmp ; cd /tmp/tmp
  • Get the YottaDB installer: wget https://gitlab.com/YottaDB/DB/YDB/raw/master/sr_unix/ydbinstall.sh
  • Make it executable: chmod +x ydbinstall.sh
  • Run it with your choice of directory where you want it installed (omit the --verbose option for less output): sudo ./ydbinstall.sh --utf8 default --verbose. If you do not specify an installation directory with --installdir, the script installs YottaDB in /usr/local/lib/yottadb/r### where r### is the release, e.g., r122.
  1. Set up the environment: source $(pkg-config --variable=prefix yottadb)/ydb_env_set. This defaults to an environment in $HOME/.yottadb; to use another directory, set the environment variable ydb_dir to the desired directory.
  2. Put your C program in the $ydb_dir directory, #include the file libyottadb.h in your C program and compile it. As a sample program, you can download the wordfreq.c program, with a reference input file and corresponding reference output file and compile it with gcc $(pkg-config --libs --cflags yottadb) -o wordfreq wordfreq.c -lyottadb.
  3. Run your program and verify that the output matches the reference output. For example:
$ cd $ydb_dir
$ gcc $(pkg-config --libs --cflags yottadb) -o wordfreq wordfreq.c -lyottadb
$ ./wordfreq <wordfreq_input.txt >wordfreq_output.tmp
$ diff wordfreq_output.tmp wordfreq_output.txt
$

Note that the wordfreq.c program randomly uses local or global variables (see Local and Global Variables).

Docker Container

You must have at least Docker 17.05 installed. Pre-built images are available at Docker Hub. The Docker image is built to provide sane defaults to begin exploring YottaDB. It is not meant for production usage.

To run a pre-built image: docker run --rm -it yottadb/yottadb to run the image but not persist any changes you make, and docker run -it yottadb/yottadb for persistent changes.

Volumes are supported by mounting the /data directory. To mount the local directory ydb-data into the container to save your database and routines locally and use them in the container, add an appropriate command line parameter before the yottadb/yottadb argument, e.g., docker run -it -v \`pwd\`/ydb-data:/data yottadb/yottadb

This creates a ydb-data directory in your current working directory. After the container is shutdown and removed, delete the directory if you want to remove all data created in the YottaDB container (such as your database and routines).

Concepts

Keys, Values, Nodes, Variables, and Subscripts

The fundamental core data structure provided by YottaDB is key-value tuples. For example, the following is a set of key value tuples:

["Capital","Belgium","Brussels"]
["Capital","Thailand","Bangkok"]
["Capital","USA","Washington, DC"]

Note that data in YottaDB is always ordered. [1] Even if you input data out of order, YottaDB always stores them in order. In the discussion below, data is therefore always shown in order. For example, the data below may well have been loaded by country.

[1]The terms “collate”, “order”, and “sort” are equivalent.

Each of the above tuples is called a node. In an n-tuple, the first n-1 items can be thought of as the keys, and the last item is the value associated with the keys.

While YottaDB itself assigns no meaning to the data in each node, by convention, application maintainability is improved by using meaningful keys, for example:

["Capital","Belgium","Brussels"]
["Capital","Thailand","Bangkok"]
["Capital","USA","Washington, DC"]
["Population","Belgium",1367000]
["Population","Thailand",8414000]
["Population","USA",325737000]

As YottaDB assigns no inherent meaning to the keys or values, its key value structure lends itself to implementing Variety. [2] For example, if an application wishes to add historical census results under “Population”, the following is a perfectly valid set of tuples (source: United States Census):

["Capital","Belgium","Brussels"]
["Capital","Thailand","Bangkok"]
["Capital","USA","Washington, DC"]
["Population","Belgium",1367000]
["Population","Thailand",8414000]
["Population","USA",325737000]
["Population","USA",17900802,3929326]
["Population","USA",18000804,5308483]
…
["Population","USA",20100401,308745538]

In the above, 17900802 represents August 2, 1790, and an application would determine from the number of keys whether a node represents the current population or historical census data.

[2]Variety is one of the three “V”s of “big data” — Velocity, Volume, and Variety. YottaDB handles all three very well.

In YottaDB, the first key is called a variable, and the remaining keys are called subscripts allowing for a representation both compact and familiar to a programmer, e.g., Capital("Belgium")="Brussels". The set of all nodes under a variable is called a tree (so in the example, there are two trees, one under Capital and the other under Population). The set of all nodes under a variable and a leading set of its subscripts is called a subtree (e.g., Population("USA") is a subtree of the Population tree). [3]

[3]Of course, the ability to represent the data this way does not in any way detract from the ability to represent the same data another way with which you are comfortable, such as XML or JSON. However, note while any data that can be represented in JSON can be stored in a YottaDB tree not all trees that YottaDB is capable of storing can be represented in JSON, or at least, may require some encoding (for example, see JSON-M) - in order to be represented in JSON.

With this representation, the Population tree can be represented as follows:

Population("Belgium")=1367000
Population("Thailand")=8414000
Population("USA")=325737000
Population("USA",17900802)=3929326
Population("USA",18000804)=5308483
…
Population("USA",20100401)=308745538

YottaDB has functions for applications to traverse trees in both breadth-first and depth-first order.

If the application designers now wish to enhance the application to add historical dates for capitals, the Capital("Thailand") subtree might look like this (source: The Four Capitals of Thailand).

Capital("Thailand")="Bangkok"
Capital("Thailand",1238,1378)="Sukhothai"
Capital("Thailand",1350,1767)="Ayutthaya"
Capital("Thailand",1767,1782)="Thonburi"
Capital("Thailand",1782)="Bangkok"

Variables vs. Subscripts vs. Values

When viewed as ["Capital","Belgium","Brussels"] each component is a string, and in an abstract sense they are all conceptually the same. When viewed as Capital("Belgium")="Brussels" differences become apparent:

  • Variables are ASCII strings from 1 to 31 characters, the first of which is “%”, or a letter from “A” through “Z” and “a” through “z”. Subsequent characters are alphanumeric (“A” through “Z”, “a” through “z”, and “0” through “9”). Variable names are case-sensitive, and variables of a given type are always in ASCII order (i.e., “Capital” always precedes “Population”).
  • Subscripts are sequences of bytes from 0 bytes (the null or empty string, “”) to 1048576 bytes (1MiB). When a subscript is a canonical number, YottaDB internally converts it to, and stores it as, a number. When ordering subscripts:
    • Empty string subscripts precede all numeric subscripts. By default, YottaDB prohibits empty string subscripts for global variables but permits them for local variables (see Local and Global Variables). Note: YottaDB recommends against the practice of using empty string subscripts in applications. [4]
    • Numeric subscripts precede string subscripts. Numeric subscripts are in numeric order.
    • String subscripts follow numeric subscripts and collate in byte order. Where the natural byte order does not result in linguistically and culturally correct ordering of strings, YottaDB has a framework for an application to create and use custom collation routines.
[4]The YottaDB code base includes code for a legacy subscript collation in which empty strings collate after numeric subscripts and before non-empty strings. This is supported only in M code for backward compatibility reasons, and is not supported for use with C or any other language. Any attempt to bypass protections and use this legacy collation with new code will almost certainly result in buggy applications that are hard to debug.

Like subscripts, values are sequences of bytes, except that ordering of values is not meaningful unlike ordering of subscripts. YottaDB automatically converts between numbers and strings, depending on the type of operand required by an operator or argument required by a function (see Numeric Considerations).

This means that if an application were to store the current capital of Thailand as Capital("Thailand","current")="Bangkok" instead of Capital("Thailand")="Bangkok", the above subtree would have the following order:

Capital("Thailand",1238,1378)="Sukhothai"
Capital("Thailand",1350,1767)="Ayutthaya"
Capital("Thailand",1767,1782)="Thonburi"
Capital("Thailand",1782)="Bangkok"
Capital("Thailand","current")="Bangkok"

Local and Global Variables

YottaDB is a database, and data in a database must persist and be shared. The variables discussed above are specific to an application process (i.e., are not shared).

  • Local variables reside in process memory, are specific to an application process, are not shared between processes, and do not persist beyond the lifetime of a process. [5]
  • Global variables reside in databases, are shared between processes, and persist beyond the lifetime of any individual process.
[5]In other words, what YottaDB calls a local variable, the C programming language calls a global variable. There is no C counterpart to a YottaDB global variable.

Syntactically, local and global variables look alike, with global variable names having a caret (“^”) preceding their names. Unlike the local variables above, the global variables below are shared between processes and are persistent.

^Population("Belgium")=1367000
^Population("Thailand")=8414000
^Population("USA")=325737000

Even though they may appear superficially similar, a local variable is distinct from a global variable of the same name. Thus ^X can have the value 1 and X can at the same time have the value "The quick brown fox jumps over the lazy dog." For maintainability YottaDB strongly recommends that applications use different names for local and global variables, except in the special case where a local variable is an in-process cached copy of a corresponding global variable.

Global Directories

To application software, files in a file system provide persistence. This means that global variables must be stored in files for persistence. A global directory file provides a process with a mapping from the name of every possible global variable name to one or more regions. A database is a set of regions, which in turn map to database files. Global directories are created and maintained by a utility program, which is discussed at length in Chapter 4 Global Directory Editor of the YottaDB Administration and Operations Guide and is outside the purview of this document.

The name of the global directory file required to access a global variable such as ^Capital, is provided to the process at startup by the environment variable ydb_gbldir.

In addition to the implicit global directory an application may wish to use alternate global directory names. For example, consider an application that wishes to provide an option to display names in other languages while defaulting to English. This can be accomplished by having different versions of the global variable ^Capital for different languages, and having a global directory for each language. A global variable such as ^Population would be mapped to the same database file for all languages, but a global variable such as ^Capital would be mapped to a database file with language-specific entries. So a default global directory Default.gld mapping a ^Capital to a database file with English names can be specified in the environment variable ydb_gbldir but a different global directory file, e.g., ThaiNames.gld can have the same mapping for a global variable such as ^Population but a different database file for ^Capital. The intrinsic special variable $zgbldir can be set to a global directory name to change the mapping from one global directory to another.

Thus, we can have:

$zgbldir="ThaiNames.gld"
^Capital("Thailand")="กรุ่งเทพฯ"
^Capital("Thailand",1238,1378)="สุโขทัย"
^Capital("Thailand",1350,1767)="อยุธยา"
^Capital("Thailand",1767,1782)="ธนบุรี"
^Capital("Thailand",1782)="กรุ่งเทพฯ"

Client/Server Operation

In common usage, database files reside on the same computer system as that running application code. However, as described in Chapter 13 GT.CM Client/Server of the Administration and Operations Guide, database files can reside on a computer system different from that running application code. This mapping of global variables to regions that map to remote files is also performed using global directories, and is transparent to application code except that YottaDB client/server operation does not support transaction processing.

Furthermore, there are configurations that impliticly invoke transaction processing logic, such as distributing a global variable over multiple database regions, or a trigger invocation (see Chapter 14 Triggers of the YottaDB M Programmers Guide). Operations that invoke implicit transaction processing are not supported for global variables that reside on remote database files.

Intrinsic Special Variables

In addition to local and global variables, YottaDB also has a set of Intrinsic Special Variables. Just as global variables are distinguished by a “^” prefix, intrinsic special variables are distinguished by a “$” prefix. Unlike local and global variable names, intrinsic special variable names are case-insensitive and so $zgbldir and $ZGblDir refer to the same intrinsic special variable. Intrinsic special variables have no subscripts.

While the majority of intrinsic special variables as enumerated in Chapter 8 (Intrinsic Special Variables) of the YottaDB M Programmers Guide are useful to M application code, others are more generally useful and documented here.

$tlevel

Application code can read the intrinsic special variable $tlevel to determine whether it is executing inside a transaction. $tlevel>0 means that it is inside a transaction, and $tlevel>1 means that it is inside a nested transaction. Note that a transaction can be started explicitly, e.g., by calling ydb_tp_s() or ydb_tp_st(),or implicitly by a trigger resulting from a ydb_delete_s(), ydb_delete_st(), ydb_set_s() or ydb_set_st().

$trestart

Application code inside a transaction can read the intrinsic special variable $trestart to determine how many times a transaction has been restarted. Although YottaDB recommends against accessing external resources within a transaction, logic that needs to access an external resource (e.g., to read data in a file), can use $trestart to restrict that access to the first time it executes ($trestart=0).

$zgbldir

$zgbldir is the name of the current global directory file; any global variable reference that does not explicitly specify a global directory uses $zgbldir. For example, an application can set an intrinsic special variable $zgbldir="ThaiNames.gld" to use the ThaiNames.gld mapping. At process startup, YottaDB initializes $zgbldir from the environment variable value $ydb_gbldir.

$zmaxtptime

$zmaxtptime provides a limit in seconds for the time that a transaction can be open (see Transaction Processing). $zmaxtptime is initialized at process startup from the environment variable ydb_maxtptime, with values greater than 60 seconds truncated to 60 seconds. In the unlikely event that an application legitimately needs a timeout greater than 60 seconds, use ydb_set_s() or ydb_set_st() to set it.

$zstatus

$zstatus provides additional details of the last error. Application code can retrieve $zstatus using ydb_get_s() or ydb_get_st(). $zstatus consists of several comma-separated substrings.

  • The first is an error number.
  • The second is always "(SimpleAPI)".
  • The remainder is more detailed information about the error, and may contain commas within.

Note that a race condition exists for a multi-threaded application: after a call that returns an error, it is possible for another call from a different thread to perturb the value of $zstatus. Use the errstr parameter discussed in Threads to get the correct $zstatus in a multi-threaded application.

$zyrelease

$zyrelease identifies the YottaDB release in use. It consists of four space separated pieces:

  1. Always “YottaDB”.
  2. The release number, which starts with “r” and is followed by two numbers separated by a period (“.”), e.g., “r1.24”. The first is a major release number and the second is a minor release number under the major release. Even minor release numbers indicate formally released software. Odd minor release numbers indicate software builds from “in flight” code under development, between releases.
  3. The operating system. e.g., “Linux”.
  4. The CPU architecture, e.g., “x86_64”.

Transaction Processing

YottaDB provides a mechanism for an application to implement ACID (Atomic, Consistent, Isolated, Durable) transactions, ensuring strict serialization of transactions, using optimistic concurrency control.

Here is a simplified view [6] of YottaDB’s implementation of optimistic concurrency control:

  • Each database file header has a field of the next transaction number for updates in that database.
  • The block header of each database block in a database file has the transaction number when that block was last updated.
  • When a process is inside a transaction, it keeps track of every database block it has read, and the transaction number of that block when read. Other processes are free to update the database during this time.
  • The process retains updates in its memory, without committing them to the database, so that its own logic sees the updates, but no other process does. As every block that the process wishes to write must also be read, tracking the transaction numbers of blocks read suffices to track them for blocks to be written.
  • To commit a transaction, a process checks whether any block it has read has been updated since it was read. If none has, the process commits the transaction to the database, incrementing the file header fields of each updated database file for the next transaction.
  • If even one block has been updated, the process discards its work, and starts over. If after three attempts, it is still unable to commit the transaction, it executes the transaction logic on the fourth attempt with updates by all other processes blocked so that the transaction at commit time will not encounter database changes made by other processes.
[6]At the high level at which optimistic concurrency control is described here, a single logical database update (which can span multiple blocks and even multiple regions) is a transaction that contains a single update.

In YottaDB’s API for transaction processing, an application packages the logic for a transaction into a function, passing the function to the ydb_tp_s() or ydb_tp_st() functions. YottaDB then calls that function.

  • If the function returns a YDB_OK, YottaDB attempts to commit the transaction. If it is unable to commit as described above, or if the called function returns a YDB_TP_RESTART return code, it calls the function again.
  • If the function returns a YDB_TP_ROLLBACK, ydb_tp_s() or ydb_tp_st() return to the caller with that return code after discarding the uncommitted database updates and releasing any locks acquired within the transaction.
  • To protect applications against poorly coded transactions, if a transaction takes longer than the number of seconds specified by the intrinsic special variable $zmaxtptime, YottaDB aborts the transaction and the ydb_tp_s() or ydb_tp_st() functions return the YDB_ERR_TPTIMEOUT error.

Sections Threads and Threads and Transaction Processing provide important information pertinent to transaction processing in a multi-threaded application.

Nested Transactions

YottaDB allows transactions to be nested. In other words, code executing within a transaction may itself call ydb_tp_s() or ydb_tp_st(). Although ACID properties are only meaningful at the outermost level, nested transactions are nevertheless useful. For example:

  • Application logic can be programmed modularly. Logic that requires ACID properties can be coded as a transaction, without the need to determine whether or not the caller of that logic is itself within a transaction.
  • That local variables can be saved, and restored on transaction restarts, provides useful functionality that nested transactions can exploit.

Locks

YottaDB locks are a fast, lightweight tool for multiple processes to coordinate their work. An analogy with the physical world may help to explain the functionality. When it is locked, the lock on a door prevents you from going through it. In contrast, a traffic light does not stop you from driving through a street intersection: it works because drivers by convention stop when their light is red and drive when it is green.

YottaDB locks are more akin to traffic lights than door locks. Each lock has a name: as lock names have the same syntax as local or global variable names, Population, ^Capital, and ^Capital("Thailand",1350,1767) are all valid lock names. Features of YottaDB locks include:

  • Locks are exclusive: one and only one process can acquire a lock with the resource name. For example, if process P1 acquires lock Population("USA"), process P2 cannot simultaneously acquire that lock. However, P2 can acquire lock Population("Canada") at the same time that process P1 acquires Population("USA").
  • Locks are hierarchical: a process that has a lock at a higher level blocks locks at lower levels and vice versa. For example, if a process P0 must wait for processes P1, P2, … to complete, each of P1, P2, … can acquire lock Process(pid). P0’s subsequent attempt to acquire lock Process is blocked till processes P1, P2, … complete.
  • Locks include counters: a process that acquires ^Capital("Belgium") can acquire that lock again, incrementing its count to 2. This simplifies application code logic: for example, a routine in application code that requires ^Capital("Belgium") can simply incrementally acquire that lock without needing to test whether a higher level routine has already acquired it. More importantly, when it completes its work, the routine can decrementally release the lock without concern for whether or not a higher level routine needs that lock. When the count goes from 1 to 0, the lock becomes available for acquisition by another process.
  • Locks are robust: while normal process exit releases locks held by that process, if a process holding a lock exits abnormally without releasing it, another process that needs the lock, and finding it held by a non-existent process will automatically scavenge the lock.

Although YottaDB lock names are the same as local and global variable names, YottaDB imposes no connection between a lock name and the same variable name. By convention, and for application maintainability, it is good practice to use lock names associated with the variables to which application code requires exclusive access, e.g., use a lock called ^Population to protect or restrict access to a global variable called ^Population. [7]

[7]Since a process always has exclusive access to its local variables, access to them never needs protection from a lock. So, it would be reasonable to use a lock Population to restrict access to the global variable ^Population.

Since YottaDB lock acquisitions are always timed for languages other than M, it is not in principle possible for applications to deadlock on YottaDB locks. Consequently defensive application code must always validate the return code of calls to acquire locks. As a practical matter, it is possible to set timeouts that are long enough that users may perceive applications to be hung.

Since YottaDB resources such as locks belong to a process rather than a thread within a process (see discussion under Threads), design rules to avoid deadlocks (such as acquiring locks in a predefined order that all processes must respect) must be respected by all threads in a process (or for a language such as Go, by all Goroutines in a process).

Locks and Transaction Processing

Transaction Processing and Locks solve overlapping though not congruent use cases. For example, consider application code to transfer $100 from a customer’s savings account to that same customer’s savings account, which would likely include the requirement that business transactions on an account must be serializable. This can be implemented by acquiring a lock on that customer (with an application coded so that other accesses to that customer are blocked till the lock is released) or by executing the transfer inside a YottaDB transaction (which provides ACID properties). Unless the application logic or data force pathological transaction restarts that cannot be eliminated or worked around, transaction processing’s optimistic concurrency control typically results in better application throughput than the pessimistic concurrency control that locks imply.

In general, we recommend using either transaction processing or locks, and not mixing them. However, there may be business logic that requires the use of locks for some logic, but otherwise permits the use of transaction processing. If an application must mix them, the following rules apply:

  • A lock that a process acquires prior to starting a transaction cannot be released inside the transaction - it can only be released after the transaction is committed or rolled back. Locks acquired inside a transaction can be released either inside the transaction, or after the transaction is committed or rolled back.

Programming in C

Symbolic Constants

The libyottadb.h file defines several symbolic constants, which are one of the following types:

  • Function Return Codes, which in turn are one of:
    • Normal Return Codes
    • Error Return Codes
  • Limits
  • Other

Symbolic constants all fit within the range of a C int.

Function Return Codes

Return codes from calls to YottaDB are usually of type int and occasionally other types. Normal return codes are non-negative (greater than or equal to zero); error return codes are negative.

Normal Return Codes

Symbolic constants for normal return codes have YDB_ prefixes other than YDB_ERR_.

YDB_LOCK_TIMEOUT — This return code from lock acquisition functions indicates that the specified timeout was reached without the requested locks being acquired.

YDB_OK — This the standard return code of all functions following successful execution.

YDB_TP_RESTART — Return code to YottaDB from an application function that implements a transaction to indicate that it wishes YottaDB to restart the transaction, or by a YottaDB function invoked within a transaction to its caller that the database engine has detected that it will be unable to commit the transaction and will need to restart. Application code designed to be executed within a transaction should be written to recognize this return code and in turn perform any cleanup required and return to the YottaDB ydb_tp_s() / ydb_tp_st() invocation from which it was called. See Transaction Processing for a discussion of restarts.

YDB_TP_ROLLBACK — Return code to YottaDB from an application function that implements a transaction, and in turn returned to the caller indicating that the transaction was not committed.

Error Return Codes

Symbolic constants for error codes returned by calls to YottaDB are prefixed with YDB_ERR_ and are all less than zero. The symbolic constants below are not a complete list of all error messages that YottaDB functions can return — error return codes can indicate system errors and database errors, not just application errors. A process that receives a negative return code, including one not listed here, can call ydb_get_s() / ydb_get_st() to get the value of $zstatus.

Error messages can be raised by the YottaDB runtime system or by the underlying operating system.

Remember that the error codes returned by YottaDB functions are the negated numeric values of the error codes above.

YDB_ERR_CALLINAFTEREXIT – A YottaDB function was called after ydb_exit() was called.

YDB_ERR_FATALERROR1 – A fatal error occurred. The process is generating a core dump and terminating. As a process cannot receive a fatal error code, this error appears in the syslog.

YDB_ERR_FATALERROR2 – A fatal error occurred. The process is terminating without generating a core dump. As a process cannot receive a fatal error code, this error appears in the syslog.

YDB_ERR_GVUNDEF — No value exists at a requested global variable node.

YDB_ERR_INVNAMECOUNT – A namecount parameter has an invalid value.

YDB_ERR_INSUFFSUBS — A call to ydb_node_next_s() / ydb_node_next_st() or ydb_node_previous_s() / ydb_node_previous_st() did not provide enough parameters for the return values. Note that as the number of parameters is a count, when array subscripts start at 0, an array subscript of n corresponds to n+1 parameters.

YDB_ERR_INVSTRLEN — A buffer provided by the caller is not long enough for a string to be returned, or the length of a string passed as a parameter exceeds YDB_MAX_STR. In the event the return code is YDB_ERR_INVSTRLEN and if *xyz is a ydb_buffer_t structure whose xyz->len_alloc indicates insufficient space, then xyz->len_used is set to the size required of a sufficiently large buffer. In this case the len_used field of a ydb_buffer_t structure is greater than the len_alloc field, and the caller is responsible for correcting the xyz->len_used field.

YDB_ERR_INVSVN — A special variable name provided by the caller is invalid.

YDB_ERR_INVVARNAME — A variable name provided by the caller is invalid.

YDB_ERR_KEY2BIG — The length of a global variable name and subscripts exceeds the limit configured for the database region to which it is mapped.

YDB_ERR_LVUNDEF — No value exists at a requested local variable node.

YDB_ERR_MAXNRSUBSCRIPTS — The number of subscripts specified in the call exceeds YDB_MAX_SUBS.

YDB_ERR_MINNRSUBSCRIPTS – The number of subscripts cannot be negative.

YDB_ERR_NAMECOUNT2HI – The number of variable names specified to ydb_delete_excl_s() / ydb_delete_excl_st() or ydb_tp_s() / ydb_tp_st() exceeded the YDB_MAX_NAMES.

YDB_ERR_NODEEND — In the event a call to ydb_node_next_s() / ydb_node_next_st(), ydb_node_previous_s() / ydb_node_previous_st(), ydb_subscript_next_s() / ydb_subscript_next_st(), or ydb_subscript_previous_s() / ydb_subscript_previous_st() wish to report that there no further nodes/subscripts in their traversals, they return this value.

YDB_NOTOKydb_file_name_to_id() was called with a NULL pointer to a filename.

YDB_ERR_NUMOFLOW — A ydb_incr_s() / ydb_incr_st() operation resulted in a numeric overflow.

YDB_ERR_PARAMINVALID — A parameter provided by the caller is invalid.

YDB_ERR_SIMPLEAPINEST – An attempt was made to nest Simple API calls, which cannot be nested.

YDB_ERR_SUBSARRAYNULL – The subs_used parameter of a function is greater than zero, but the subsarray parameter is a NULL pointer.

YDB_ERR_SVNOSET — the application inappropriately attempted to modify the value of an intrinsic special variable such as an attempt to modify $trestart using ydb_set_s() / ydb_set_st().

YDB_ERR_TIME2LONG – This return code indicates that a value greater than YDB_MAX_TIME_NSEC was specified for a time duration.

YDB_ERR_TPTIMEOUT — This return code from ydb_tp_s() / ydb_tp_st() indicates that the transaction took too long to commit.

YDB_ERR_UNIMPLOP — An operation that is not supported for an intrinsic special variable – of the Simple API functions only ydb_get_s() / ydb_get_st() and ydb_set_s() / ydb_set_st() are supported – was attempted on an intrinsic special variable.

YDB_ERR_VARNAME2LONG – A variable name length exceeds YottaDB’s limit.

Limits

Symbolic constants for limits are prefixed with YDB_MAX_ or YDB_MIN_.

YDB_MAX_IDENT — The maximum space in bytes required to store a complete variable name, not including the preceding caret for a global variable. Therefore, when allocating space for a string to hold a global variable name, add 1 for the caret.

YDB_MAX_NAMES – The maximum number of variable names that can be passed to ydb_delete_excl_s() / ydb_delete_excl_st() or ydb_tp_s() / ydb_tp_st().

YDB_MAX_STR — The maximum length of a string (or blob) in bytes. A caller to ydb_get_s() / ydb_get_st() whose *ret_value parameter provides a buffer of YDB_MAX_STR will never get a YDB_ERR_INVSTRLEN error.

YDB_MAX_SUBS — The maximum number of subscripts for a local or global variable.

YDB_MAX_TIME_NSEC — The maximum value in nanoseconds that an application can instruct libyottab to wait, e.g., until the process is able to acquire locks it needs before timing out, or for ydb_hiber_start().

YDB_MAX_YDBERR – The absolute (positive) value of any YottaDB function error return code. If the absolute value of an error return code is greater than YDB_MAX_YDBERR, then it is an error code from elsewhere, e.g., e.g. errno. Also, see YDB_IS_YDBERR().

YDB_MIN_YDBERR - The absolute (positive) value of any YottaDB function error return code. If the absolute value of an error return code is less than YDB_MIN_YDBERR, then it is an error code from elsewhere, e.g., e.g. errno. Also, see YDB_IS_YDBERR().

Severity

Symbolic constants for the severities of message numbers in return codes and $zstatus are prefixed with YDB_SEVERITY_.

YDB_SEVERITY_ERROR – The number corresponds to an error from which the process can recover.

YDB_SEVERITY_FATAL – The number corresponds to an error that terminated the process.

YDB_SEVERITY_INFORMATION – The number corresponds to an informational message.

YDB_SEVERITY_SUCCESS – The number corresponds to the successful completion of a requested operation.

YDB_SEVERITY_WARNING – The number corresponds to a warning, i.e., it indicates a possible problem.

Other

Other symbolic constants have a prefix of YDB_.

YDB_DEL_NODE and YDB_DEL_TREE — As values of the deltype parameter, these values indicate to ydb_delete_s() / ydb_delete_st() whether to delete an entire subtree or just the node at the root, leaving the subtree intact.

YDB_NOTTP – As a value of the tptoken parameter of the Simple API multi-threaded functions – those ending in _st(), indicates that the caller is not within a transaction.

Data Structures & Type Definitions

ydb_buffer_t is a descriptor for a string [8] value, and consists of the following fields:

  • buf_addr — pointer to an unsigned char, the starting address of a string.
  • len_alloc and len_used — fields of type unsigned int where:
    • len_alloc is the number of bytes allocated to store the string,
    • len_used is the length in bytes of the currently stored string, and
    • len_alloclen_used except when a YDB_ERR_INVSTRLEN occurs.
[8]Strings in YottaDB are arbitrary sequences of bytes that are not null-terminated. Other languages may refer to them as binary data or blobs.

ydb_string_t is a descriptor for a string provided for compatibility with existing code, and consists of the following fields:

  • address — pointer to an unsigned char, the starting address of a string.
  • length — the length of the string starting at the address field.

ydb_tpfnptr_t is a pointer to a function which returns an integer, with one parameter, a pointer to an arbitrary structure:

typedef int (*ydb_tpfnptr_t)(void *tpfnparm);

ydb_tp2fnptr_t is a pointer to a function which returns an integer, with three parameters, a tptoken, a *errstr pointer, and a pointer to an arbitrary structure:

typedef int (*ydb_tp2fnptr_t)(uint64_t tptoken,
        ydb_buffer_t *errstr, void *tpfnparm)

Functions to implement transaction processing logic for single-threaded applications are referenced by ydb_tpfnptr_t and functions to implement transaction processing logic for multi-threaded applications are referenced by ydb_tp2fnptr_t.

Macros

YDB_ASSERT(x) – Conditionally include this macro in code for debugging and testing purposes. If x is non-zero, it prints an error message on stderr and generates a core file by calling ydb_fork_n_core().

YDB_BUFFER_IS_SAME(buffer1, buffer2) – Use this macro to test whether the memory locations (strings) pointed to by two ydb_buffer_t structures have the same content, returning FALSE (0) if they differ and a non-zero value if the contents are identical.

YDB_COPY_BUFFER_TO_BUFFER(source, destination, done) – Use this macro to copy the memory locations (strings) pointed to by source to the memory locations pointed to by destination and set:

  • destination->len_used to source->len_used; and
  • done to TRUE if destination->len_allocsource->len_used and the underlying memcpy() completed successfully, and FALSE otherwise.

YDB_COPY_LITERAL_TO_BUFFER(literal, buffer, done) - Use this macro to copy a literal string to previously allocated memory referenced by a ydb_buffer_t structure (for example, to set an initial subscript to sequence through nodes). It sets:

  • buffer->len_used to the size of the literal; and
  • done to TRUE if buffer->len_alloc ≥ the size of the literal excluding its terminating null byte and the underlying memcpy() completed successfully, and FALSE otherwise.

YDB_COPY_STRING_TO_BUFFER(string, buffer, done) – Use this macro to copy a null-terminated string to previously allocated memory referenced by a ydb_buffer_t structure. This macro requires the code to also #include <string.h>. It sets:

  • buffer->len_used to the size of the copied string; and
  • done to TRUE if buffer->len_alloc ≥ the size of the string to be copied and the underlying memcpy() completed successfully, and FALSE otherwise.

YDB_FREE_BUFFER(BUFFERP) - Use this macro to free the buffer malloced using YDB_MALLOC_BUFFER.

  • free() call is used on BUFFERP->buf_addr.

YDB_LITERAL_TO_BUFFER(literal, buffer) – Use this macro to set a ydb_buffer_t structure to refer to a literal (such as a variable name). With a string literal, and a pointer to a ydb_buffer_t structure, set:

  • buffer->buf_addr to the address of literal; and
  • buffer->len_used and buffer->len_alloc to the length of literal excluding the terminating null byte.

YDB_IS_YDBERR(msgnum) – returns TRUE if the absolute value of msgnum lies between YDB_MIN_YDBERR and YDB_MAX_YDBERR.

YDB_MALLOC_BUFFER(BUFFERP,LEN) - Use this macro to to allocate a buffer using malloc() of length LEN and assign it to an already allocated ydb_buffer_t structure.

  • BUFFERP->buf_addr is set to the malloced buffer.
  • BUFFERP->len_alloc is set to the malloced length.
  • BUFFERP->len_used is set to 0.

YDB_SEVERITY(msgnum, severity) – The error return code from a function indicates both the nature of an error as well as its severity. For message msgnum, the variable severity is set to one of the YDB_SEVERITY_* symbolic constants. YDB_SEVERITY() is only meaningful for error return codes and not other numbers. Use YDB_IS_YDBERR() to determine whether a return code is a YottaDB error return code.

YottaDB functions are divided into:

  • Simple API — a core set of functions that provides easy-to-use access to the major features of YottaDB.
  • Comprehensive API — a more elaborate set of functions for specialized or optimized access to additional functionality within libyottadb.so that YottaDB itself uses. The Comprehensive API is a project for the future.
  • Utility Functions — Functions useful to a C application using YottaDB.

Simple API

As all subscripts and node data passed to YottaDB using the Simple API are strings, use the sprintf() and atoi()/strtoul() family of functions to convert between numeric values and strings which are canonical numbers.

To allow the YottaDB Simple API functions to handle a variable tree whose nodes have varying numbers of subscripts, the actual number of subscripts is itself passed as a parameter. In the prototypes of functions, parameters of the form:

  • ydb_buffer_t *varname refers to the name of a variable;
  • int subs_used and int *subs_used refer to an actual number of subscripts; and
  • ydb_buffer_t *subsarray refers to an array of ydb_buffer_t structures used to pass subscripts whose actual number is defined by subs_used or *subs_used parameters.

To pass an intrinsic special variable, or unsubscripted local or global variable, subs_used should be zero and *subsarray should be NULL.

Caveat: Specifying a subs_used that exceeds the actual number of parameters passed in *subsarray will almost certainly result in an unpleasant bug that is difficult to troubleshoot.

Functions specific to the YottaDB Simple API for single-threaded applications end in _s() and those for multi-threaded applications end in _st(), with the latter functions typically differing from their counterparts of the former type with two additional parameters, tptoken, and errstr. The discussion in Threads provides more detailed information.

ydb_data_s() / ydb_data_st()

int ydb_data_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        unsigned int *ret_value);

int ydb_data_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        unsigned int *ret_value);

In the location pointed to by ret_value, ydb_data_s() and ydb_data_st() return the following information about the local or global variable node identified by *varname, subs_used and *subsarray.

  • 0 — There is neither a value nor a subtree, i.e., it is undefined.
  • 1 — There is a value, but no subtree
  • 10 — There is no value, but there is a subtree.
  • 11 — There are both a value and a subtree.

It is an error to call ydb_data_s() or ydb_data_st() on an intrinsic special variable; doing so results in the YDB_ERR_UNIMPLOP error. ydb_data_s() / ydb_data_st() returns:

The error YDB_ERR_PARAMINVALID is returned when

  • ret_value is NULL
  • len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript, in subsarray.

ydb_delete_s() / ydb_delete_st()

int ydb_delete_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int deltype);

int ydb_delete_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int deltype);

Delete nodes in the local or global variable tree or subtree specified. A value of YDB_DEL_NODE or YDB_DEL_TREE for deltype specifies whether to delete just the node at the root, leaving the (sub)tree intact, or to delete the node as well as the (sub)tree.

Intrinsic special variables cannot be deleted.

ydb_delete_s() and ydb_delete_st() return YDB_OK, a YDB_ERR_UNIMPLOP if deltype is neither YDB_DEL_NODE nor YDB_DEL_TREE, YDB_ERR_PARAMINVALID is returned when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray, or another error return code.

  • YDB_OK;
  • YDB_ERR_UNIMPLOP if deltype is neither YDB_DEL_NODE nor YDB_DEL_TREE; or
  • another error return code.

ydb_delete_excl_s() / ydb_delete_excl_st()

int ydb_delete_excl_s(int namecount,
        ydb_buffer_t *varnames);

int ydb_delete_excl_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        int namecount, ydb_buffer_t *varnames);

ydb_delete_excl_s() and ydb_delete_excl_st() delete the trees of all local variables except those in the *varnames array. It is an error for *varnames to include a global or intrinsic special variable.

In the special case where namecount is zero, ydb_delete_excl_s() and ydb_delete_excl_st() delete all local variables.

If your application mixes M and non M code, and you wish to use ydb_delete_excl_s() to delete local variables that are aliases, formal parameters, or actual parameters passed by reference, make sure you understand what (sub)trees are being deleted. This warning does not apply to applications that do not include M code.

ydb_delete_excl_s() and ydb_delete_excl_st()`return :CODE:`YDB_OK, YDB_ERR_NAMECOUNT2HI if more than YDB_MAX_NAMES are specified, or another error return code. YDB_ERR_PARAMINVALID is returned when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one variable name in “code:varnames.

Note that specifying a larger value for namecount than the number of variable names actually provided in *varnames can result in a buffer overflow.

ydb_get_s() / ydb_get_st()

int ydb_get_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

int ydb_get_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

To the location pointed to by ret_value->buf_addr, ydb_get_s() and ydb_get_st() copy the value of the specified node or intrinsic special variable, setting ret_value->len_used on both normal and error returns (the latter case as long as the data exists). Return values are:

  • YDB_OK for a normal return;
  • YDB_ERR_GVUNDEF, YDB_ERR_INVSVN, or YDB_ERR_LVUNDEF as appropriate if no such variable or node exists;
  • YDB_ERR_INVSTRLEN if ret_value->len_alloc is insufficient for the value at the node;
  • YDB_ERR_PARAMINVALID when ret_value is NULL or ret_value->buf_addr is NULL and the return value has a non-zero len_used; or len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray; or
  • another applicable error return code.

Notes:

  • In the unlikely event an application wishes to know the length of the value at a node, but not access the data, it can call ydb_get_s() or ydb_get_st() and provide an output buffer (retvalue->len_alloc) with a length of zero, since even in the case of a YDB_ERR_INVSTRLEN error, retvalue->len_used is set.
  • Within a transaction implemented by ydb_tp_s() / ydb_tp_st() application code observes stable data at global variable nodes because YottaDB transaction processing ensures ACID properties, restarting the transaction if a value changes.
  • Outside a transaction, a global variable node can potentially be changed by another, concurrent, process between the time that a process calls ydb_data_s() / ydb_data_st() to ascertain the existence of the data and a subsequent call to ydb_get_s() / ydb_get_st() to get that data. A caller of ydb_get_s() / ydb_get_st() to access a global variable node should code in anticipation of a potential YDB_ERR_GVUNDEF, unless it is known from application design that this cannot happen.

ydb_incr_s() / ydb_incr_st()

int ydb_incr_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *increment,
        ydb_buffer_t *ret_value);

int ydb_incr_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *increment,
        ydb_buffer_t *ret_value);

ydb_incr_s() and ydb_incr_st() atomically:

  • convert the value in the specified node to a number if it is not one already, using a zero value if the node does not exist;
  • increment it by the value specified by *increment, converting the value to a number if it is not a canonical number, defaulting to 1 if the parameter is NULL; and
  • store the value as a canonical number in *ret_value.

Return values:

  • The normal return value is YDB_OK.
  • If the atomic increment results in a numeric overflow, the function returns a YDB_ERR_NUMOFLOW error; in this case, the value in the node is untouched and that in *ret_value is unreliable.
  • YDB_ERR_INVSTRLEN if ret_value->len_alloc is insufficient for the result. As with ydb_get_s() / ydb_get_st(), in this case ret_value->len_used is set to the required length.
  • Other errors return the corresponding error return code.

Notes:

  • Intrinsic special variables cannot be atomically incremented, and an attempt to do so returns the YDB_ERR_UNIMPLOP error.
  • The value of the empty string coerced to a numeric value is 0.

ydb_lock_s() / ydb_lock_st()

int ydb_lock_s(unsigned long long timeout_nsec,
        int namecount[,
        [ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray], ...]);

int ydb_lock_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        unsigned long long timeout_nsec,
        int namecount[,
        [ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray], ...]);

namecount is the number of variable names in the call.

Release any locks held by the process, and attempt to acquire all the requested locks. Except in the case of an error, the release is unconditional. On return, the function will have acquired all requested locks or none of them. If no locks are requested (namecount is zero), the function releases all locks and returns YDB_OK.

timeout_nsec specifies a time in nanoseconds that the function waits to acquire the requested locks. If timeout_nsec is zero, the function makes exactly one attempt to acquire the locks

Return values:

  • If all requested locks are successfully acquired, the function returns YDB_OK.
  • If it is not able to acquire all requested locks in the specified time, it acquires no locks, returning with a YDB_LOCK_TIMEOUT return value.
  • If the requested timeout_nsec exceeds YDB_MAX_TIME_NSEC, the function immediately returns YDB_ERR_TIME2LONG.
  • YDB_ERR_PARAMINVALID

is returned when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray. - In other cases, the function returns an error return code.

ydb_lock_decr_s() / ydb_lock_decr_st()

int ydb_lock_decr_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray);

int ydb_lock_decr_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray);

Decrements the count of the specified lock held by the process. As noted in the Concepts section, a lock whose count goes from 1 to 0 is released. A lock whose name is specified, but which the process does not hold, is ignored.

As releasing a lock cannot fail, the function returns YDB_OK, unless there is an error such as an invalid name that results in the return of an error code such as YDB_ERR_INVVARNAME. Errors result in an appropriate error return code. YDB_ERR_PARAMINVALID is returned when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray.

ydb_lock_incr_s() / ydb_lock_incr_st()

int ydb_lock_incr_s(unsigned long long timeout_nsec,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray);

int ydb_lock_incr_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        unsigned long long timeout_nsec,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray);

Without releasing any locks held by the process, attempt to acquire the requested lock incrementing it if already held.

timeout_nsec specifies a time in nanoseconds that the function waits to acquire the requested locks. If timeout_nsec is zero, the function makes exactly one attempt to acquire the locks

Return values:

  • If all requested locks are successfully acquired, the function returns YDB_OK.
  • If it is not able to acquire all requested locks in the specified time, it acquires no locks, returning with a YDB_LOCK_TIMEOUT return value.
  • If the requested timeout_nsec exceeds YDB_MAX_TIME_NSEC, the function immediately returns YDB_ERR_TIME2LONG.
  • YDB_ERR_PARAMINVALID

is returned when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray. - In other cases, the function returns an error return code.

ydb_node_next_s() / ydb_node_next_st()

int ydb_node_next_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int *ret_subs_used,
        ydb_buffer_t *ret_subsarray);

int ydb_node_next_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int *ret_subs_used,
        ydb_buffer_t *ret_subsarray);

ydb_node_next_s() and ydb_node_next_st() facilitate depth-first traversal of a local or global variable tree. As the number of subscripts can differ between the input node of the call and the output node reported by the call *ret_subs_used is an input as well as an output parameter:

  • On input, *ret_subs_used specifies the number of elements allocated for returning the subscripts of the next node.
  • On normal output (YDB_OK return code), *ret_subs_used contains the actual number of subscripts returned. See below for error return codes

Return values of ydb_node_next_s() and ydb_node_next_st() are:

  • YDB_OK with the next node, if there is one, changing *ret_subs_used and *ret_subsarray parameters to those of the next node. If there is no next node (i.e., the input node is the last), *ret_subs_used on output is YDB_NODE_END.
  • YDB_ERR_INSUFFSUBS if *ret_subs_used specifies insufficient parameters to return the subscript. In this case *ret_subs_used reports the actual number of subscripts required.
  • YDB_ERR_INVSTRLEN if one of the ydb_buffer_t structures pointed to by *ret_subsarray does not have enough space for the subscript. In this case, *ret_subs_used is the index into the *ret_subsarray array with the error, and the len_used field of that structure specifies the size required.
  • YDB_ERR_NODEEND to indicate that that there are no more nodes. In this case, *ret_subs_used is unchanged.
  • YDB_ERR_PARAMINVALID if ret_subs_used is NULL or ret_subsarray is NULL or one of the ydb_buffer_t structures pointed to by *ret_subsarray has a NULL buf_addr. In the last case, *ret_subs_used is the index into the *ret_subsarray array with the NULL buf_addr.
  • Another error return code, in which case the application should consider the values of *ret_subs_used and the *ret_subsarray to be undefined.

ydb_node_previous_s() / ydb_node_previous_st()

int ydb_node_previous_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int *ret_subs_used,
        ydb_buffer_t *ret_subsarray);

int ydb_node_previous_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        int *ret_subs_used,
        ydb_buffer_t *ret_subsarray);

Analogous to ydb_node_next_s() / ydb_node_next_st(), ydb_node_previous_s() and ydb_node_previous_st() facilitate reverse breadth-first traversal of a local or global variable tree, except that ydb_node_previous_s() and ydb_node_previous_st() search for and report the predecessor node. Unlike ydb_node_next_s() / ydb_node_next_st(), *ret_subs_used can be zero if the previous node is the unsubscripted root.

Return values of ydb_node_previous_s() and ydb_node_previous_st() are:

  • YDB_OK with the previous node, if there is one, changing *ret_subs_used and *ret_subsarray parameters to those of the previous node.
  • YDB_ERR_INSUFFSUBS if *ret_subs_used specifies insufficient parameters to return the subscript. In this case *ret_subs_used reports the actual number of subscripts required.
  • YDB_ERR_INVSTRLEN if one of the ydb_buffer_t structures pointed to by *ret_subsarray does not have enough space for the subscript. In this case, *ret_subs_used is the index into the *ret_subsarray array with the error, and the len_used field of that structure specifies the size required.
  • YDB_ERR_NODEEND to indicate that that there are no more nodes. In this case, *ret_subs_used is unchanged.
  • YDB_ERR_PARAMINVALID if ret_subs_used is NULL or ret_subsarray is NULL or one of the ydb_buffer_t structures pointed to by *ret_subsarray has a NULL buf_addr. In the last case, *ret_subs_used is the index into the *ret_subsarray array with the NULL buf_addr.
  • Another error return code, in which case the application should consider the values of *ret_subs_used and the *ret_subsarray to be undefined.

ydb_set_s() / ydb_set_st()

int ydb_set_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *value);

int ydb_set_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *value);

ydb_set_s() and ydb_set_st() copy the value->len_used bytes at value->buf_addr as the value of the specified node or intrinsic special variable specified. A NULL value parameter is treated as equivalent to one that points to a ydb_buffer_t specifying an empty string. Return values are:

  • YDB_OK for a normal return;
  • YDB_ERR_INVSVN if no such intrinsic special variable exists;
  • YDB_ERR_PARAMINVALID when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray or increment; or
  • another applicable error return code.

ydb_str2zwr_s() / ydb_str2zwr_st()

int ydb_str2zwr_s(ydb_buffer_t *str, ydb_buffer_t *zwr);

int ydb_str2zwr_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *str, ydb_buffer_t *zwr);

In the buffer referenced by *zwr, ydb_str2zwr_s() and ydb_str2zwr_st() provide the zwrite formatted version of the string pointed to by *str, returning:

  • YDB_OK;
  • YDB_ERR_INVSTRLEN if the *zwr buffer is not long enough;
  • YDB_ERR_PARAMINVALID if zwr is NULL or zwr->buf_addr is NULL and the return value has a non-zero len_used; or
  • another applicable error return code.

ydb_subscript_next_s() / ydb_subscript_next_st()

int ydb_subscript_next_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

int ydb_subscript_next_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

ydb_subscript_next_s() and ydb_subscript_next_st() provide a primitive for implementing breadth-first traversal of a tree by searching for the next subscript at the level specified by subs_used, i.e., the next subscript after the one referred to by subsarray[subs_used-1].buf_addr. A node need not exist at the subscripted variable name provided as input to the function. If subsarray[subs_used-1].len_used is zero, ret_value->buf_addr points to first node at that level with a subscript that is not the empty string. ydb_subscript_next_s() and ydb_subscript_next_st() return:

  • YDB_OK, in which case ret_value->buf_addr points to the value of that next subscript;
  • YDB_ERR_NODEEND when there are no more subscripts at that level, in which case *ret_value is unchanged;
  • YDB_ERR_PARAMINVALID when
    • ret_value is NULL;
    • ret_value->buf_addr is NULL and the return value has a non-zero len_used; or
    • len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray
  • or another error return code.

In the special case where subs_used is zero, and the function returns YDB_OK, ret_value->buf_addr points to the next local or global variable name, with YDB_ERR_NODEEND indicating an end to the traversal.

ydb_subscript_previous_s() / ydb_subscript_previous_st()

int ydb_subscript_previous_s(ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

int ydb_subscript_previous_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *varname,
        int subs_used,
        ydb_buffer_t *subsarray,
        ydb_buffer_t *ret_value);

ydb_subscript_previous_s() and ydb_subscript_previous_st() provide a primitive for implementing reverse breadth-first traversal of a tree by searching for the previous subscript at the level specified by subs_used. i.e. the subscript preceding the one referred to by subsarray[subs_used-1].buf_addr. A node need not exist at the subscripted variable name provided as input to the function. If subsarray[subs_used-1].len_used is zero, ret_value->buf_addr points to last node at that level with a subscript that is not the empty string. ydb_subscript_previous_s() and ydb_subscript_previous_st() return:

  • YDB_OK, in which case ret_value->buf_addr points to the value of that previous subscript;
  • YDB_ERR_NODEEND when there are no more subscripts at that level, in which case *ret_value is unchanged;
  • YDB_ERR_PARAMINVALID when
    • ret_value is NULL;
    • ret_value->buf_addr is NULL and the return value has a non-zero len_used; or
    • len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one subscript in subsarray
  • or another error return code.

In the special case where subs_used is zero, and the function returns YDB_OK, ret_value->buf_addr points to the previous local or global variable name, with YDB_ERR_NODEEND indicating an end to the traversal.

ydb_tp_s() / ydb_tp_st()

int ydb_tp_s(ydb_tpfnptr_t tpfn,
        void *tpfnparm,
        const char *transid,
        int namecount,
        ydb_buffer_t *varnames);

int ydb_tp_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_tp2fnptr_t tpfn,
        void *tpfnparm,
        const char *transid,
        int namecount,
        ydb_buffer_t *varnames);

ydb_tp_s() and ydp_tp_st() call the function :code:referenced by tpfn passing it tpfnparm as a :code:parameter. Additionally, ydb_tp_st() also generates a :code:new tptoken that it passes as a parameter to the :code:function referenced by its tpfn parameter.

As discussed under Transaction Processing, a function implementing transaction processing logic should use the intrinsic special variable $trestart to manage any externally visible action (which YottaDB recommends against, but which may be unavoidable). The function referenced by tpfn should return one of the following:

  • YDB_OK — application logic indicates that the transaction can be committed (the YottaDB engine may still decide that a restart is required to ensure ACID transaction properties) as discussed under Transaction Processing.
  • YDB_TP_RESTART — application logic indicates that the transaction should restart.
  • YDB_TP_ROLLBACK — application logic indicates that the transaction should not be committed.
  • YDB_ERR_PARAMINVALID when len_alloc < len_used or the len_used is non-zero and buf_addr is NULL in at least one variable name in varnames.
  • An error return code returned by a YottaDB function called by the function.

transid is a string, up to the first 8 bytes of which are recorded in the commit record of journal files for database regions participating in the transaction. If not NULL or the empty string, a case-insensitive value of "BA" or "BATCH" indicates that at transaction commit, YottaDB need not ensure Durability (it always ensures Atomicity, Consistency, and Isolation). Use of this value may improve latency and throughput for those applications where an alternative mechanism (such as a checkpoint) provides acceptable Durability. If a transaction that is not flagged as "BATCH" follows one or more transactions so flagged, Durability of the later transaction ensures Durability of the the earlier "BATCH" transaction(s).

If namecount>0, varnames[i] where 0≤i<namecount specifies local variable names whose values are restored to their original values when the transaction is restarted. In the special case where namecount=1 and varnames[0] provides the value "*", all local variables are restored on a restart. It is an error for a varnames to include a global or intrinsic special variable.

A top level ydb_tp_s() and ydb-tp_st() can return:

A ydb_tp_s() or ydb_tp_st() call that is within another transaction (i.e., a nested transaction) can also return YDB_TP_RESTART to its caller. [9]

[9]An enclosing transaction can result not just from another ydb_tp_s() or ydb_tp_st() higher in the stack, but also (for single-threaded applications) from an M tstart command as well as a database trigger resulting from a ydb_delete_s() / ydb_delete_st(), or ydb_set_s() / ydb_set_st().

ydb_zwr2str_s() / ydb_zwr2str_st()

int ydb_zwr2str_s(ydb_buffer_t *zwr, ydb_buffer_t *str);

int ydb_zwr2str_st(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_buffer_t *zwr, ydb_buffer_t *str);

In the buffer referenced by *str, ydb_zwr2str_s() and ydb_zwr2str_st() provide the string described by the zwrite formatted string pointed to by *zwr, returning

  • YDB_OK (with str->len_used set to zero if the zwrite formatted string has an error);
  • YDB_ERR_INVSTRLEN error if the *str buffer is not long enough;
  • YDB_ERR_PARAMINVALID either if the *str buffer is NULL or the return value contains a non-zero len_used and the str->buf_addr is NULL.

Comprehensive API

The Comprehensive API is a project for the future.

Utility Functions

Utility functions are functions that are not core to YottaDB functionality, but which are useful to application code.

Utility functions whose names end in _t() are for use by multi-threaded applications, and those which do not are for single-threaded applications. The discussion in Threads provides more detailed information.

Functions such as ydb_exit(), ydb_fork_n_core(), and ydb_init(), which do not have separate variants for single- and multi-threaded applications, are suitable for both.

See also the description of the ydb_ci_t() and ydb_cip_t() functions in the Programmers Guide.

ydb_child_init()

YottaDB r1.22 and before required the use of a function ydb_child_init() immediately after a fork() to avoid database damage and other possible side-effects.

Effective YottaDB r1.24, this function is not needed. It gets automatically invoked by YottaDB as needed. Any existing usages of this function in an application can be safely removed assuming YottaDB r1.24 or later is in use.

ydb_exit()

int ydb_exit(void)

When a caller no longer wishes to use YottaDB, a call to ydb_exit() cleans up the process connection/access to all databases and cleans up its data structures. Therafter, any attempt to call a YottaDB function produces a YDB_ERR_CALLINAFTEREXIT error.

Note that a typical application should not need to call ydb_exit(), but should instead just terminate with a call to exit() which will perform any cleanup needed by YottaDB.

ydb_file_id_free() / ydb_file_id_free_t()

int ydb_file_id_free(ydb_fileid_ptr_t fileid)

int ydb_file_id_free_t(uint64_t tptoken,
        ydb_buffer_t *errstr, ydb_fileid_ptr_t fileid)

Releases the memory used by a fileid structure previously generated by ydb_file_name_to_id() or ydb_file_name_to_id_t(). Calling the function twice for the same pointer, unless it has been returned a second time by a different ydb_file_name_to_id() or ydb_file_name_to_id_t() is an application error with undefined consequences.

ydb_file_is_identical() / ydb_file_is_identical_t()

int ydb_file_is_identical(ydb_fileid_ptr_t fileid1,
        ydb_fileid_ptr_t fileid2)

int ydb_file_is_identical_t(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_fileid_ptr_t fileid1,
        ydb_fileid_ptr_t fileid2)

Given two pointers to fileid structures (see ydb_file_name_to_id() / ydb_file_name_to_id_t()), ydb_file_is_identical() and ydb_file_is_identical_t() return YDB_OK if the two fileid structures are the same file and YDB_NOTOK otherwise.

ydb_file_name_to_id() / ydb_file_name_to_id_t()

int ydb_file_name_to_id(ydb_string_t *filename,
        ydb_fileid_ptr_t *fileid)

int ydb_file_name_to_id_t(uint64_t tptoken,
        ydb_buffer_t *errstr,
        ydb_string_t *filename,
        ydb_fileid_ptr_t *fileid)

As a file is potentially reachable through different paths, and application code may need to check whether two paths do indeed lead to the same file, YottaDB provides a mechanism to do so. Provided with a path to a file, YottaDB creates an internal structure called a fileid that uniquely identifies the file if such a structure does not already exist for that file, and provides the caller with a pointer to that structure. The layout and contents of the fileid structure are opaque to the caller, which must not modify the pointer or the structure it points to.

When the fileid structure for a file is no longer needed, an application should call ydb_file_id_free() or ydb_file_id_free_t() to release the structure and avoid a memory leak.

ydb_file_name_to_id() and ydb_file_name_to_id_t() :code:return YDB_OK, YDB_NOTOK if the filename is :code:NULL, or an error return code.

ydb_fork_n_core()

void ydb_fork_n_core(void)

A core is a snapshot of a process, to help debug application code, for example to troubleshoot an out-of-design condition. When a process executes ydb_fork_n_core(), it forks. The child process sends itself a signal to generate a core and terminate. On termination of the child process, the parent process continues execution. Note that depending on the nature of the condition necessitating a core, an exit() may well be the right action for the parent process. An exit() call will drive YottaDB exit handlers to perform clean shutdown of databases and devices the process has open.

The content, location, and naming of cores is managed by the operating system – see man 5 core for details. We recommend that you set kernel.core_uses_pid to 1 to make it easier to identify and track cores. As cores will likely contain protected confidential information, you must ensure appropriate configuration and management of cores.

In a multi-threaded environment, only the thread that executes ydb_fork_n_core() or ydb_fork_n_core() survives in the child and is dumped.

ydb_free()

void ydb_free(void *ptr)

Releases memory previously allocated by ydb_malloc(). Passing ydb_free() a pointer not previously provided to the application by ydb_malloc() can result in unpredictable behavior. The signature of ydb_free() matches that of the POSIX free() call.

Just like other SimpleAPI functions, ydb_free() should not be used in multiple threads in multi-threaded programs. (See the Threads section for details). However, the YDB_FREE_BUFFER macro is safe to use in multiple threads.

ydb_hiber_start()

int ydb_hiber_start(unsigned long long sleep_nsec)

The process or thread sleeps for the time in nanoseconds specified by sleep_nsec. If a value greater than YDB_MAX_TIME_NSEC is specified, ydb_hiber_start() immediately returns with a YDB_ERR_TIME2LONG error; otherwise they return YDB_OK after the elapsed time.

ydb_hiber_start_wait_any()

int ydb_hiber_start_wait_any(unsigned long long sleep_nsec)

The process or thread sleeps for the time in nanoseconds specified by sleep_nsec or until it receives a signal. If a value greater than YDB_MAX_TIME_NSEC is specified, ydb_hiber_start_wait_any() immediately returns with a YDB_ERR_TIME2LONG error; otherwise they return YDB_OK after the elapsed time or when the wait is terminated by a signal.

ydb_init()

int ydb_init(void)

ydb_init() initializes the YottaDB runtime environment. As YottaDB automatically initializes the runtime on the first call to its API or first M code invocation, there is usually no need to explicitly call ydb_init(). The exception is when an application wishes to set its own signal handlers (see Signals): ydb_init() sets signal handlers, and in case an application wishes to set its own signal handlers for signals not used by YottaDB, it can call ydb_init() before setting its signal handlers.

ydb_malloc()

void *ydb_malloc(size_t size)

With a signature matching that of the POSIX malloc() call, ydb_malloc() returns an address to a block of memory of the requested size, or NULL if it is unable to satisfy the request. ydb_malloc() uses a buddy system, and provides debugging functionality under the control of the environment variable ydb_dbglvl whose values are a mask as described in gtmdbglvl.h.

Just like other SimpleAPI functions, ydb_malloc() should not be used in multiple threads in multi-threaded programs. (See the Threads section for details). However, the YDB_MALLOC_BUFFER macro is safe to use in multiple threads.

ydb_message() / ydb_message_t()

int ydb_message(int errnum, ydb_buffer_t *msg_buff)

int ydb_message_t(uint64_t tptoken, ydb_buffer_t *errstr,
        int errnum, ydb_buffer_t *msg_buff)

The functions return the error message text template for the error number specified by errnum.

  • If errnum does not correspond to an error that YottaDB recognizes, the return the error YDB_ERR_UNKNOWNSYSERR, leaving the structures referenced by msg_buff unaltered.
  • Otherwise, if the length of the text exceeds msg_buff->len_alloc they return the error YDB_ERR_INVSTRLEN. In this case msg_buff->len_used is greater than msg_buff->len_alloc.
  • Otherwise, if msg_buff->buf_addr is NULL, they return the error YDB_ERR_PARAMINVALID.
  • Otherwise, the copy the text to the buffer specified by msg_buff->buf_addr, set msg_buff->len_used to its length, and return YDB_OK.

ydb_stdout_stderr_adjust() / ydb_stdout_stderr_adjust_t()

int ydb_stdout_stderr_adjust(void)

int ydb_stdout_stderr_adjust_t(uint64 tptoken,
        ydb_buffer_t *errstr)

The functions check whether stdout (file descriptor 1) and stderr (file descriptor 2) are the same file, and if so, route stderr writes to stdout instead. This ensures that output appears in the order in which it was written; otherwise owing to IO buffering, output can appear in an order different from that in which it was written. Application code which mixes C and M code, and which explicitly redirects stdout or stderr (e.g., using dup2()), should call one of these functions as soon as possible after the redirection. ydb_stdout_stderr_adjust() and ydb_stdout_stderr_adjust_t() return YDB_OK.

ydb_thread_is_main()

int ydb_thread_is_main(void)

The functions return YDB_OK if the thread is the main thread of the process, and another value if the thread is not. YottaDB recommends against application code that requires use of these functions, which exist only to provide backward compatibility to a specific application code base (see discussion under Threads).

ydb_timer_cancel() / ydb_timer_cancel_t()

void ydb_timer_cancel(intptr_t timer_id)

void ydb_timer_cancel_t(uint64_t tptoken,
        ydb_buffer_t *errstr, intptr_t timer_id)

Cancel a timer identified by timer_id and previously started with ydb_timer_start() or ydb_timer_start_t().

ydb_timer_start() / ydb_timer_start_t()

typedef void (*ydb_funcptr_retvoid_t)(intptr_t timer_id,
        unsigned int handler_data_len,
        char *handler_data);

int ydb_timer_start(intptr_t timer_id,
        unsigned long long limit_nsec,
        ydb_funcptr_retvoid_t handler,
        unsigned int handler_data_len
        char *handler_data);

int ydb_timer_start_t(uint64_t tptoken,
        ydb_buffer_t *errstr,
        intptr_t timer_id,
        unsigned long long limit_nsec,
        ydb_funcptr_retvoid_t handler,
        unsigned int handler_data_len
        char *handler_data);

Start a timer. Unless canceled, when the timer expires, ydb_timer_start() and ydb_timer_start_t() invoke a handler function, providing that function with input data.

timer_id is an identifier for the the timer. It is the responsibility of application code to ensure that timer_id is different from those of any other active / pending timers.

limit_nsec is the minimum number of nanoseconds before the timer expires and invokes the handler function. Owing to overhead and system load, the actual time will almost always be greater than this value.

handler is a pointer to the function to be called when the timer expires.

handler_data is a pointer to the data to be passed to handler and handler_data_len is the length of the data at *handler_data. Note that the data it points to must be on the heap rather than on the stack, as the stack frame may no longer be valid when the timer expires.

If the requested timeout_nsec exceeds YDB_MAX_TIME_NSEC, the functions return YDB_ERR_TIME2LONG; otherwise they return YDB_OK.

Programming in Go

Programming YottaDB in the Go language is accomplished through a wrapper for Simple API threaded functions that uses cgo to provide a “yottadb” package for access from Go application code. The wrapper must be installed on a system after YottaDB is installed.

There are two Go APIs:

  • Go Easy API aims to be a straighforward, easy-to-use API to access YottaDB without limiting the functionality of YottaDB. The Go Easy API consists of Go Easy API Functions that use standard Go data types and structures.
  • Go Simple API aims to improve performance by reducing copying between Go and YottaDB heaps by defining structures BufferT, BufferTArray, and KeyT which contain pointers to structures and data in the YottaDB heap. Go Simple API functionality is provided by Go methods where a method can meaningfully be associated with a structure, and by Go functions where there is no meaningful association with a structure.

Except for triggers, which are written in M and which can exist in the same process as Go code because they run in a special, isolated, environment, Go and M code in the same processs is not supported.

As the Go language has important differences from C (for example, it has structures with methods but lacks macros), below are Go-specific sections of the Quick Start, Concepts, Symbolic Constants, Data Structures & Type Definitions, Simple API and Utility Functions sections above. The sections below that are specific to Go are intended to supplement, but not subsume, their C counterparts.

Go application code must not directly use the YottaDB C API structures and functions (those prefixed by C. or described in the C Simple API above) as such usage bypasses important controls, but should instead use the structures, methods and functions exposed by the YottaDB Go wrapper. C. prefixed structures and functions are mentioned only for clarity in documentation and brevity of explanation. For example, C.ydb_buffer_t is the C ydb_buffer_t structure defined in Data Structures & Type Definitions.

All subsections of the Programming in Go section are prefixed with “Go” to ensure unique names for hyperlinking.

As Go implementations are inherently multi-threaded, where the C Simple API provides separate functions for use in multi-threaded applications, e.g., ydb_get_s() vs. ydb_get_st()), the Go wrapper wraps the function for use in multi-threaded applications. Also, as Go is multi-threaded, calls include a errstr parameter to get the correct $zstatus for each call.

Go Quick Start

The YottaDB Go wrapper requires a minimum YottaDB version of r1.24 and is tested with a minimum Go version of 1.11. If the Golang packages on your operating system are older, and the Go wrapper does not work, please obtain and install a newer Golang implementation.

The Go Quick Start assumes that YottaDB has already been installed as described in the Quick Start section. After completing step 1 (Installing YottaDB), download the Go wrapper, install it and test it.

$ go get lang.yottadb.com/go/yottadb
$ go build lang.yottadb.com/go/yottadb
$ source $(pkg-config --variable=prefix yottadb)/ydb_env_set
$ go get -t lang.yottadb.com/go/yottadb
$ go test lang.yottadb.com/go/yottadb
ok      lang.yottadb.com/go/yottadb     0.194s
$

There are a number of programs in the go/src/lang.yottadb.com/go/yottadb directory that you can look at.

  1. Put your GO program in a directory of your choice, e.g., $ydb_dir directory and change to that directory. As a sample program, you can download the wordfreq.go program, with a reference input file and corresponding reference output file. Compile it thus: go build wordfreq.go.
  2. Run your program and verify that the output matches the reference output. For example:
$ cd $ydb_dir
$ wget https://gitlab.com/YottaDB/DB/YDBTest/raw/master/go/inref/wordfreq.go
$ go build wordfreq.go
$ wget https://gitlab.com/YottaDB/DB/YDBTest/raw/master/simpleapi/outref/wordfreq_input.txt
$ wget https://gitlab.com/YottaDB/DB/YDBTest/raw/master/simpleapi/outref/wordfreq_output.txt
$ ./wordfreq <wordfreq_input.txt >wordfreq_output_go.txt
$ diff wordfreq_output_go.txt wordfreq_output.txt
$

Note that the wordfreq.go program randomly uses local or global variables (see Local and Global Variables).

Go Concepts

As the YottaDB wrapper is distributed as a Go package, function calls to YottaDB are prefixed in Go code with yottadb. (e.g., application code to call the ValST() function would be written yottadb.ValST(…).

Go Error Interface

YottaDB has a comprehensive set of error return codes. Each has a unique number and a mnemonic. Thus, for example, to return an error that a buffer allocated for a return value is not large enough, YottaDB uses the INVSTRLEN error code, which has the numeric value yottadb.YDB_ERR_INVSTRLEN. YottaDB attempts to maintain stability of the numeric values and mnemonics from release to release, to ensure applications remain compatible when the underlying YottaDB releases are upgraded. While the Go error interface provides for a call to return an error as a string (with nil for a successful return), applications in other languages, such as C, expect a numeric return value.

Where C application code calling YottaDB functions will check the return code, and if it is not YDB_OK access the intrinsic special variable $zstatus for more detailed information (through the errstr parameter in a multi-threaded application), Go application code calling YottaDB methods and functions will check the error interface to determine whether it is nil. This means that Go application code will never see a yottadb.YDB_OK return.

The YottaDB Go error interface has a structure and a method. Sample usage:

 _, err := yottadb.ValE(yottadb.NOTTP, nil, "^hello", []string{})
if err != nil {
    errcode := yottadb.ErrorCode(err)
 }

In the documentation:

  • Error codes specific to each function are noted. However, common errors can also be returned. For example, while the Go BufferT ValStr() method can return INVSTRLEN, it can also return errors from the YottaDB engine.
  • An error name such as INVSTRLEN refers to the underlying error, whether application code references the numeric value or the string.

Go Symbolic Constants

YottaDB symbolic constants are available in the YottaDB package, for example, yottadb.YDB_ERR_INVSTRLEN.

NOTTP (yottadb.NOTTP) as a value for parameter tptoken indicates to the invoked YottaDB method or function that the caller is not inside a transaction.

Go Easy API

A global or local variable node, or an intrinsic special variable, is specified using the construct varname string, subary []string. For an intrinsic special variable, subary must be the null array, []string{}, or nil. For a global or local variable, a null array or nil for subary refers to the root node, the entire tree, or both, depending on the function and context.

As the Go Easy API involves more copying of data between the Go and YottaDB runtime systems, it requires the CPU to perform a little more work than the Go Simple API does. Whether or not this has a measurable impact on performance depends on the application and workload.

The length of strings (values and subscripts) in YottaDB is variable, as is the number of subscripts a local or global variable can have. The Go Simple API requires application code to allocate memory for buffers, raising errors when allocated memory (either size or number of buffers) is insufficient. Requiring application code using the Go Easy API to similarly allocate memory would be at odds with our goal of having it “just work”. Although YottaDB provides functionality to a priori determine the length of a value in order to allocate required memory, doing this for every call would adversely affect performance. The Go Easy API therefore allocates buffers initially of a size and number we believe to be reasonable. Whenever a result exceeds its allocation and returns an error, YottaDB expands the allocation transparently to the caller, and repeats the operation, remembering the expanded size for future allocations in the process.

Go Easy API Functions

Go DataE()

func DataE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) (uint32, error)

Matching Go DataST(), DataE() function wraps and returns the result of ydb_data_st() (0, 1, 10, or 11). In the event of an error, the return value is unspecified.

Go DeleteE()

func DeleteE(tptoken uint64,  errstr *BufferT,
        deltype int, varname string, subary []string) error

Matching Go DeleteST(), DeleteE() wraps ydb_delete_st() to delete a local or global variable node or (sub)tree, with a value of yottadb.YDB_DEL_NODE for deltype specifying that only the node should be deleted, leaving the (sub)tree untouched, and a value of yottadb.YDB_DEL_TREE specifying that the node as well as the (sub)tree are to be deleted.

Go DeleteExclE()

func DeleteExclE(tptoken uint64,
         errstr *BufferT, varnames []string) error

Matching Go DeleteExclST(), DeleteExclE() wraps ydb_delete_excl_st() to delete all local variables except those specified. In the event varnames has no elements (i.e., []string{}), DeleteExclE() deletes all local variables.

In the event that the number of variable names in varnames exceeds yottadb.YDB_MAX_NAMES, the error return is ERRNAMECOUNT2HI. Otherwise, if ydb_delete_excl_st() returns an error, the function returns the error.

As mixing M and Go application code in the same process is not supported, the warning in ydb_delete_excl_s() does not apply.

Go IncrE()

func IncrE(tptoken uint64, errstr *BufferT,
        incr, varname string, subary []string) (string, error)

Matching Go IncrST(), IncrE() wraps ydb_incr_st() to atomically increment the referenced global or local variable node coerced to a number with incr coerced to a number, with the result stored in the node and returned by the function.

  • If ydb_incr_st() returns an error such as NUMOFLOW or INVSTRLEN, the function returns the error.
  • Otherwise, it returns the incremented value of the node.

With a nil value for incr, the default increment is 1. Note that the value of the empty string coerced to an integer is zero, but 1 is a more useful default value for an omitted parameter in this case.

Go LockDecrE()

func LockDecrE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) error

Matching Go LockDecrST() LockDecrE() wraps ydb_lock_decr_st() to decrement the count of the lock name referenced, releasing it if the count goes to zero or ignoring the invocation if the process does not hold the lock.

Go LockE()

func LockE(tptoken uint64, errstr *BufferT,
        timeoutNsec uint64, namesnsubs ... interface{}) error

Matching Go LockST(), LockE() releases all lock resources currently held and then attempts to acquire the named lock resources referenced. If no lock resources are specified, it simply releases all lock resources currently held and returns.

interface{} is a series of pairs of varname string and subary []string parameters, where a null subary parameter ([]string{}) specifies the unsubscripted lock resource name.

If lock resources are specified, upon return, the process will have acquired all of the named lock resources or none of the named lock resources.

  • If timeoutNsec exceeds yottadb.YDB_MAX_TIME_NSEC, the function returns with an error return of TIME2LONG.
  • If the lock resource names exceeds the maximum number supported (currently eleven), the function returns a PARMOFLOW error.
  • If namesnsubs is not a series of alternating string and []string parameters, the function returns the INVLNPAIRLIST error.
  • If it is able to aquire the lock resource(s) within timeoutNsec nanoseconds, the function returns holding the lock resource(s); otherwise it returns LOCKTIMEOUT. If timeoutNsec is zero, the function makes exactly one attempt to acquire the lock resource(s).

Go LockIncrE()

func LockIncrE(tptoken uint64, errstr *BufferT,
        timeoutNsec uint64,
        varname string, subary []string) error

Matching Go LockIncrST(), LockIncrE() wraps ydb_lock_incr_st() to attempt to acquire the referenced lock resource name without releasing any locks the process already holds.

  • If the process already holds the named lock resource, the function increments its count and returns.
  • If timeoutNsec exceeds yottadb.YDB_MAX_TIME_NSEC, the function returns with an error return TIME2LONG.
  • If it is able to aquire the lock resource within timeoutNsec nanoseconds, it returns holding the lock, otherwise it returns LOCKTIMEOUT. If timeoutNsec is zero, the function makes exactly one attempt to acquire the lock.

Go NodeNextE()

func NodeNextE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) ([]string, error)

Matching Go NodeNextST(), NodeNextE() wraps ydb_node_next_st() to facilitate depth first traversal of a local or global variable tree. A node or subtree does not have to exist at the specified key.

  • If there is a next node, it returns the subscripts of that next node.
  • If there is no node following the specified node, the function returns the NODEEND error.

Go NodePrevE()

func NodePrevE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) ([]string, error)

Matching Go NodePrevST(), NodePrevE() wraps ydb_node_previous_st() to facilitate reverse depth first traversal of a local or global variable tree. A node or subtree does not have to exist at the specified key.

  • If there is a previous node, it returns the subscripts of that previous node; an empty string array if that previous node is the root.
  • If there is no node preceding the specified node, the function returns the NODEEND error.

Go SetValE()

func SetValE(tptoken uint64, errstr *BufferT,
        value, varname string, subary []string) error

Matching Go SetValST(), at the referenced local or global variable node, or the intrinsic special variable, SetValE() wraps ydb_set_st() to set the value specified by value.

Go SubNextE()

func SubNextE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) (string, error)

Matching Go SubNextST(), SubNextE() wraps ydb_subscript_next_st() to facilitate breadth-first traversal of a local or global variable sub-tree. A node or subtree does not have to exist at the specified key.

  • At the level of the last subscript, if there is a next subscript with a node and/or a subtree, it returns that subscript.
  • If there is no next node or subtree at that level of the subtree, the function returns the NODEEND error.

In the special case where subary is the null array, SubNextE() returns the name of the next global or local variable, and the NODEEND error if there is no global or local variable following varname.

Go SubPrevE()

func SubPrevE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) (string, error)

Matching Go SubPrevST(), SubPrevE() wraps ydb_subscript_previous_st() to facilitate reverse breadth-first traversal of a local or global variable sub-tree. A node or subtree does not have to exist at the specified key.

  • At the level of the last subscript, if there is a previous subscript with a node and/or a subtree, it returns that subscript.
  • If there is no previous node or subtree at that level of the subtree, the function returns the NODEEND error.

In the special case where subary is the null array SubNextE() returns the name of the previous global or local variable, and the NODEEND error if there is no global or local variable preceding varname.

Go TpE()

func TpE(tptoken uint64, errstr *BufferT,
        tpfn func(uint64, *BufferT) int32, transid string,
        varnames []string) error

Matching Go TpST(), TpE() wraps ydb_tp_st() to implement Transaction Processing.

Refer to Go TpST() for a more detailed discussion of YottaDB Go transaction processing.

Go ValE()

func ValE(tptoken uint64, errstr *BufferT,
        varname string, subary []string) (string, error)

Matching Go ValST(), ValE() wraps ydb_get_st() to return the value at the referenced global or local variable node, or intrinsic special variable.

  • If ydb_get_s() returns an error such as GVUNDEF, INVSVN, LVUNDEF, the function returns the error.
  • Otherwise, it returns the value at the node.

Go Simple API

The Go Simple API consists of Go Data Structures & Type Definitions, Go Simple API Access Methods, Go Simple API BufferT Methods, Go Simple API BufferTArray Methods, Go Simple API KeyT Methods and Go Simple API Functions. Each of them wraps a function in the C Simple API – refer to the descriptions of those functions for more detailed information. The majority of the functionality is in Go Simple API KeyT Methods.

Go Data Structures & Type Definitions

The C.ydb_buffer_t structure, which is the ydb_buffer_t structure described in Data Structures & Type Definitions is used to pass values between Go application code and YottaDB. The design pattern is that the ydb_buffer_t structures are in memory managed by YottaDB. Go structures contain pointers to the YottaDB structures so that when the Go garbage collector moves Go structures, the pointers they contain remain valid.

There are three structures for the interface between YottaDB and Go: BufferT for data, BufferTArray for a list of subscripts or a set of variable names, KeyT for keys where a key in turn consists of a variable or lock resource name and subscripts, as discussed in Concepts.

Methods for each structure are classified as either Go Simple API Access Methods or Go Simple API methods. Go Simple API Access Methods are methods implemented in the Go wrapper for managing the structures for data interchange. Go Simple API methods wrap functionality exposed by the YottaDB API.

Go Simple API Access Methods

Go Simple API Access Methods for BufferT

Go BufferT Alloc()
func (buffer *BufferT) Alloc(nBytes uint32)

Allocate a buffer in YottaDB heap space of size nBytes; and set BufferT structure to provide access to that buffer.

Go BufferT Dump()
func (buffer *BufferT) Dump()

For debugging purposes, dump on stdout:

  • cbuft as a hexadecimal address;
  • for the C.ydb_buffer_t structure referenced by cbuft:
    • buf_addr as a hexadecimal address, and
    • len_alloc and len_used as integers; and
  • at the address buf_addr, the lower of len_used or len_alloc bytes in zwrite format.

As this function is intended for debugging and provides details of internal structures, its output is likely to change as internal implementations change, even when stability of the external API is maintained.

Go BufferT DumpToWriter()
func (buffer *BufferT) DumpToWriter(writer io.writer)

For debugging purposes, dump on writer:

  • cbuft as a hexadecimal address;

  • for the C.ydb_buffer_t structure referenced by cbuft:

    • buf_addr as a hexadecimal address, and
    • len_alloc and len_used as integers; and
  • at the address buf_addr, the lower of len_used or len_alloc bytes in zwrite format.

    As this function is intended for debugging and provides details of

internal structures, its output is likely to change as internal implementations change, even when stability of the external API is maintained.

Go BufferT Free()
func (buffer *BufferT) Free()

The inverse of the Alloc() method: release the buffer in YottaDB heap space referenced by the C.ydb_buffer_t structure, release the C.ydb_buffer_t, and set cbuft in the BufferT structure to nil.

Go BufferT BufferTFromPtr()
func (buffer *BufferT) BufferTFromPtr(errstr unsafe.Pointer)

This method is intended for use in advanced cases, such as those encountered internally to the wrapper, when a BufferT object must wrap an existing C.ydb_buffer_t struct.

Note: Modifying errstr, or accessing memory it references may lead to code that behaves unpredictably and is hard to debug.

Go BufferT LenAlloc()
func (buffer *BufferT) LenAlloc(tptoken uint64,
        errstr *BufferT) (uint32, error)
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • Otherwise, return the len_alloc field of the C.ydb_buffer_t structure referenced by cbuft.
Go BufferT LenUsed()
func (buffer *BufferT) LenUsed(tptoken uint64,
        errstr *BufferT) (uint32, error)
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the len_used field of the C.ydb_buffer_t structure is greater than its len_alloc field (owing to a prior INVSTRLEN error), return an INVSTRLEN error and the len_used field of the C.ydb_buffer_t structure referenced by cbuft.
  • Otherwise, return the len_used field of the C.ydb_buffer_t structure referenced by cbuft.
Go BufferT SetLenUsed()
func (buffer *BufferT) SetLenUsed(tptoken uint64,
        errstr *BufferT, newLen uint32) error

Use this method to change the length of a used substring of the contents of the buffer referenced by the buf_addr field of the referenced C.ydb_buffer_t.

  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If newLen is greater than the len_alloc field of the referenced C.ydb_buffer_t, make no changes and return with an error return of INVSTRLEN.
  • Otherwise, set the len_used field of the referenced C.ydb_buffer_t to newLen.

Note that even if newLen is not greater than the value of len_alloc, setting a len_used value greater than the number of meaningful bytes in the buffer will likely lead to hard-to-debug errors.

Go BufferT SetValBAry()
func (buffer *BufferT) SetValBAry(tptoken uint64,
        errstr *BufferT, val *[]byte) error
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the length of val is greater than the len_alloc field of the C.ydb_buffer_t structure referenced by cbuft, make no changes and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferT SetValStr()
func (buffer *BufferT) SetValStr(tptoken uint64,
        errstr *BufferT, val *string) error
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the length of val is greater than the len_alloc field of the C.ydb_buffer_t structure referenced by cbuft, make no changes and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferT SetValStrLit()
func (buffer *BufferT) SetValStrLit(tptoken uint64,
        errstr *BufferT, val string) error
  • If the the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the length of val is greater than the len_alloc field of the C.ydb_buffer_t structure referenced by cbuft, make no changes and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferT ValBAry()
func (buffer *BufferT) ValBAry(tptoken uint64,
        errstr *BufferT) (*[]byte, error)
  • If the the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the len_used field of the C.ydb_buffer_t structure is greater than its len_alloc field (owing to a prior INVSTRLEN error), return an INVSTRLEN error and len_alloc bytes as a byte array.
  • Otherwise, return len_used bytes of the buffer as a byte array.
Go BufferT ValStr()
func (buffer *BufferT) ValStr(tptoken uint64,
        errstr *BufferT) (*string, error)
  • If the the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If the len_used field of the C.ydb_buffer_t structure is greater than its len_alloc field (owing to a prior INVSTRLEN error), return an INVSTRLEN error and len_alloc bytes as a string.
  • Otherwise, return len_used bytes of the buffer as a string.

Go Simple API Access Methods for BufferTArray

Go BufferTArray Alloc()
func (buftary *BufferTArray) Alloc(numBufs, nBytes uint32)

Allocate an array of numSubs buffers in YottaDB heap space, each of of size bufSiz, referenced by the BufferTArray structure.

Go BufferTArray Dump()
func (buftary *BufferTArray) Dump()

For debugging purposes, dump on stdout:

  • cbuftary as a hexadecimal address;
  • elemsAlloc and elemsUsed as integers;
  • for each element of the smaller of elemsAlloc and elemsUsed elements of the C.ydb_buffer_t array referenced by cbuftary:
    • buf_addr as a hexadecimal address, and
    • len_alloc and len_used as integers; and
    • the smaller of len_used and len_alloc bytes at the address buf_addr, in zwrite format.

As this function is intended for debugging and provides details of internal structures, its output is likely to change as internal implementations change, even when stability of the external API is maintained.

Go BufferTArray DumpToWriter()
func (buftary *BufferTArray) DumpToWriter(writer io.writer)

For debugging purposes, dump on writer:

  • cbuftary as a hexadecimal address;
  • elemsAlloc and elemsUsed as integers;
  • for each element of the smaller of elemsAlloc and elemsUsed elements of the C.ydb_buffer_t array referenced by cbuftary:
    • buf_addr as a hexadecimal address, and
    • len_alloc and len_used as integers; and
    • the smaller of len_used and len_alloc bytes at the address buf_addr, in zwrite format.

As this function is intended for debugging and provides details of internal structures, its output is likely to change as internal implementations change, even when stability of the external API is maintained.

Go BufferTArray ElemAlloc()
func (buftary *BufferTArray) ElemAlloc() uint32
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • Otherwise, return the number of allocated buffers.
Go BufferTArray ElemLenAlloc()
func (buftary *BufferTArray)
        ElemLenAlloc(tptoken uint64) uint32
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • Otherwise, return the len_alloc from the C.ydb_buffer_t structures referenced by cbuftary, all of which have the same value.
Go BufferTArray ElemLenUsed()
func (buftary *BufferTArray) ElemLenUsed(tptoken uint64,
        errstr *BufferT, idx uint32) (uint32, error)
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to the elemsAlloc of the BufferTArray structure, return with an error return of INSUFFSUBS.
  • Otherwise, return the len_used field of the array element specifed by idx of the C.ydb_buffer_t array referenced by cbuftary.
Go BufferTArray ElemUsed()
func (buftary *BufferTArray) ElemUsed() uint32
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • Otherwise, return the value of the elemsUsed field.
Go BufferTArray Free()
func (buftary *BufferTArray) Free()

The inverse of the Alloc() method: release the numSubs buffers and the C.ydb_buffer_t array. Set cbuftary to nil, and elemsAlloc and elemsUsed to zero.

Go BufferTArray SetElemLenUsed()
func (buftary *BufferTArray)
        SetElemLenUsed(tptoken uint64, errstr *BufferT,
        idx, newLen uint32) error

Use this method to set the number of bytes in C.ydb_buffer_t structure referenced by cbuft of the array element specified by idx, for example to change the length of a used substring of the contents of the buffer referenced by the buf_addr field of the referenced C.ydb_buffer_t.

  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc, make no changes and return an INSUFFSUBS error.
  • If newLen is greater than the len_alloc field of the referenced C.ydb_buffer_t, make no changes and return an INVSTRLEN error.
  • Otherwise, set the len_used field of the referenced C.ydb_buffer_t to newLen.

Note that even if newLen is not greater than the value of len_alloc, using a len_used value greater than the number of meaningful bytes in the buffer will likely lead to hard-to-debug errors.

Go BufferTArray SetElemUsed()
func (buftary *BufferTArray)
        SetElemUsed(tptoken uint64, errstr *BufferT,
        newUsed uint32) error

Use this method to set the current number of valid strings (subscripts or variable names) in the BufferTArray.

  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If newUsed is greater than elemsAlloc, make no changes and return with an error return of INSUFFSUBS.
  • Otherwise, set elemsUsed to newUsed.

Note that even if newUsed is not greater than the value of elemsAlloc, using an elemsUsed value greater than the number of valid values in the array will likely lead to hard-to-debug errors.

Go BufferTArray SetValBAry()
func (buftary *BufferTArray) SetValBAry(tptoken uint64,
        errstr *BufferT, idx uint32, val *[]byte) error
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc make no changes and return with an error return of INSUFFSUBS.
  • If the length of val is greater than the len_alloc field of the array element specified by idx, make no changes, and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferTArray SetValStr()
func (buftary *BufferTArray)
        SetValStr(tptoken uint64, errstr *BufferT,
        idx uint32, value *string) error
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc make no changes and return with an error return of INSUFFSUBS.
  • If the length of val is greater than the len_alloc field of the array element specified by idx, make no changes, and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferTArray SetValStrLit()
func (buftary *BufferTArray)
        SetValStrLit(tptoken uint64, errstr *BufferT,
        idx uint32, value string) error
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc make no changes and return with an error return of INSUFFSUBS.
  • If the length of val is greater than the len_alloc field of the C.ydb_buffer_t structure indexed by idx and referenced by cbuft, make no changes and return INVSTRLEN.
  • Otherwise, copy the bytes of val to the referenced buffer and set the len_used field to the length of val.
Go BufferTArray ValBAry()
func (buftary *BufferTArray)
        ValBAry(tptoken uint64, errstr *BufferT,
        idx uint32) (*[]byte, error)
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc, return a zero length byte array and an error return of INSUFFSUBS.
  • If the len_used field of the C.ydb_buffer_t structure specified by idx is greater than its len_alloc field (owing to a previous INVSTRLEN error), return a byte array containing the len_alloc bytes at buf_addr and an INVSTRLEN error.
  • Otherwise, return a byte array containing the len_used bytes at buf_addr.
Go BufferTArray ValStr()
func (buftary *BufferTArray)
        ValStr(tptoken uint64, errstr *BufferT,
        idx uint32) (*string, error)
  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If idx is greater than or equal to elemsAlloc, return a zero length string and an error return of INSUFFSUBS.
  • If the len_used field of the C.ydb_buffer_t structure specified by idx is greater than its len_alloc field (owing to a previous INVSTRLEN error), return a string containing the len_alloc bytes at buf_addr and the INVSTRLEN error.
  • Otherwise, return a string containing the len_used bytes at buf_addr.

Go Simple API Access Methods for KeyT

As the members of KeyT are visible to Go programs (they start with upper-case letters), and application code can call the BufferT methods on Varnm and BufferTArray methods on SubAry, the Go KeyT Alloc(), Go KeyT Dump() and Go KeyT Free() methods are available for programming convenience.

Go KeyT Alloc()
func (key *KeyT) Alloc(varSiz, numSubs, subSiz uint32)

Invoke Varnm.Alloc(varSiz) (see Go BufferT Alloc()) and SubAry.Alloc(numSubs, subSiz) (see Go BufferTArray Alloc()).

Go KeyT Dump()
func (key *KeyT) Dump()

Invoke Varnm.Dump() (see Go BufferT Dump()) and SubAry.Dump() (see Go BufferTArray Dump()).

Go KeyT DumpToWriter()
func (key *KeyT) DumpToWriter(writer io.writer)

Invoke Varnm.Dump() (see Go BufferT Dump()) and SubAry.Dump() (see Go BufferTArray Dump()), sending the output to writer.

Go KeyT Free()
func (key *KeyT) Free()

Invoke Varnm.Free() (see Go BufferT Free()) and SubAry.Free() (see Go BufferTArray Free()).

Go Simple API BufferT Methods

Go Str2ZwrST()

func (buft *BufferT) Str2ZwrST(tptoken uint64,
        errstr *BufferT, zwr *BufferT) error

The method wraps ydb_str2zwr_st() to provide the string in zwrite format.

  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If len_alloc is not large enough, set len_used to the required length, and return an INVSTRLEN error. In this case, len_used will be greater than len_alloc until corrected by application code, and the value referenced by zwr is unspecified.
  • Otherwise, set the buffer referenced by buf_addr to the zwrite format string, and set len_used to the length.

Note that the length of a string in zwrite format is always greater than or equal to the string in its original, unencoded format.

Go Zwr2StrST()

func (buft *BufferT) Zwr2StrST(tptoken uint64,
        errstr *BufferT, str *BufferT) error

This method wraps ydb_zwr2str_st() and is the inverse of Go Str2ZwrST().

  • If the underlying structures have not yet been allocated, return the STRUCTNOTALLOCD error.
  • If len_alloc is not large enough, set len_used to the required length, and return an INVSTRLEN error. In this case, len_used will be greater than len_alloc until corrected by application code.
  • If str has errors and is not in valid zwrite format, set len_used to zero, and return the error code returned by ydb_zwr2str_s() e.g., INVZWRITECHAR`.
  • Otherwise, set the buffer referenced by buf_addr to the unencoded string, set len_used to the length.

Go Simple API BufferTArray Methods

Go DeleteExclST()

func (buftary *BufferTArray)
        DeleteExclST(tptoken uint64, errstr *BufferT) error

DeleteExclST() wraps ydb_delete_excl_st() to delete all local variable trees except those of local variables whose names are specified in the BufferTArray structure. In the special case where elemsUsed is zero, the method deletes all local variable trees.

In the event that the elemsUsed exceeds yottadb.YDB_MAX_NAMES, the error return is ERRNAMECOUNT2HI.

As mixing M and Go application code in the same process is not supported, the warning in ydb_delete_excl_s() does not apply.

Go TpST()

func (buftary *BufferTArray) TpST(tptoken uint64,
        errstr *BufferT, tpfn func(uint64, *BufferT) int,
        transid string) error

TpST() wraps ydb_tp_st() to implement Transaction Processing. tpfn is a function with two parameters, the first of which is a tptoken and the second is an errstr.

As an alternative to parameters for tpfn, create closures.

A function implementing logic for a transaction should return int with one of the following:

  • A normal return (YDB_OK) to indicate that per application logic, the transaction can be committed. The YottaDB database engine will commit the transaction if it is able to, as discussed in Transaction Processing, and if not, will call the function again.
  • YDB_TP_RESTART to indicate that the transaction should restart, either because application logic has so determined or because a YottaDB function called by the function has returned TPRESTART.
  • YDB_TP_ROLLBACK to indicate that TpST() should not commit the transaction, and should return ROLLBACK to the caller.

The BufferTArray receiving the TpST() method is a list of local variables whose values should be saved, and restored to their original values when the transaction restarts. If the cbuftary structures have not been allocated or elemsUsed is zero, no local variables are saved and restored; and if elemsUsed is 1, and that sole element references the string “*” all local variables are saved and restored.

A case-insensitive value of “BA” or “BATCH” for transid indicates to YottaDB that it need not ensure Durability for this transaction (it continues to ensure Atomicity, Consistency, and Isolation), as discussed under ydb_tp_st().

Please see both the description of ydb_tp_st() and the sections on Transaction Processing and Threads and Transaction Processing for details.

Go Simple API KeyT Methods

KeyT methods return errors returned by methods that invoke the underlying Varnm and SubAry members of KeyT structures.

Go DataST()

func (key *KeyT) DataST(tptoken uint64,
        errstr *BufferT) (uint32, error)

Matching Go DataE(), DataST() returns the result of ydb_data_st() (0, 1, 10, or 11). In the event of an error, the return value is unspecified.

Go DeleteST()

func (key *KeyT) DeleteS(tptoken uint64,
        errstr *BufferT, deltype int) error

Matching Go DeleteE(), DeleteST() wraps ydb_delete_st() to delete a local or global variable node or (sub)tree, with a value of yottadb.YDB_DEL_NODE for deltype specifying that only the node should be deleted, leaving the (sub)tree untouched, and a value of yottadb.YDB_DEL_TREE specifying that the node as well as the (sub)tree are to be deleted.

Go IncrST()

func (key *KeyT) IncrST(tptoken uint64,
        errstr *BufferT, incr, retval *BufferT) error

Matching Go IncrE(), IncrST() wraps ydb_incr_st() to atomically increment the referenced global or local variable node coerced to a number, with incr coerced to a number. It stores the result in the node and also returns it through the BufferT structure referenced by retval.

  • If ydb_incr_st() returns an error such as NUMOFLOW, the method makes no changes to the structures under retval and returns the error.
  • If the length of the data to be returned exceeds retval.lenAlloc, the method sets the len_used of the C.ydb_buffer_t referenced by retval to the required length, and returns an INVSTRLEN error. The value referenced by retval is unspecified.
  • Otherwise, it copies the data to the buffer referenced by the retval.buf_addr, sets retval.lenUsed to its length.

With a nil value for incr, the default increment is 1. Note that the value of the empty string coerced to an integer is zero, but 1 is a more useful default value for an omitted parameter in this case.

Go LockDecrST()

func (key *KeyT) LockDecrS(tptoken uint64,
        errstr *BufferT) error

Matching Go LockDecrE() LockDecrST() wraps ydb_lock_decr_st() to decrement the count of the lock name referenced, releasing it if the count goes to zero or ignoring the invocation if the process does not hold the lock.

Go LockIncrST()

func (key *KeyT) LockIncrST(tptoken uint64,
        errstr *BufferT, timeoutNsec uint64) error

Matching Go LockIncrE(), LockIncrST() wraps ydb_lock_incr_st() to attempt to acquire the referenced lock resource name without releasing any locks the process already holds.

  • If the process already holds the named lock resource, the method increments its count and returns.
  • If timeoutNsec exceeds yottadb.YDB_MAX_TIME_NSEC, the method returns with an error return TIME2LONG.
  • If it is able to aquire the lock resource within timeoutNsec nanoseconds, it returns holding the lock, otherwise it returns LOCK_TIMEOUT. If timeoutNsec is zero, the method makes exactly one attempt to acquire the lock.

Go NodeNextST()

func (key *KeyT) NodeNextST(tptoken uint64,
        errstr *BufferT, next *BufferTArray) error

Matching Go NodeNextE(), NodeNextST() wraps ydb_node_next_st() to facilitate depth first traversal of a local or global variable tree. A node or subtree does not have to exist at the specified key.

  • If there is a subsequent node:
    • If the number of subscripts of that next node exceeds next.elemsAlloc, the method sets next.elemsUsed to the number of subscripts required, and returns an INSUFFSUBS error. In this case the elemsUsed is greater than elemsAlloc.
    • If one of the C.ydb_buffer_t structures referenced by next (call the first or only element n) has insufficient space for the corresponding subscript, the method sets next.elemsUsed to n, and the len_alloc of that C.ydb_buffer_t structure to the actual space required. The method returns an INVSTRLEN error. In this case the len_used of that structure is greater than its len_alloc.
    • Otherwise, it sets the structure next to reference the subscripts of that next node, and next.elemsUsed to the number of subscripts.
  • If there is no subsequent node, the method returns the NODEEND error (yottadb.YDB_ERR_NODEEND), making no changes to the structures below next.

Go NodePrevST()

func (key *KEyT) NodePrevST(tptoken uint64,
        errstr *BufferT, prev *BufferTArray) error

Matching Go NodePrevE(), NodePrevST() wraps ydb_node_previous_st() to facilitate reverse depth first traversal of a local or global variable tree. A node or subtree does not have to exist at the specified key.

  • If there is a previous node:
    • If the number of subscripts of that previous node exceeds prev.elemsAlloc, the method sets prev.elemsUsed to the number of subscripts required, and returns an INSUFFSUBS error. In this case the elemsUsed is greater than elemsAlloc.
    • If one of the C.ydb_buffer_t structures referenced by prev (call the first or only element n) has insufficient space for the corresponding subscript, the method sets prev.elemsUsed to n, and the len_alloc of that C.ydb_buffer_t structure to the actual space required. The method returns an INVSTRLEN error. In this case the len_used of that structure is greater than its len_alloc.
    • Otherwise, it sets the structure prev to reference the subscripts of that prev node, and prev.elemsUsed to the number of subscripts.
  • If there is no previous node, the method returns the NODEEND error making no changes to the structures below prev.

Go SetValST()

func (key *KeyT) SetST(tptoken uint64,
        errstr *BufferT, value *BufferT) error

Matching Go SetValE(), at the referenced local or global variable node, or the intrinsic special variable, SetValST() wraps ydb_set_st() to set the value specified by value.

Go SubNextST()

func (key *KeyT) SubNextST(tptoken uint64,
        errstr *BufferT, retval *BufferT) error

Matching Go SubNextE(), SubNextST() wraps ydb_subscript_next_st() to facilitate breadth-first traversal of a local or global variable sub-tree. A node or subtree does not have to exist at the specified key.

  • At the level of the last subscript, if there is a next subscript with a node and/or a subtree:
    • If the length of that next subscript exceeds sub.len_alloc, the method sets sub.len_used to the actual length of that subscript, and returns an INVSTRLEN error. In this case sub.len_used is greater than sub.len_alloc.
    • Otherwise, it copies that subscript to the buffer referenced by sub.buf_addr, and sets sub.len_used to its length.
  • If there is no next node or subtree at that level of the subtree, the method returns the NODEEND error.

Go SubPrevST()

func (key *KeyT) SubPrevST(tptoken uint64,
        errstr *BufferT, retval *BufferT) error

Matching Go SubPrevE(), SubPrevST() wraps ydb_subscript_previous_st() to facilitate reverse breadth-first traversal of a local or global variable sub-tree. A node or subtree does not have to exist at the specified key.

  • At the level of the last subscript, if there is a previous subscript with a node and/or a subtree:
    • If the length of that previous subscript exceeds sub.len_alloc, the method sets sub.len_used to the actual length of that subscript, and returns an INVSTRLEN error. In this case sub.len_used is greater than sub.len_alloc.
    • Otherwise, it copies that subscript to the buffer referenced by sub.buf_addr, and sets buf.len_used to its length.
  • If there is no previous node or subtree at that level of the subtree, the method returns the NODEEND error.

Go ValST()

func (key *KeyT) ValST(tptoken uint64,
        errstr *BufferT, retval *BufferT) error

Matching Go ValE(), ValST() wraps ydb_get_st() to return the value at the referenced global or local variable node, or intrinsic special variable, in the buffer referenced by the BufferT structure referenced by retval.

  • If ydb_get_st() returns an error such as GVUNDEF, INVSVN, LVUNDEF, the method makes no changes to the structures under retval and returns the error.
  • If the length of the data to be returned exceeds retval.GetLenAlloc(), the method sets the len_used of the C.ydb_buffer_t referenced by retval to the required length, and returns an INVSTRLEN error.
  • Otherwise, it copies the data to the buffer referenced by the retval.buf_addr, and sets retval.lenUsed to the length of the returned value.

Go Simple API Functions

Go LockST()

func LockST(tptoken uint64, errstr *BufferT,
        timeoutNsec uint64, lockname ... *KeyT) error

Matching Go LockE(), LockST() wraps ydb_lock_st() to release all lock resources currently held and then attempt to acquire the named lock resources referenced. If no lock resources are specified, it simply releases all lock resources currently held and returns.

If lock resources are specified, upon return, the process will have acquired all of the named lock resources or none of the named lock resources.

  • If timeoutNsec exceeds yottadb.YDB_MAX_TIME_NSEC, the method returns with a TIME2LONG error.
  • If the number of lock resource names exceeds the maximum number supported (currently eleven), the function returns a PARMOFLOW error.
  • If it is able to aquire the lock resource(s) within timeoutNsec nanoseconds, it returns holding the lock resource(s); otherwise it returns LOCKTIMEOUT. If timeoutNsec is zero, the method makes exactly one attempt to acquire the lock resource(s).

Go Comprehensive API

The Go Comprehensive API is a project for the future, to follow the C Comprehensive API

Go Utility Functions

Go Exit()

func Exit() error

For a process that wishes to close YottaDB databases and no longer use YottaDB, the function wraps ydb_exit() so that any further calls to YottaDB return a CALLINAFTEREXIT` error.

Although in theory typical processes should not need to call Exit() because normal process termination should close databases cleanly, in practice thread shutdown may not always ensure that databases are closed cleanly. So, application code should invoke Exit() prior to process exit, or when an application intends to continue with other work beyond use of YottaDB.

Go IsLittleEndian()

func IsLittleEndian() bool

The function returns true if the underlying computing infrastructure is little endian and false otherwise.

Go MessageT()

func Message(tptoken uint64, errstr *BufferT,
        status int) (string, error)

MessageT() returns the text template for the error number specified by status.

  • If status does not correspond to an error that YottaDB recognizes, it returns the error UNKNOWNSYSERR.
  • Otherwise, it returns the error message text template for the error number specified by status.

Go ReleaseT()

func ReleaseT(tptoken uint64, errstr *BufferT) string

Returns a string consisting of six space separated pieces to provide version information for the Go wrapper and underlying YottaDB release:

  • The first piece is always “gowr” to idenfify the Go wrapper.
  • The Go wrapper release number, which starts with “v” and is followed by three numbers separated by a period (“.”), e.g., “v0.90.0” mimicking Semantic Versioning. The first is a major release number, the second is a minor release number under the major release and the third is a patch level. Even minor and patch release numbers indicate formally released software. Odd minor release numbers indicate software builds from “in flight” code under development, between releases. Note that although they follow the same format, Go wrapper release numbers are different from the release numbers of the underlying YottaDB release as reported by $zyrelease.
  • The third through sixth pieces are $zyrelease from the underlying YottaDB relase.

Go Programming Notes

These Go Programming Notes supplement rather than supplant the more general Programming Notes for C.

Goroutines

In order to avoid restricting Go applications to calling the single-threaded YottaDB engine from a single goroutine (which would be unnatural to a Go programmer), the YottaDB Go wrapper calls the functions of the C Simple API that support multi-threaded applications, and includes logic to maintain the integrity of the engine.

Directly calling YottaDB C API functions bypasses this protection, and may result in unpredictable results (Murphy says that unpredictable results will occur when you least expect them). Therefore, Go application code should only call the YottaDB API exposed in this Programming in Go section.

Goroutines in a process are dynamically mapped by the Go implementation to run on threads within that process. Since YottaDB resources are held by the process rather than by the thread or the Goroutine, refer to the Threads discussion about the need for applications to avoid race conditions when accessing YottaDB resources.

Programming in M

YottaDB includes a complete implementation of the M programming language (also known as MUMPS - see The Heritage and Legacy of M (MUMPS) – and the Future of YottaDB )) that mostly conforms to ISO/IEC 11756:1999. The YottaDB M Programmers Guide documents programming YottaDB in M and is not duplicated here.

YottaDB supports calling between M and C application code, as
documented in Chapter 11 (Integrating External Routines) of the M Programmers Guide.

Programming Notes (Avoiding Common Pitfalls)

As YottaDB is likely different from other data management and persistence engines you have used, this section provides information about features of YottaDB intended to help you avoid common pitfalls.

Numeric Considerations

To ensure the accuracy of financial calculations, [10] YottaDB internally stores numbers as, and performs arithmetic using, a scaled packed decimal representation with 18 significant decimal digits, with optimizations for values within a certain subset of its full range. YottaDB efficiently converts between strings and numbers.

[10]For example, since a number such as .01 is not exactly representable as a binary or hexadecimal floating point number adding a list of currency values using floating point arithmetic does not guarantee that the result will be correct to the penny, which is a requirement for financial calculations.

When passed a string that is a canonical number for use as a subscript, YottaDB automatically converts it to a number. This automatic internal conversion is immaterial for applications:

  • that simply store and retrieve data associated with subscripts, potentially testing for the existence of nodes; or
  • whose subscripts are all numeric, and should be collated in numeric order.

This automatic internal conversion is material to applications that use:

  • numeric subscripts and expect the subscripts to be sorted in lexical order rather than numeric order; or
  • mixed numeric and non-numeric subscripts, including subscripts that are not canonical numbers.

Applications that are affected by automatic internal conversion should prefix their subscripts with a character such as “x” which ensures that subscripts are not canonical numbers.

In contexts where a string is coerced to a number (for example ydb_incr_s() / ydb_incr_st()) the coercion rules are the same as the M “+” unary operator.

Canonical Numbers

Conceptually, a canonical number is a string from the Latin character set that represents a decimal number in a standard, concise, form.

  1. Any string of decimal digits, optionally preceded by a minus sign (“-“), the first of which is not “0” (except for the number zero itself), that represents an integer of no more than 18 significant digits.
    • The following are canonical numbers: “-1”, “0”, “3”, “10”, “99999999999999999999”, “999999999999999999990”. Note that the last string has only 18 significant digits even though it is 19 characters long.
    • The following are not canonical numbers: “+1” (starts with “+”), “00” (has an extra leading zero), “999999999999999999999” (19 significant digits), “-0” (the canonical representation of 0 is “0”).
  2. Any string of decimal digits, optionally preceded by a minus sign that includes one decimal point (“.”), the first and last of which are not “0”, that represents a number of no more than 18 significant digits.
    • The following are canonical numbers: “-.1”, “.3”, “.99999999999999999999”.
    • The following are not canonical numbers “+.1” (starts with “+”), “0.3” (first digit is “0”), “.999999999999999999990” (last digit is “0”), “.999999999999999999999” (more than 18 significant digits).
  3. Any of the above two forms followed by “E” (upper case only) followed by a canonical integer in the range -43 to 47 such that the magnitude of the resulting number is between 1E-43 through .1E47.

Zwrite Format

Strings used as subscripts and as values can include unprintable bytes, for example control characters or binary data. YottaDB’s zwrite format is an encoding in printable ASCII of any sequence of bytes. Unlike formats such as Base64, the zwrite format attempts to preserve readability of printable ASCII characters. Note that a zwrite formatted string is always longer than the original string (at the very least, it has enclosing quotes).

Signals

As YottaDB includes a real-time database engine that resides within the address space of a process, applications that use signals must not interfere with database operation.

When the YottaDB database engine initializes on the first call into the API, it initializes signal handling as follows:

  • SIGALRM – YottaDB uses this signal extensively and sets its own signal handler for SIGALRM. Application code should not use SIGALRM, and must never replace YottaDB’s handler. YottaDB provides an API for applications that need timing functionality (see Utility Functions).
  • SIGCHLD (formerly SIGCLD) – Set to SIG_DFL for the default action.
  • SIGTSTP, SIGTTIN, and SIGTTOU – As suspending a real-time database engine at an inopportune moment is undesirable, YottaDB sets a signal handler for these signals that defers process suspension until the engine is in a state where it is safe to suspend.
  • SIGCONT - YottaDB sets a handler that continues a suspended process, and nothing if the process is not suspended.
  • SIGINT – YottaDB sets a handler for Ctrl-C. In call-in or simple API mode, this handler first calls the non-YDB main program’s Ctrl-C handler (if one exists) and if that call returns, exits with a -1 return code. Also, if a call-in is done in this environment and the M code uses either the CENABLE or NOCENABLE device parameters, those parameters are IGNORED. In M mode with a mumps executable, behavior is as documented in the M Programmer’s Guide.
  • SIGUSR1 – As YottaDB uses this signal to asynchronously execute the M code in the $zinterrupt intrinsic special variable, it sets an appropriate handler. If non-M code is currently active when the process receives a SIGUSR1, the handler defers the signal till such time as M code is active. If an application uses no M code whatsoever, and does not intend to, it can change the SIGUSR1 handler after the first call to YottaDB. If an application has, or in the future may have, M code, it is best to leave the YottaDB handler in place.
  • SIGUSR2 – As YottaDB processes other than the servers for client/server operation do not use SIGUSR2, YottaDB sets a SIG_IGN handler. SIGUSR2 is available for applications to use. To do so, set a handler after the first call to YottaDB.
  • SIGQUIT – YottaDB sets a handler to terminate the process without generating a core dump.
  • SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGIOT, SIGSEGV, SIGTERM, and SIGTRAP – These signals are fatal, and the YottaDB handler terminates the process with a core dump. See the discussion about core dumps in the description of ydb_fork_n_core(). Although YottaDB normally cleans up processes’ interaction with databases on exit, these signals can indicate that the process is in a bad state and that its code and data cannot be trusted. The process therefore does not attempt to clean up before exit. After a fatal signal, no YottaDB functions can be called except ydb_exit(). In the event an application must use its own handler for one of these signals, it must either save YottaDB’s handler, and drive it before process termination or call ydb_exit() prior to process exit. [11]
  • YottaDB saves an application’s signal handler during initialization and restores it if ydb_exit() is explicitly called prior to process exit. YottaDB does not reset existing signal handlers for signals it does not handle but calls the saved signal handler if the YottaDB handler returns (and doesn’t exit).
[11]Other YottaDB processes will attempt to automatically clean up after a process terminates abnormally. However, this is not guaranteed. Also, if the abnormally terminating process is the last process accessing a database file, there are no remaining processes to attempt a cleanup. Avoid using these signals to terminate processes unless you know what you are doing.

As database operations such as ydb_set_s() set timers, subsequent system calls can terminate prematurely with an EINTR. Such system calls should be wrapped to restart them when this occurs. An example from the file eintr_wrappers.h demonstrates how YottaDB itself is coded to handle system calls that terminate prematurely with an EINTR:

#define FGETS_FILE(BUF, LEN, FP, RC)                            \
{                                                               \
        do                                                      \
        {                                                       \
                FGETS(BUF, LEN, FP, RC);                        \
        } while (NULL == RC && !feof(FP) && ferror(FP) && EINTR == errno);      \
}

If YottaDB is used within a process with other code that cannot co-exist, or be made to co-exist, with YottaDB, for example, by safely saving and restoring handlers, separate the logic into multiple processes or use a client/server database configuration to place application logic and the database engine in separate processes (see Client/Server Operation).

To reiterate because of its importance: never replace YottaDB’s SIGALRM handler.

Forking

Before a process that performs buffered IO executes fork(), it should execute fflush(). Otherwise, the child process will inherit unflushed buffers from the parent, which the child process will flush when it executes an fflush(). This is a general programming admonition, not specific to YottaDB except to the extent that M code within a parent process may have executed write commands which are still buffered when C code within the same process calls fork().

Threads

Important Notes:

  • Local variables, locks and transaction contexts are held by the process and not by the thread. In other words, these resources are shared by threads in a multi-threaded application, and YottaDB assumes that the threads of an application cooperate to manage the resources, e.g.
    • One thread may set a local variable node, and another thread may delete it.
    • One thread may acquire a lock and another may release it.
    • A global variable update within a transaction by one thread is immediately visible to another thread within the process, but is not visible to other processes until the transaction commits.
  • It is the responsibility of the application to avoid race conditions between threads in their use of resources managed by YottaDB at the level of the process. YottaDB does not ensure the absence of race conditions in accessing these resources because to do so would unduly restrict the freedom of application designers. For example, it is a legitimate design pattern to have one thread that provides one subscript of a node, and a different thread that provides a different subscript.
  • Simple API functions use an *errstr parameter to avoid a race condition and ensure they get the correct $zstatus when function has an error return. If an application calls ydb_get_s() / ydb_get_st() for the value of $zstatus for the complete error text when a YottaDB function returns an error return code, for a single-threaded application, $zstatus has correct and current information, since calls to YottaDB are entirely under the control of that single application thread. For a multi-threaded application, between the time a function returns with an error return code, and a subsequent call to ydb_get_st() to get the value of $zstatus, another thread may call YottaDB, and the $zstatus returned will be from that subsequent call. A *errstr parameter in functions for multi-threaded applications provides the $zstatus for that call to the caller.
    • An application that does not want the $zstatus string can pass a NULL value for *errstr.
    • The string in errstr->buf_addr is always null terminated, which allows *errstr to be passed to standard system functions like printf().
    • In the event a buffer provided by an application is not long enough for a $zstatus, YottaDB truncates the string to be reported, rather than issuing an INVSTRLEN error (since a second error while attempting to report an error is likely to add confusion rather than enlightenment).
      • errstr->len_used is always set to the length of $zstatus, whether or not it is truncated.
      • If errstr->len_used is greater than errstr->len_alloc-1 it means $zstatus has been truncated.
  • A multi-threaded application is permitted to use the YottaDB single-thread functions as long as the application ensures that all YottaDB access is performed only by one thread. A thread may use the ydb_thread_is_main() to determine whether it is the thread that is calling YottaDB. YottaDB strongly recommends against this application design pattern: this functionality only exists to provide backward compatibility to a specific existing application code base.

Even though the YottaDB data management engine is single-threaded and operates in a single thread, [12] it supports both single- and multi-threaded applications. Multi-threaded applications may call multi-threaded Simple API functions – those whose names end in _st() – as well as utility functions – those whose names end in _t(). Single-threaded applications may call the Simple API single-threaded functions – those whose names end in _s() – as well as utility functions – those whose names do not end in _t(). An application must not call both single-threaded and multi-threaded Simple API functions, and any attempt to do so results in a YottaDB error returned to the caller.

[12]Although there is functionality within YottaDB that may invoke multiple threads under the covers (such as asynchronous database IO), these perform certain very limited and specific operations. The YottaDB engine itself is single threaded.

When a single-threaded application calls a YottaDB function, the application code blocks until YottaDB returns, the standard single threaded application behavior for a function call, also known as synchronous calls.

In a multi-threaded application, the YottaDB engine runs in its own thread, which is distinct from any application thread. When a multi-threaded application calls a YottaDB function, the function puts a request on a queue for the YottaDB engine, and blocks awaiting a response – in other words, any call to YottaDB is synchronous as far as the caller is concerned, even if servicing that call results in asynchronous activity within the process. Meanwhile, other application threads continue to run, with the YottaDB engine handling queued requests one at at time. An implication of this architecture is that multi-threaded functions of the Simple API cannot recurse – a call to a multi-threaded function when another is already on the C stack of a thread results in a SIMPLEAPINEST error. While this is conceptually simple for applications that do not use Transaction Processing, transaction processing in a threaded environment requires special consideration (see Threads and Transaction Processing).

Programming in M is single-threaded and single-threaded applications can call into M code, and M code can call single threaded C code as documented in Chapter 11 (Integrating External Routines) of the M Programmers Guide Multi-threaded C applications are able to call M code through the functions ydb_ci_t() and ydb_cip_t() functions as documented there, with the restriction that if M code called through ydb_ci_t() or ydb_cip_t() calls out to C code, that C code is not permitted to start a transaction using ydb_tp_st().

Note that triggers, which are written in M, run in the thread of the YottaDB engine, and are unaffected by multi-threaded Simple API calls already on an application process thread’s stack. However, if a trigger calls C code, and that C code calls ydb_ci_t() or ydb_cip_t(), that C code is not permitted to call ydb_tp_st().

Threads and Transaction Processing

As discussed in Transaction Processing, ydb_tp_s() or ydb_tp_st() are called with a pointer to the function that is called to execute an application’s transaction logic.

In a single-threaded application, the YottaDB engine calls the TP function and blocks until it returns. The function may itself call YottaDB recursively, and the existence of a single thread ensures that any call to YottaDB occurs at the correct transaction nesting level.

In a multi-threaded application, the YottaDB engine invokes the TP function in another thread, but cannot block until it gets the message that the function has terminated with a value to be returned, because the engine must listen for messages from that function, as well as threads it spawns. Furthermore, one of those threads may itself call ydb_tp_st(). Therefore

  • The YottaDB engine must know the transaction nesting level at which it is operating, responding to requests for service at that level, and block any transaction invocations at a higher (enclosing) level until the current transactio is closed (committed or rolled back).
  • After a transaction has closed, any further calls from threads invoking YottaDB for the closed transaction must receive errors.

To accomplish this, the Simple API functions for threaded applications – those ending in _st() – have a tptoken first parameter used as follows to provide the required transaction context of a thread.

  • When an application calls a Simple API function outside a transaction, it provides a value of YDB_NOTTP for tptoken.
  • When an application calls ydb_tp_st(), it generates provides a tptoken as the first parameter when it calls the function that implements the logic for the transaction. Any threads that this function spawns must provide this tptoken to YottaDB. Passing in a different or incorrect tptoken can result in hard-to-debug application behavior, including deadlocks.
  • When a Simple API function is called:
    • If tptoken is that of the current transaction, the request is processed.
    • If tptoken is that of a higher level transaction within which the current transaction is nested, the call blocks until the nested transaction completes (or nested transactions complete, since there may be multiple nesting levels).
    • If tptoken does not correspond to a higher level transaction (e.g., if it corresponds to a closed transaction or a nonexistent one), YottaDB returns an error.

Note: if the function implementing a transaction spawns threads (or coroutines executing in threads), those threads/coroutines must:

  • terminate before the function returns to YottaDB;
  • use a current tptoken when invoking YottaDB (in effect, switching transaction contexts ­ technically this violates ACID transaction properties but perhaps reasonable in a few restricted cases, such as creating background worker threads); or
  • not invoke YottaDB.

Should a thread/coroutine spawned in a function implementing transaction logic invoke YottaDB after the function has returned, the thread/coroutine will get an invalid token error message unless it uses a current tptoken.

Note: Sharing or passing tptoken values between threads/coroutines can lead to deadlocks and other hard-to-debug situations. YottaDB strongly recommends against such usage. If you have a legitimate use case, design it so that you can debug it when the inevitable error condition occurs.

Timers and Timeouts

Although the Simple API uses nanosecond resolution to specify all time intervals, in practice underlying functions may have more granular resolutions (microseconds or milliseconds). Furthermore, even with a microsecond or millisecond resolution, the accuracy is always determined by the underlying hardware and operating system, as well as factors such as system load.

Memory Allocation

Memory allocated by ydb_malloc() must be explicitly freed by ydb_free(). ydb_exit() does not free memory, and any memory allocated but not freed prior to ydb_exit() is released only on process exit.

Syslog

Issues that pertain to the application and on which application code can take reasonable action are reported to the application (YDB_ERR_GVUNDEF being an example) and issues that pertain to operations and which application code cannot take reasonable action but which operations staff can (like running low on filesystem space, which are not discussed here, as this is a Programmers Guide) are reported to the syslog. In the event that a syslog does not exist (e.g., in default Docker containers), a process’ syslog messages go to its stderr.

YottaDB uses the existence of /dev/log as an indicator of the existence of a syslog.

IO

Although YottaDB does not prohibit it, we recommend against performing IO to the same device from M and non-M code in a process unless you know exactly what you are doing and have the expertise to debug unexpected behavior. Owing to differences in buffering, and in the case of interactive sessions, setting terminal characteristics, performing IO to the same device from both M and non-M code will likely result in hard to troubleshoot race conditions and other behavior.