This is the canonical GitHub-compatible Markdown README. The legacy README file is kept as a compatibility pointer for tooling and distribution paths that still expect that filename.
For an applied explanation of the cryptographic and data-security concepts used by CaumeDSE, see TUTORIAL.md.
For tested curl examples based on the live verifier fixtures, see
API_EXAMPLES.md.
For a machine-readable route reference for the stable live-verifier-covered API surface, see openapi.yaml.
For safe AI-agent and automation patterns, cookbook recipes, and an operational checklist around the API, see AI_USAGE.md.
For a guarded Python AI-agent workflow sample, see samples/ai-agent/.
For a delegated scoped-token broker sample that keeps CaumeDSE organization keys out of agent prompts, see samples/delegated-token-broker/.
For an allowlisted, redacting retrieval connector that prepares bounded CSV snippets for AI context, see samples/agent-rag-connector/.
For a human-in-the-loop document and generated-parser review workspace, see samples/review-workspace/.
For a redacted structured-audit and live-verifier report generator, see samples/audit-dashboard/.
For the supported read-only MCP stdio tool surface over safe REST reads, see samples/mcp-server/.
AI agents and MCP clients can discover safe automation capabilities with
GET /agentCapabilities, which returns public JSON metadata without requiring
or exposing CaumeDSE credentials.
API responses include an X-Request-Id header. When outputType=json is
requested and a non-HEAD request fails with common authentication,
authorization, not-found, method, conflict, not-implemented, or server errors,
CaumeDSE returns a JSON envelope:
{
"error": {
"code": "authentication_required",
"message": "Authentication is required.",
"httpStatus": 401,
"requestId": "cdse-...",
"safeForAgent": true
}
}Use requestId to correlate client-side failures with LogsDB response-header
entries and retained verifier artifacts.
- Purpose and Philosophy
- Status
- Cryptography and Data Security Tutorial
- API Examples
- OpenAPI Route Reference
- AI-Safe API Usage, Cookbook, And Checklist
- AI Agent Capability Manifest
- AI Agent Sample
- Delegated Token Broker Sample
- Agent RAG Connector Sample
- Secure Document Review Workspace
- Compliance Audit Dashboard
- MCP Read-Only Server
- License
- Architecture and Functionality
- REST Resource API Reference
- Collaborating
- Security Considerations
The idea behind CaumeDSE is to provide a free software solution that allowed the creation of reasonably secure and isolated workspaces to process and store sensitive data, within uncontrolled environments.
With concepts such as the cloud and bring your own device as well as an increasing demand for mobile devices, it has become clear that the way in which companies operate is changing dramatically, and with it, the way in which we secure information and the infrastructure that supports business processes.
CaumeDSE has been designed as a service platform that provides security to data by using free, well known and robust cryptographic software, as well as open an simple data structures and interfaces to make portability and extensibility easy.
CaumeDSE is not an end-user solution. It is a service platform for supporting front end applications.
CaumeDSE is designed with solid security principles, that take into account new models of doing business and emerging information technologies. It also takes into consideration the increasing number of data privacy regulations.
Trust is essential for security. In fact, it is one of the main reasons why we put security in place on computer systems: to be able to trust them. We aim to provide trust by following these principles:
-
Transparency. With free, open source software, you can check for yourself what this software does, and know that others can and will do it as well to help improving it.
-
Portability. With open standards and data structures, you avoid vendor lock-ins with closed programs/technologies. Moreover, by being free and open source, this platform can be ported to many environments and modified to suit your own needs. You don't depend on a company being there to continue its development.
-
Simplicity. We favor simplicity for doing all tasks as much as possible. With the adoption of REST, we expect that interaction and development with this platform will be easy and straightforward.
-
Secure by design. We developed this tool with security as a primary goal. That is not to say it is perfect, but its development process has been structured to ensure continued improvement in this area, independently of other important requirements such as performance and interoperability
-
Need to know and use. We restrict as much as possible the access to information. One way that we do it is by not storing encryption keys within the software. This means that each organization (in the scope of this software) is responsible of managing keys securely, but it also allows them to have full control of who access what.
-
Traceability. Logging securely all relevant actions provides good control on who did what, when, as well as evidence in case of an incident.
-
Thoroughness. We aim to do extensive debugging and security assessments before any major release. This won't eliminate all problems, but should limit their number and provide a more stable platform. We favor stability and simplicity over new functionality, since the later can be developed easily on top of this platform by other software.
There are some important aspects regarding trust. Achieving trust is difficult, particularly since we human beings posses a natural bias towards risk assessment (e.g. we tend to think that the closer to us things are, the more secure they are).
No matter how sensitive a particular piece of data is or how much money you are willing to invest to protect it, in the end, for it to be useful, you will need to trust someone and/or something.
There are many arguments in favor of hardware based protections, and it is true that such devices are usually more resistant to certain types of attacks than software, but you still end trusting the manufacturer of such products as much as you need to trust the developer of security software. Also, specialized hardware may offer better performance than software for some tasks, such as higher resistance against reverse engineering and internal processing monitoring.
Like there are disadvantages to a software based approach, there are also some advantages to it. Transparency is one of them.
Even if your won't be checking the source code yourself, you know that it is available for many others to take a look at it (so the chances of finding problems and fixing them may be higher than that of closed hardware solutions).
Another advantage is portability. Think about some possible scenarios you may face with new operation models such as the cloud. An infrastructure provider in the cloud may not be offering hardware based security solutions, or you may not be able to install in their premises your preferred hardware based solution. But still, if you have some control at some layer of the environment where data will be stored, transferred or processed, you can setup your "rules of engagement" there and create a virtual perimeter that may offer a reasonable level of protection for your data.
In terms of compliance, remember that when you contract external services and personnel you may share with them some responsibility for protecting sensitive data and operations, but you cannot share accountability; that will remain on your side.
Right now, the software is in a stable release:
-
Basic functionality is complete and tested but some features are still pending.
-
Sample applications added (v1.0.9): a new samples/hsm-db-crypto/ directory provides four independent client implementations that demonstrate the CDSE API as a hybrid HSM / encrypted database / general cryptographic command interface:
- Web (HTML5 + JavaScript SPA with a local CORS proxy)
- Python 3 CLI (requires: requests)
- Go CLI (standard library only, no external dependencies)
- Perl CLI (requires: LWP::UserAgent, HTTP::Request::Common) All CLI variants support interactive sessions (credentials entered once) and one-shot parameter mode (credentials per call, or via environment variables CDSE_SERVER/CDSE_USER_ID/CDSE_ORG_ID/ CDSE_ORG_KEY/CDSE_STORAGE). Operations covered: encrypted secret storage, encrypted CSV databases (insert/query/update/delete), user/org info, and transaction audit log.
-
Constant-time comparisons (v1.0.8): all post-decryption string and MAC comparisons that could leak information via timing (strcmp/strncmp on decrypted values and HMAC results) replaced with new cmeStrSafeEq() and cmeMemSafeEq() functions. Closes GitHub issue #5.
-
Concurrent multi-user support added (v1.0.7): libmicrohttpd now runs a configurable worker-thread pool (default: 4 threads, adjustable at build time with --with-max-threads=N). Shared mutable state is protected:
- SQL result tables (cmeResultMemTable) use thread-local storage (__thread) so each worker thread has its own copy.
- The embedded Perl interpreter used by legacy/debug helper paths is serialised with a POSIX mutex (cmePerlMutex); parserScripts requests run Perl parser code in child processes.
- The engine power-status flag is serialised with a second mutex (cmePowerMutex).
- Per-connection start-time and upload-size counters moved from static locals to the per-connection info struct. A new testThreadSafety() test verifies correct parallel SQLite operation across cmeDefaultMaxThreads threads.
-
gen_test_certs.shupdated to support 7 algorithm presets: rsa2048, rsa4096, ecp256, ecp384, ecp521, ed25519, ed448. Each preset automatically pairs the correct hash algorithm and sets the right X.509 keyUsage extensions (keyEncipherment for RSA; digitalSignature only for EC/EdDSA). Default preset: ecp384. -
TLS digital certificates regenerated with ECDSA P-384 / SHA-384 (replacing RSA 4096-bit / SHA-256 from v1.0.5), negotiating TLS 1.3 with TLS_AES_256_GCM_SHA384. Handshake is ~2.4x faster than RSA 4096 on the same hardware (30ms vs 71ms). Server cert includes a Subject Alternative Name (SAN) for localhost.
-
Column-level HMAC protection is now implemented for "MAC" (HMAC of plaintext before encryption) and "MACProtected" (HMAC of ciphertext after encryption) column attributes. These allow data-integrity verification independent of the encryption layer.
-
AES-256-GCMis the default symmetric encryption algorithm as of version 1.0.3. All internal databases (ResourcesDB, RolesDB, LogsDB) and encrypted data are created with authenticated encryption. The full test suite passes with 0 decrypt errors and 0 runtime errors against fresh GCM databases. -
Default HTTP and HTTPS ports are 8080 and 8443 respectively, allowing the engine to run without root privileges on standard Linux systems.
-
Basic documentation is available.
If you would like to collaborate Please take a look at section VI (Collaborating).
Caume Data Security Engine (also called CaumeDSE) is released under the GNU General Public License version 3 by the Copyright holder, with the additional exemption that compiling, linking, and/or using OpenSSL is allowed.
The software is Copyright 2010-2026 by Omar Alejandro Herrera Reyna.
Check licensing and copyright details for CaumeDSE and other included software in the file called COPYING and in the headers of the source code available for distribution.
The architecture of CaumeDSE is composed of several layers:
-
API - Web based, RESTful API that handles requests to resources using standard HTTP methods (GET, POST, PUT, DELETE, HEAD and OPTIONS). The requests can be received by HTTP (TCP port 8080; only in DEBUG mode) or by HTTPS (TCP port 8443). Libmicrohttpd is used as an internal, standalone web server to handle the requests, while TLS security is handled by GnuTLS. Standard query parameters and multipart/form-data encoded parameters (for POST method) are supported in resource requests.
-
Authentication - Authentication of users and applications is handled at the web server level. Right now, client authentication with digital certificates with TLS is supported. OAuth authentication is not performed inside CaumeDSE. OAuth deployments must place an external engine manager in front of the engine; that manager validates OAuth credentials, creates and later removes delegated CaumeDSE organization/user/role/resource scopes, and forwards requests to the engine with the delegated
userId,orgId, andorgKey. -
Authorization - Authorization is handled internally by CaumeDSE with role tables for each user. Each role table maps each available resource type with available HTTP methods (GET, POST, PUT,...). All roles are encrypted with the organization's key (see resource hierarchy below).
-
Resource access - CaumeDSE maintains internal index databases that map data resources to their location. All resources and index database registers are encrypted with the organization's key. Supported data resources include CSV files, raw-compatible file document types and Perl scripts. File resources in addition are split in several parts (CSV files are split in columns and each column in different parts of a specific size).
-
Resource protection - The resulting encrypted files are named with pseudo random generated hexadecimal strings, and stored in their corresponding storage directory (the relation between the original files and each encrypted part is maintained in the internal index database). No encryption key is ever stored in an internal database, and values in internal databases are salted before being encrypted with one salt per register, and a second salt per value (salts per register are not encrypted). Since encryption keys are rather handled as passwords, internally these keys are processed using PBKDF2 (PKCS5v2.0 with HMAC-SHA256 + a counter greater than 1) to generate a temporary key and its corresponding initialization vector (iv) that are the ones actually being used by the encryption/decryption algorithms. This KEY/IV generation mechanism is not compatible anymore with the OpenSSL's command line tool (You will need to generate manually the keys and iv with this method before you can decrypt manually each file and database register with this tool). Resource indexes are shuffled when new elements are added (for CSV files, columns are reconstructed maintaining the order of their registers, but the order in which columns appear changes. Scripts should therefore access columns by name and not by their position).
ResourcesDB document indexes also keep deterministic protected lookup values for documentId, storageId and orgResourceId. These lookup values are keyed HMACs with field-specific domain separation, and are used only to narrow exact-match searches on these high-cardinality identifiers. They do not replace the encrypted and MAC-protected register values: matching registers are still decrypted and verified before being returned or deleted. Lookup values are intentionally not generated for low-cardinality or arbitrary document attributes, preserving the protection provided by encrypted values and column shuffling. Older databases are migrated with nullable lookup columns, and registers without lookup values remain readable through the normal protected-value verification path. -
Resource processing - All data processing including encryption and decryption takes place in memory (with a few exceptions, such as file uploads that are stored in a specific directory before being encrypted, overwritten and deleted). Internal secure databases (data tables organized in rows and columns like CSV files) can be processed by Perl scripts in memory, using and embedded Perl interpreter.
-
Sessions and multi-threading - CaumeDSE uses stateless REST connections. As of v1.0.7 a configurable number of simultaneous users is supported through a libmicrohttpd worker-thread pool (default: 4 threads; set at build time with --with-max-threads=N). Each thread handles one connection at a time; concurrent connections are distributed across the pool. Shared state (Perl interpreter, engine power flag) is protected by POSIX mutexes; SQL result tables use thread-local storage so parallel requests do not interfere with each other's results. The maximum number of queued connections is 4x the thread-pool size.
Resources are organized hierarchically using a REST approach. Reading the uniform resource identifier (URI) within an HTTP request from left to right will show resources and resource types that USE or CONTAIN the resource or resource type to the right of it within the hierarchy, until you find the requested resource at the end (before any parameters).
The resource hierarchy is listed below:
https://{engine}
|-- /organizations
| `-- /{organization}
| |-- /users
| | `-- /{user}
| | |-- /roleTables
| | | `-- /{roleTable}
| | |-- /filterWhitelist
| | | `-- /{filterUser}
| | `-- /filterBlacklist
| | `-- /{filterUser}
| `-- /storage
| `-- /{storage}
| |-- /documentTypes
| | `-- /{documentType}
| | `-- /documents
| | `-- /{document}
| | |-- /parserScripts
| | | `-- /{parserScript}
| | |-- /content
| | |-- /contentRows
| | | `-- /{contentRow}
| | `-- /contentColumns
| | `-- /{contentColumn}
| `-- /dbNames
| `-- /{dbName}
| `-- /dbTables
| `-- /{dbTable}
| |-- /tableRows
| | `-- /{tableRow}
| `-- /tableColumns
| `-- /{tableColumn}
|-- /engineCommands
|-- /transactions
`-- /favicon.ico
[not implemented] marks resources that might be implemented in the future.
Resources (listed within keys, { and }), must be called by their unique name. Resource types are not listed within keys and they must be named exactly as they are listed.
For example, to get the resource representation of user EngineAdmin that is registered in organization EngineOrg at the CaumeDSE instance in ip address 192.168.1.1, you would do an HTTPS GET request with an URI similar to this:
https://192.168.1.1/organizations/EngineOrg/users/EngineAdmin?userId=EngineAdmin& orgId=EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
Check section V for access methods, parameters and examples for each resource/ resourceType requests and their expected results.
Common parameters to a resource request are passed either via a query string or as a multipart/form-data encoded body (in the case of POST requests).
There are 3 types of parameters
- Authorizations & Encryption parameters
- Resource attribute parameters (to match or update)
- Optional parameters
These parameters are required in every request to decrypt resource indexes roles and resources, and allows the system to verify if the request is authorized (note that authentication takes place before any authorization and decryption).
userId
Specifies the {user} to verify authorization (note that this same
user is authenticated previously). The user referred to by userId
must be a resource of the organization defined within orgId.
orgId
Specifies the {organization} to verify authorization and perform
resource operations. This parameter does no need to match the
organization defined in the URI. For example, a user with the right
privileges can create a new organization (defined in the URI) with a
POST request, but still authenticate and be authorized using the
userId and orgId parameters.
orgKey
Specifies the organization key that is used to decrypt resources and
internal database values. Every resource registered within the same
organization are encrypted with the same key (although with
different salt and initialization vector). This includes users, and
is the reason why authentication is a separate process.
These parameters, allow you to specify matching registers (exact matches only), if prepended with an underscore (_) and combined with methods such as GET, HEAD, PUT or DELETE, or to update resource values in POST and PUT methods if prepended with an asterisk (*).
For example, a request using a PUT method to update all user resources (resource type users) whose resourceInfo value is 'new', by setting their certificate value to 'undefined' you would use a PUT request that would look similar to this one:
https://192.168.1.1/organizations/EngineOrg/users?userId=EngineAdmin& orgId=EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC&_resourceInfo=new& *certificate=undefined
Attribute parameters could be considered resources contained by the corresponding resource or resource type where they are being used. As such, they may be included in the resource hierarchy in the future to provide a consistent and alternative form to be accessed. Unfortunately, managing every attribute as a resource also means more requests to perform a simple function (e.g. Suppose you have a resource with 5 attributes, you can create the resource and update all 5 attributes in a single request if you use attribute parameters, but if you manage attributes as resources contained by the main resource, it would take 6 requests to create the resource and each of its attributes)
To check which attribute parameters are supported by each resource and resource type refer to section V (REST (Resource) API reference).
Resources/ResourceTypes: ANY
Prefixes: match (_) only.For POST/PUT updates the value is always
taken from the URI
When used with _ matches against the userId attribute of a resource.
This attribute contains the last user to modify (e.g. with PUT) the
resource, or the user who created the resource (with POST) if it has
not been modified.
For other uses check: 3.1 Authorizations & Encryption parameters
Resources/ResourceTypes: ANY
Prefixes: match (_) only.For POST/PUT updates the value is always
taken from the URI
When used with _ matches against the orgId attribute of a resource.
This attribute contains the orgId corresponding to the last user to
modify (e.g. with PUT) the resource or to the user who created the
resource (with POST) if it has not been modified.
For other uses check: 3.1 Authorizations & Encryption parameters
Resources/ResourceTypes: organizations, {organization}, users,
{user}, storage {storage}, documents, {document}
Prefixes: match (_) and update (*).
Contains general information regarding the resource.
Resources/ResourceTypes: organizations, {organization}, users,
{user}
Prefixes: match (_) and update (*).
Stores digital certificate information related to the resource. Note
that CaumeDSE does not use this information in any way for
authentication. Organizations can store any string here (PEM
certificates, certificate issuer, certificate hash,...).
Resources/ResourceTypes: organizations, {organization}, users, {user}
Prefixes: match (_) and update (*).
Stores public key information related to the resource. Note that
CaumeDSE does not use this information in any way for
authentication. Organizations can store any string here (PEM public
keys, public key file names, public key hash,...).
Resources/ResourceTypes: users
Prefixes: match (_) only. For POST ({user}), this attribute is taken
from the URI.
Stores the identifier of a {user} resource.
Resources/ResourceTypes: organizations, transactions
Prefixes: match (_) only. For POST ({organization}) this attribute
is taken from the URI.
Stores the identifier of an {organization} resource. For
transactions, if the user was correctly authenticated, then this
value will be protected (indicated by the 'authenticated'
parameter). For ResourcesDB document registers, exact-match
queries may use the protected orgResourceIdLookup index as a
candidate filter before normal protected-value verification.
Resources/ResourceTypes: users, {user}
Prefixes: match (_) and update (*).
*FUNCTIONALITY NOT YET IMPLEMENTED*: This attribute may contain
password hashes for user authentication via the HTTP basic
authentication method in the future.
Resources/ResourceTypes: users, {user}
Prefixes: match (_) and update (*).
OAuth is delegated to an external engine manager. This attribute is
available for manager-owned metadata but is not used by CaumeDSE for
internal OAuth token validation.
Resources/ResourceTypes: users, {user}
Prefixes: match (_) and update (*).
OAuth is delegated to an external engine manager. This attribute is
available for manager-owned metadata but is not used by CaumeDSE for
internal OAuth token validation.
Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the name (hexadecimal string generated randomly) that
contains the encrypted content for the corresponding partId and
columnId.
Resources/ResourceTypes: documents
Prefixes: match (_) only. For POST ({document}) this attribute is
taken from the URI.
Stores the identifier of a {document} resource.
For ResourcesDB document registers, exact-match queries may use the
protected documentIdLookup index as a candidate filter before normal
protected-value verification.
Resources/ResourceTypes: storage
Prefixes: match (_) only. For POST ({storage}) this attribute is
taken from the URI.
Stores the identifier of a {storage} resource.
For ResourcesDB document registers, exact-match queries may use the
protected storageIdLookup index as a candidate filter before normal
protected-value verification.
Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the hash of the encrypted content in the corresponding
columnFile.
Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the number of encrypted parts in which the original file was
divided. For CSV files, this refers to the number of parts in which
every column was divided if you want to know the number of files
that contain pieces of a CSV files, just do (number of
columns)*totalParts. The highest number in columnId is the number
of columns that a CSV resource has.
partId Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the sequential id for the part for the corresponding columnId.
Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the Unix time (epoch) corresponding to the last time that the
resource was updated (PUT), or to the time it was created (POST) if
it has not been updated yet.
Resources/ResourceTypes: documents, {document}
Prefixes: match (_) only. For POST/PUT this attribute is calculated
by CaumeDSE.
Stores the column id corresponding to the column File. for document
types different from CSV files (e.g. raw files), this value is
always 1 (i.e. they consist of a single column).
Resources/ResourceTypes: storage, {storage}
Prefixes: match (_) and update (*).
Stores information related to the location of the {storage}
resource. You can store any text string here, it is not used by
CaumeDSE.
Resources/ResourceTypes: storage, {storage}
Prefixes: match (_) and update (*).
*FUNCTIONALITY NOT YET IMPLEMENTED*: This attribute may specify the
type of storage to allow CaumeDSE to connect and use remote storage
in the future. Right now, only regular file storage available
through standard input and output functions is supported.
Resources/ResourceTypes: storage, {storage}
Prefixes: match (_) and update (*).
*FUNCTIONALITY NOT YET IMPLEMENTED*: This attribute may specify an
access identifier to allow CaumeDSE to connect and use remote
storage in the future. Right now, only regular file storage
available through standard input and output functions is supported.
Resources/ResourceTypes: storage, {storage}
Prefixes: match (_) and update (*).
*FUNCTIONALITY NOT YET IMPLEMENTED*: This attribute may specify the
user identifier to allow CaumeDSE to connect and use remote storage
in the future. Right now, only regular file storage available
through standard input and output functions is supported.
Resources/ResourceTypes: storage, {storage}
Prefixes: match (_) and update (*).
*FUNCTIONALITY NOT YET IMPLEMENTED*: This attribute may specify
access credentials to allow CaumeDSE to connect and use remote
storage in the future. Right now, only regular file storage
available through standard input and output functions is supported.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform GET requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform POST requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform PUT requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform DELETE requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform HEAD requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: {roleTable}
Prefixes: match (_) and update (*).
NOTE: with the match prefix you would have a 2 underscores at in the
parameter name.
This attribute specifies whether the corresponding {user} at the
specified {organization} is allowed to perform OPTIONS requests on
resources related to {roleTable}.
String "1" is for allow. string "0" is for deny.
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the HTTP method used by the user request.
If the user was correctly authenticated, then this value will be
protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the URL used as part of the user request.
If the user was correctly authenticated, then this value will be
protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the HTTP Headers used as part of the user
request. If the user was correctly authenticated, then this value
will be protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the timestamp of the moment at which the
user request was received. If the user was correctly authenticated,
then this value will be protected (indicated by the 'authenticated'
parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the timestamp of the moment at which the
user request was answered. If the user was correctly authenticated,
then this value will be protected (indicated by the 'authenticated'
parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the size of the body of the HTTP user's
request. If the user was correctly authenticated, then this value
will be protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the size of the body of the HTTP engine's
answer. If the user was correctly authenticated, then this value
will be protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the IP address (v4 or v6) of the user
sending the request. If the user was correctly authenticated, then
this value will be protected (indicated by the 'authenticated'
parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the response code of the HTTP engine's
answer. If the user was correctly authenticated, then this value
will be protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies the response headers of the HTTP engine's
answer. If the user was correctly authenticated, then this value
will be protected (indicated by the 'authenticated' parameter).
Resources/ResourceTypes: transactions
Prefixes: match (_) only. For POST this attribute will be defined
automatically by CaumeDSE
This attribute specifies whether the user was authenticated ('1') or
not ('0'). If the user was correctly authenticated, then values
depending on the authenticated attribute will be encrypted;
otherwise they will be copied in cleartext. This attribute is never
encrypted.
Note that there is a situation when an authenticated user uses an
incorrect orgKey. In this case, transaction logs won't be readable
since they will be protected using the erroneous key. Transactions
will be protected as whenever the user is authenticated, even if
he/she does not have the right priviledges to execute the request.
Resources/ResourceTypes: engineCommands
Prefixes: match (_) only. This parameter activates (value = on,
default) or deactivates (value = off) engine requests with the PUT
method.
salt Specifies the salt to be used with all values of an internal database register or a data resource for encryption/decryption. if not included, it will be generated by a pseudo random algorithm by CaumeDSE. If included it must be an hexadecimal string 16 characters long (i.e. representing 8 bytes).
(Note that for values within the same register another salt is
generated and prepended to values before being encrypted (internally
this is called valueSalt), and removed after decryption, this second
salt is necessary to avoid same ciphertext for identical database
values within the same internal database register.)
newOrgKey In POST requests only (i.e 'create'), this parameter specifies the organization key (orgKey) to be used for creating the new resource, which is different from the orgKey used to protect the the userId's roles of the requesting user (of course, the user must have POST permission within its organization that can be decrypted with orgKey).
This allows for example, an administrator to create a different
organization or resources within a different organization. After
being created though, this administrator would need a user with the
right credentials to be authenticated and the corresponding
permissions within the role table for this new organization in order
to access the newly created resources.
If omitted, orgKey will be used to protect any new resource.
outputType
Specifies the type of output for the result. Available values are
csv, json and HTML (the latter is the default). This is
particularly useful for results that return data tables, such as
resource specification queries, content rows/columns, parser script
output, secure DB browsing, and requests for the contents of csv
type files. JSON table responses use this shape:
{"columns":[...],"rows":[{"column":"value"}],"pagination":{"offset":0,"limit":100,"returnedRows":N,"totalRows":N,"hasMore":false}}.
For outputType=json, optional limit and offset parameters bound
returned rows. limit defaults to 100 and must be 1..1000; offset
defaults to 0 and must be non-negative.
schema
For agent-safe CSV inspection, read schema metadata before reading rows,
columns, or parser output:
/organizations/{org}/storage/{storage}/documentTypes/file.csv/documents/{document}/schema
and
/organizations/{org}/storage/{storage}/dbNames/{db}/dbTables/{table}/schema.
These JSON endpoints return column names/types, row counts, pagination
limits, document/table identifiers, and parser policy metadata.
These parameters are required in every POST request of {document} resources.
file Specifies the file name and contents (which does not necessarily will be the same as the documentId of the resource in the URI) of the file to be uploaded, to create a {document} resource. POST requests will contain the attribute parameters and the file contents encoded in multipart/form-data format within the body.
shuffle Optional. Enables row-order shuffling metadata for secure CSV storage. If omitted, the default encryption algorithm is used. Values 0, false, no, off and none disable this attribute; values 1, true, yes and on use the default encryption algorithm; any other value is stored as the attribute algorithm.
protect Optional. Enables value protection metadata for secure CSV storage. If omitted, the default encryption algorithm is used. Values 0, false, no, off and none disable this attribute; values 1, true, yes and on use the default encryption algorithm; any other value is stored as the attribute algorithm.
replaceDB Optional for file.csv uploads. If set to 1, true, yes or on, an existing secure CSV database with the same documentId is replaced. If omitted or set to 0, false, no, off or none, duplicate document resources are rejected.
vacuumDB Optional for file.csv uploads and content row/column writes. If set to 1, true, yes or on, each temporary memory database is vacuumed before it is saved as a durable secure database file. Secure imports with protection attributes still vacuum by default to remove slack space that may contain unprotected data.
Specifies the value for the specified 'column_name'. The column name
must match an existing column name within a document resource of
type file.csv.
GET /agentCapabilities returns a public JSON manifest for AI agents, MCP
clients, SDK generators, and automation supervisors. It does not require
userId, orgId, or orgKey, and it does not include organization data or
secrets.
The manifest reports:
- required authentication parameters for data routes;
- preferred and supported response formats;
- JSON pagination defaults and schema discovery routes;
- parser policy settings such as timeout, result limits, isolation profile, and reviewed/profile enforcement;
- core route names, path templates, supported methods, and whether authentication is required;
- links to
README.md,AI_USAGE.md,API_EXAMPLES.md,openapi.yaml,samples/ai-agent/, andsamples/mcp-server/.
Use this endpoint before invoking data routes so an agent can choose JSON responses, avoid passing organization keys to an LLM, and respect parser-review policy before uploading or executing scripts.
GET /metrics returns public-safe in-process runtime metrics without requiring
userId, orgId, or orgKey. The endpoint intentionally avoids organization,
user, document, URL, certificate-path, and query-value labels. JSON is the
default response format; Prometheus text is available with
outputType=prometheus.
The metrics include request totals by fixed method and status class,
authentication failures, authorization denials, parser policy denials, parser
timeouts, parser limit rejections, placeholder counters for DB/crypto failure
classes, service uptime, and configured runtime/parser limits. Treat these
counters as process-local: they reset when the CaumeDSE process restarts.
Use LogsDB and CaumeDSE AuditJSON records when per-request correlation or
longer retention is required.
Note that with the POST method you must define values for every updateable parameter, to ensure that every attribute is initialized by the user. With the PUT method you can update just the values that need changes.
While most examples of POST requests include data in the URI, be aware that the standard way is to encode both parameters (and any file) using multipart/form-data format. Any POST request can be encoded in this way (CaumeDSE supports both encoding formats: URI appended parameters and multipart/form-data parameters). However, you must use multipart/form-data encoding if you are uploading a file (See {document} resource examples below).
Supported HTTP methods: GET PUT OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
<NONE>
UPDATE:
*setEnginePower
RESPONSE HEADERS:
Engine-results: <engine active status>
RESPONSE BODY:
<Attribute table for matching resources>
Example 1) Deactivate engine
METHOD:
PUT
URI:
https://localhost/engineCommands?userId=EngineAdmin&
orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&setEnginePower=off
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _certificate _publicKey
_orgResourceId
UPDATE:
*resourceInfo *certificate *publicKey
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resources>
Example 1) List all organizations with publicKey = 'undefined'
in a CSV style list
METHOD:
GET
URI:
https://localhost/organizations?userId=EngineAdmin&
orgId=EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
&_publicKey=undefined&outputType=csv
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _certificate _publicKey
UPDATE:
*resourceInfo *certificate *publicKey
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Example 1) Create a new organization called BusinessOrg, using
a new organization key: 3132333440414243, with
account EngineAdmin from EngineOrg (EngineAdmin is
assumed to have POST privileges for organizations)
METHOD:
POST
URI:
https://192.168.0.1/organizations/BusinessOrg?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*resourceInfo=
new%20organization&*certificate=undefined&*publicKey=
undefined&newOrgKey=3132333440414243
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) List attributes for organization EngineOrg
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 3) Set attribute publicKey to 'serial:10F46D308B39' for
organization EngineOrg
METHOD:
PUT
URI:
https://localhost/organizations/EngineOrg?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*publicKey=
serial%3A10F46D308B39
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _certificate _publicKey
_userResourceId _basicAuthPwdHash _oauthConsumerKey
_oauthConsumerSecret
UPDATE:
*resourceInfo *certificate *publicKey
*basicAuthPwdHash *oauthConsumerKey
*oauthConsumerSecret
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resources>
Example 1) List attributes of all users of organization
EngineOrg
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/users?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Set attribute publicKey to 'TBD' for all users in
organization BusinessOrg
METHOD:
PUT
URI:
https://localhost/organizations/BusinessOrg?userId=
BusinessAdmin&orgId=BusinessOrg&orgKey=3132333440414243
&*publicKey=TBD
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
{user} Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _certificate _publicKey
_basicAuthPwdHash _oauthConsumerKey
_oauthConsumerSecret
UPDATE:
*resourceInfo *certificate *publicKey *basicAuthPwdHash
*oauthConsumerKey *oauthConsumerSecret
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Example 1) Check if user1 exists within organization EngineOrg
METHOD:
HEAD
URI:
https://localhost/organizations/EngineOrg/users/user1?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Create user BusinessAdmin in organization
BusinessOrg, using BusinessOrg's (new) organization
key: 3132333440414243, with account EngineAdmin from
EngineOrg (EngineAdmin is assumed to have POST
privileges for users)
METHOD:
POST
URI:
https://192.168.0.1/organizations/BusinessOrg/
BusinessAdmin?userId=EngineAdmin&orgId=EngineOrg&
orgKey=0CDBB9AF76AF43BDB72E095989E612CC&*resourceInfo=
Administrator&*certificate=undefined&*publicKey=
undefined&*basicAuthPwdHash=undefined&
*oauthConsumerKey=undefined&*oauthConsumerSecret=
undefined&newOrgKey=3132333440414243
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId __get __post __put __delete __head
__options
UPDATE:
*_get *_post *_put *_delete *_head *_options
TABLE NAMES:
documents users roleTables parserScripts content
contentRows contentColumns dbNames dbTables tableRows
tableColumns organizations storage documentTypes
engineCommands transactions meta filterWhitelist
filterBlacklist
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Example 1) Get EngineAdmin permission table for resource table
users ( i.e. for resources/resource type: users and
{user}), if any.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/users/
EngineAdmin/roleTables/users?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Allow user BusinessAdmin in organization
BusinessOrg, using BusinessOrg's (new) organization
key: 3132333440414243, with account EngineAdmin from
EngineOrg (EngineAdmin is assumed to have POST
privileges for users), to perform GET, HEAD, OPTIONS
AND PUT requests by creating new permissions in the
the users roleTable (i.e. for resources/resource
type: users and {user}).
METHOD:
POST
URI:
https://192.168.0.1/organizations/BusinessOrg/
BusinessAdmin/roleTables/users?userId=EngineAdmin&
orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*_get=1&*_post=0&
*_put=1&*_delete=0&*_head=1&*_options=1&newOrgKey=
3132333440414243
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId __get __post __put __delete __head
__options
UPDATE:
*_get *_post *_put *_delete *_head *_options
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Filter whitelist records are stored in the encrypted ResourcesDB
filterWhitelist table. The `{filterUser}` URL element maps to
userResourceId, the parent `{organization}` maps to orgResourceId,
and the method columns (`_get`, `_post`, `_put`, `_delete`, `_head`,
`_options`) define which methods are allowlisted for that target.
`userResourceId` and `orgResourceId` are evaluated as POSIX extended
regular expressions and must match the full requested user and
organization resource values.
Role-table authorization is still evaluated first. When whitelist
records exist for a method, user-resource requests for that method
must also match a whitelist record for the requested organization
and user.
Example 1) Allow EngineAdmin to perform GET, HEAD and OPTIONS
requests against RoleTableTestUser user resources.
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/users/
RoleTableTestUser/filterWhitelist/RoleTableTestUser?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*_get=1&*_post=0&
*_put=0&*_delete=0&*_head=1&*_options=1
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Allow GET, HEAD and OPTIONS requests against every user
resource whose identifier starts with RoleTable.
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/users/
RoleTableTestUser/filterWhitelist/RoleTable.*?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*_get=1&*_post=0&
*_put=0&*_delete=0&*_head=1&*_options=1
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId __get __post __put __delete __head
__options
UPDATE:
*_get *_post *_put *_delete *_head *_options
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Filter blacklist records are stored in the encrypted ResourcesDB
filterBlacklist table. The `{filterUser}` URL element maps to
userResourceId, the parent `{organization}` maps to orgResourceId,
and the method columns (`_get`, `_post`, `_put`, `_delete`, `_head`,
`_options`) define which methods are denied for that target.
`userResourceId` and `orgResourceId` are evaluated as POSIX extended
regular expressions and must match the full requested user and
organization resource values.
Role-table authorization is evaluated first; matching blacklist
records then deny the request before whitelist allow records are
considered.
Example 1) Deny RoleTableTestUser GET, HEAD and OPTIONS requests
against RoleTableTestUser user resources.
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/users/
RoleTableTestUser/filterBlacklist/RoleTableTestUser?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*_get=1&*_post=0&
*_put=0&*_delete=0&*_head=1&*_options=1
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Deny GET, HEAD and OPTIONS requests against every user
resource whose identifier starts with RoleTable.
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/users/
RoleTableTestUser/filterBlacklist/RoleTable.*?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*_get=1&*_post=0&
*_put=0&*_delete=0&*_head=1&*_options=1
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _location _type
_storageId _accessPath _accessUser _accessPassword
UPDATE:
*resourceInfo *location *type *accessPath
*accessUser *accessPassword
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resources>
Example 1) List attributes of storage with type = 'local'
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&_type=local
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Delete all storage resources with location
'localhost'
METHOD:
DELETE
URI:
https://localhost/organizations/EngineOrg/storage?
userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&_location=localhost
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _location _type
_accessPath _accessUser _accessPassword
UPDATE:
*resourceInfo *location *type *accessPath
*accessUser *accessPassword
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resource>
Example 1) List attributes of storage EngineStorage
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage?userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&_type=local
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Create storage resource EngineStorage2
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage?userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&*resourceInfo=
storage%202&*location=localhost&*type=local&*accessPath
=/opt/storage2&*accessUser=undefined&*accessPassword=
undefined
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET HEAD OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
<NONE>
UPDATE:
<NONE>
DOCUMENT TYPES:
file.csv file.raw file.txt file.json file.xml file.html
file.pdf file.png file.jpg file.gif file.zip file.bin
script.perl script.python
RESPONSE HEADERS:
<NONE>
RESPONSE BODY:
<Document type support status for GET and OPTIONS; HEAD
returns status only>
The documentTypes collection lists the supported document type
identifiers. Individual documentType resources validate whether the
requested type is supported before document collection and document
resource handlers run. Supported types are file.csv, file.raw,
file.txt, file.json, file.xml, file.html, file.pdf, file.png,
file.jpg, file.gif, file.zip, file.bin, script.perl and
script.python.
These routes are handled through the main storage resource tree
dispatcher at `/organizations/{organization}/storage/{storage}`.
Supported HTTP methods: GET PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
_documentId
UPDATE:
*resourceInfo
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resources>
Exact-match filters for _documentId, _storageId and _orgResourceId
may use protected lookup indexes when available. These indexes only
narrow the candidate set; CaumeDSE still decrypts and verifies the
protected document attributes before returning, updating or deleting
registers. Legacy registers without lookup values remain supported.
Document collection and document resource routes are dispatched
under `/organizations/{organization}/storage/{storage}/documentTypes/{documentType}`.
Example 1) List attribute table for all document resources of
type file.raw
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.raw/documents?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Delete all document resources of type file.csv
METHOD:
DELETE
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET POST PUT HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
UPDATE:
*resourceInfo
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching resources>
Example 1) List attribute table for document myfile.bin of type
file.raw
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.raw/documents/
myfile.bin?userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Create (Upload) document myfile.bin of type file.raw
METHOD:
POST (multipart/form-data)
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.raw/documents/
myfile.bin
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<Authentication/attribute update parameters, as
well as the file parameter in multipart/form-data
format>
Example 2.1) HTML form to send parameters and file myfile.bin in
multipart/form-data format:
<html><body>
<h2>Caume Data Security Engine - Document resource post form example - file.raw
</h2><br>
<strong>RAW Document upload form:</strong><br>
<form action="https://localhost/organizations/EngineOrg/storage/EngineStorage/
documentTypes/file.raw/documents/myfile.bin" method="post" enctype=
"multipart/form-data"><br>
file: <input name="file" type="file" value="myfile.bin"><br>
userId: <input name="userId" type="text" value="EngineAdmin"><br>
orgId: <input name="orgId" type="text" value="EngineOrg"><br>
orgKey: <input name="orgKey" type="password" value=
"0CDBB9AF76AF43BDB72E095989E612CC"><br>
*resourceInfo: <input name="*resourceInfo" type="text" value=
"This is a raw file"><br>
<input type="submit" value=" Send "></form>
</body></html>Example 2.2) Body part of a multipart/form-data encoded request
using the HTML example format above:
-----------------------------204285202715621161041257990753
Content-Disposition: form-data; name="file"; filename="cleartext.txt"
Content-Type: text/plain
This is cleartext This is cleartext This is cleartext This is cleartext.
-----------------------------204285202715621161041257990753
Content-Disposition: form-data; name="userId"
EngineAdmin
-----------------------------204285202715621161041257990753
Content-Disposition: form-data; name="orgId"
EngineOrg
-----------------------------204285202715621161041257990753
Content-Disposition: form-data; name="orgKey"
0CDBB9AF76AF43BDB72E095989E612CC
-----------------------------204285202715621161041257990753
Content-Disposition: form-data; name="*resourceInfo"
This is a raw file
-----------------------------204285202715621161041257990753--Example 3) Delete document resource myfile.bin of type file.raw
METHOD:
DELETE
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.raw/documents/
myfile.bin?userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET HEAD OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
UPDATE:
<NONE>
PERL SCRIPT CALLBACK SUBROUTINES:
cmePERLProcessRow:
Called on each iteration to processes every
row. Receives the whole row as an array
(@_).
The script main function is also called
before processing every row to perform
general tasks (e.g. initialization).
cmePERLProcessColumnNames:
Called once to process the first row which
should contain the column names. Receives
the whole row as an array (@_). Should be
used to set indexes of columns to be
processed.
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Contents of parsed file.csv with parser script>
The parserScripts collection supports OPTIONS. Individual
parserScript resources load the named script.perl or script.python
document, reconstruct it through the secure-file path, and run it
against file.csv document content after normal authorization
succeeds. Perl parser requests run in a child `perl` process through
a compatibility runner that invokes `cmePERLProcessColumnNames` and
`cmePERLProcessRow` over secure temporary CSV input/output files.
Python scripts are executed with `python3` and receive two
command-line arguments: an input CSV file path and an output CSV file
path. Parser child processes use absolute interpreter paths from
`CDSE_PARSER_PERL_PATH` and `CDSE_PARSER_PYTHON_PATH`, run from the
parser temporary directory configured by `CDSE_PARSER_TMP_FILE_PATH`,
receive a minimal `PATH`/locale environment, redirect stdin/stdout/stderr
to `/dev/null`, and close inherited file descriptors above stderr before
`execve()`. Parser input/output and Perl runner files are created with
`mkstemp()` as `0600` regular files in a service-owned `0700` parser
temporary directory; existing symlink directories are refused before file
creation. Children are bounded by `CDSE_PARSER_SCRIPT_TIMEOUT_SECONDS`;
where supported, CaumeDSE also
applies `RLIMIT_CPU`, `RLIMIT_FSIZE`,
`CDSE_PARSER_SCRIPT_MAX_ADDRESS_SPACE_BYTES`,
`CDSE_PARSER_SCRIPT_MAX_OPEN_FILES`, and
`CDSE_PARSER_SCRIPT_MAX_PROCESSES`. Parser output CSV files are
rejected above `CDSE_PARSER_SCRIPT_MAX_OUTPUT_BYTES` before loading,
and parser result tables are rejected above
`CDSE_PARSER_SCRIPT_MAX_RESULT_CELLS`. These limits are compile-time
macros with conservative defaults. Optional isolation controls can be
requested with compile-time macros or service environment variables:
`CDSE_PARSER_NO_NEW_PRIVS=1` requires Linux `no_new_privs`,
`CDSE_PARSER_ISOLATE_NETWORK=1` requires a private Linux network
namespace via `unshare(CLONE_NEWNET)`, and `CDSE_PARSER_CHROOT_PATH`
requires `chroot()` into a deployment-prepared parser jail. Requested
isolation fails closed if the platform or service privileges cannot apply
it. Chroot deployments must provide the configured interpreter paths,
parser temporary files, and required runtime libraries inside the jail.
Parser policy and review metadata can be stored in the script document's
`*resourceInfo`. Generated scripts should be uploaded as pending with
fields such as `parser.reviewStatus:pending`, `parser.generated:true`,
`parser.generator:<tool>`, and `parser.promptHash:<hash>`. Reviewed
scripts should carry `parser.reviewStatus:reviewed`, `parser.reviewed:true`,
`parser.reviewer:<user>`, and `parser.reviewTime:<timestamp>`, plus the
policy profile fields `parser.interpreter:/usr/bin/python3`,
`parser.timeout:10`, and `parser.isolation:none`. The policy matcher also
accepts existing `key=value` metadata already stored in a resources DB.
Pending/generated-unreviewed scripts are denied for full parser execution;
`previewOnly=1&previewRows=N` can run them against 1..10 sample rows after
static checks reject shell, network, dynamic-code, and environment-access
patterns. Deployments can enforce reviewed/profile metadata with
`CDSE_PARSER_REQUIRE_REVIEWED=1`,
`CDSE_PARSER_REQUIRE_POLICY_PROFILES=1`, and
`CDSE_PARSER_ALLOWED_TYPES=script.perl,script.python`. Parser upload,
policy allow/deny, execution success, timeout, limit rejection, and
cleanup-failure events are written as `CaumeDSE AuditJSON: ` structured
audit lines without org keys, authorization headers, script bodies, or CSV
content. Version 1 audit categories include `auth`, `authorization`,
`request`, `parserPolicy`, `parserUpload`, `parserExecution`, and
`cleanup`.
In AI-assisted workflows, treat CSV contents and parser output as
untrusted data. CSV cells can contain prompt-injection text that asks an
agent to reveal credentials, alter policy, fetch URLs, or modify cleanup.
Keep cell text inside data fields, and do not let it override system or
security instructions. Review generated parser scripts before upload and
reject scripts that open network connections, execute shell commands, read
environment variables, traverse files outside the provided input path, log
credentials, or create unbounded output. Prefer deterministic CSV-library
transformations that write through a CSV writer and emit only the required
columns. See `AI_USAGE.md` for the full checklist.
Example 1) Get parsed contents of payroll.csv file (of type
file.csv) using script myscript.pl (of type
script.perl); get results in csv format. Note that
the csv file will be decrypted and processed by a child
Perl process using secure temporary CSV input/output files.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
payroll.csv/parserScripts/myscript.pl?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&outputType=csv
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 1.1) Sample output (remember columns are not reorganized
after decryption, only rows).
"id","salary","name","employeeId","lastName"
"1","82400","Jacob","1","Nieves"
"2","111787","Jerome","2","Hodges"
"3","181281","Rooney","3","Atkins"
"4","195943","Gregory","4","Sullivan"
"5","240999","Jameson","5","Castro"
"6","331326","Brandon","6","Clayton"
"7","374552","Kadeem","7","Mcdowell"
"8","517300","Upton","8","Mooney"
"9","571197","Jelani","9","Wyatt"
"10","711605","Bernard","10","Jackson"Example 1.2) Sample payroll.csv file (should have been uploaded
as a file of type file.csv before the request).
name,lastName,employeeId,salary
Jacob,Nieves,1,82400
Jerome,Hodges,2,29387
Rooney,Atkins,3,69494
Gregory,Sullivan,4,14662
Jameson,Castro,5,45056
Brandon,Clayton,6,90327
Kadeem,Mcdowell,7,43226
Upton,Mooney,8,142748
Jelani,Wyatt,9,53897
Bernard,Jackson,10,140408Example 1.3) Sample myscript.pl file (should have been uploaded
as a file of type script.perl before the request).
This particular scripts sums every number in the
salary columns and stores in this same column the
partial results (i.e. the last value contains the
complete sum).
if ($init eq undef)
{
print "Global initialization of Perl Script\n";
$init=1;
$colsum=0;
$index=-1;
}
else
{
$init+=1;
}
print "This is run number ".$init."\n";
sub cmePERLProcessRow #Process a row - CaumeDSE Iterations
{
my (@r) = @_;
print "PERL sub cmePERLProcessRow array: @r\n";
if ($index >= 0)
{
$colsum+=$r[$index];
$r[$index]=$colsum; #Accumulate results in this column
} else {
print "PERL sub cmePERLProcessRow, no column named - salary - found!\n";
}
print "current sum = $colsum\n";
print "PERL sub cmePERLProcessRow, result array: @r\n";
(@r);
}
sub cmePERLProcessColumnNames #Get (and optionally modify) column names
{
$index=-1; #set index for sum column
my $cont=0;
my (@cn) = @_;
foreach (@cn)
{
if ($_ eq "salary") #set index.
{
$index=$cont;
print "PERL sub cmePERLProcessColumnNames index for - salary - found: $index.\n";
}
$cont++;
}
print "PERL sub cmePERLProcessColumnNames array: @cn\n";
print "PERL sub cmePERLProcessColumnNames, result array: @cn\n";
(@cn);
}Supported HTTP methods: GET HEAD OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
UPDATE:
<NONE>
SUPPORTED FILE TYPES:
file.csv file.raw file.txt file.json file.xml file.html
file.pdf file.png file.jpg file.gif file.zip file.bin
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Contents the file>
Example 1) Get contents of payroll.csv file (of type file.csv); get
results in csv format.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
payroll.csv/content?userId=EngineAdmin&orgId=EngineOrg
&orgKey=0CDBB9AF76AF43BDB72E095989E612CC&outputType=csv
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: POST PUT GET HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
UPDATE:
[column_name]
UPDATE NOTES:
For POST, {contentRow} must be exactly the next
available numeric row (it behaves as an append);
existing column for which a new value was not
specified will receive a new empty string.
For PUT, {contentRow} must be in the range 1 - last
row; existing columns for which a new value was not
specified will retain their previous value.
SUPPORTED FILE TYPES:
file.csv
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Row contents or result of operation>
Example 1) Get row 456 of document document.csv (of type file.csv);
get results in csv format.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
document.csv/contentRows/456?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
&outputType=csv
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Update columns salary (with 100.20 as new value) and
name (with James Duncan as new value) at row 100 of
document document.csv (of type file.csv). Existing
columns lastName and employeeId will retain their
previous values.
METHOD:
PUT
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
document.csv/contentRows/100?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
&[salary]=100.20&[name]=James
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 3) Add row 101 (assuming there are currently 100 rows,
not counting column names) with column salary empty,
column lastName with value Robertson, column name
with value Sam and column employeeId with value 101,
to document document.csv (of type file.csv).
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
document.csv/contentRows/101?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
&[name]=Sam&[lastName]=Robertson&[employeeId]=100
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: POST GET HEAD DELETE OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _resourceInfo _columnFile _partHash
_totalParts _partId _lastModified _columnId
UPDATE:
<NONE>
UPDATE NOTES:
For POST, {contentColumn} can refer to a new column
in a non-existent document. In this case the
document will be created along with {contentColumn}
as its first column.
For DELETE, if {contentColumn} is the only column
left in the document, the whole document is deleted.
SUPPORTED FILE TYPES:
file.csv
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Row contents or result of operation>
Example 1) Get column Col1 of document document.csv (of type
file.csv); get results in csv format.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
document.csv/contentColumns/Col1?userId=EngineAdmin
&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&outputType=csv
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 2) Delete column salary of document document.csv (of
type file.csv).
METHOD:
DELETE
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
document.csv/contentColumns/salary?userId=EngineAdmin
&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 3) Add Column Col1 to non-existing document
newdocument.csv (which also creates the document of
type file.csv).
METHOD:
POST
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/documentTypes/file.csv/documents/
newdocument.csv/contentColumns/Col1?userId=EngineAdmin
&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET HEAD OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
<NONE>
UPDATE:
<NONE>
SECURITY SCOPE:
These resources expose a read-only diagnostics view over
registered secure document databases for file.csv
documents in the selected organization storage. They do
not browse arbitrary database files and do not expose
decrypted internal ResourcesDB, RolesDB, or LogsDB
records. Requests must be authorized through the
corresponding role table and must provide userId, orgId,
and orgKey. The secure document is loaded through the
normal verification path before data is returned.
SELECTORS:
/dbNames lists registered file.csv document database names.
/dbNames/{dbName}/dbTables lists exposed tables.
/dbTables/{dbTable} returns table rows for data or meta.
/tableRows/{tableRow} returns one 1-based row.
/tableColumns lists column names.
/tableColumns/{tableColumn} returns one column, with id.
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Matching diagnostic table>
Example 1) List secure CSV document databases in storage
EngineStorage.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/dbNames?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
Example 2) Read row 1 from the data table of document.csv.
METHOD:
GET
URI:
https://localhost/organizations/EngineOrg/storage/
EngineStorage/dbNames/document.csv/dbTables/data/
tableRows/1?userId=EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
Supported HTTP methods: GET HEAD OPTIONS
Supported ATTRIBUTE PARAMETERS:
MATCH:
_userId _orgId _requestMethod _requestUrl
_requestHeaders _startTimestamp _endTimestamp
_requestDataSize _responseDataSize _orgResourceId
_requestIPAddress _responseCode _responseHeaders
_authenticated
UPDATE:
<NONE>
RESPONSE HEADERS:
Engine-results: <number of matching registers>
RESPONSE BODY:
<Attribute table for matching transactions>
Example 1) List all transactions
METHOD:
GET
URI:
https://localhost/transactions?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Example 1) List all transactions for non authenticated users
METHOD:
GET
URI:
https://localhost/transactions?userId=
EngineAdmin&orgId=EngineOrg&orgKey=
0CDBB9AF76AF43BDB72E095989E612CC&_authenticated=0
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<EMPTY>
Supported HTTP methods: GET
Supported ATTRIBUTE PARAMETERS:
MATCH:
<NONE>
UPDATE:
<NONE>
RESPONSE HEADERS:
Content-Type: image/x-icon
RESPONSE BODY:
<Icon image of the CaumeDSE (typically used by browsers
)>
Example 1) Get contents of the favicon.ico file.
METHOD:
GET
URI:
https://localhost/favicon.ico?userId=EngineAdmin&orgId=
EngineOrg&orgKey=0CDBB9AF76AF43BDB72E095989E612CC
REQUEST HEADERS:
<NONE>
REQUEST BODY:
<image file>
Maintaining open source software with strict security requirements is hard. We appreciate all help we can get, be it by coding/ fixing code or just by submitting errors or functionality requests.
We will do our best to listen to everybody an try to answer all requests. However, to maintain control and ensure quality we must restrict the number of people making decisions and not everyone's wishes will be pleased. Still, we expect that the layered model of the software platform will allow almost any functionality to be implemented in a higher level, while we try to maintain the engine (CaumeDSE) as small and stable as possible.
If you wish to contribute in any way please visit the project page (Caume/CaumeDSE) in github.
While all operations are performed in memory and some measures have been implemented to limit data leaks, such as overwriting keys after use, it should be noted that users with sufficient privileges may still be able to access memory contents during operation (e.g. by dumping the contents of memory devices in Unix based operating systems). Also, memory swaps to disks may occur (these are controlled by the operating system).
Also, permissions of directories where files are stored, particularly of the secureTmp directory (where unencrypted files are posted before being processed/encrypted and then overwritten as part of POST requests) should be restricted to limit unauthorized access.
You may consider running this software with a limited account; just take into account that it has to be able to access Perl libraries, its own database directories, and listen to incoming HTTPS connections ( and also HTTP in DEBUG mode). Alternatively you may consider running CaumeDSE in a chrooted environment.
Note that DEBUG mode makes use of HTTP (and then switches to HTTPS after the first [enter] is pressed) which does not encrypt incoming connections. Debug mode also dumps to the console sensitive information including passwords. The default is to compile in release mode. To change to debug mode (which is slower), use the following parameter when running ./configure:
--enable-DEBUG
Before running CaumeDSE in DEBUG mode, copy the contents of TEST/testfiles to /opt/cdse/testfiles and TEST/testDB_opt_cdse to /opt/cdse (or the directory specified by PATH_DATADIR) to provide data for the internal tests.
The DEBUG component verification script can run the build, install the test
database under /tmp/cdse-verify, and execute the noninteractive component
harness, including bounded HTTP and HTTPS startup checks and live API request
flows:
TEST/run_debug_components.sh
For CI smoke coverage, use:
TEST/run_debug_components.sh --ci-smoke
The full mode uses CDSE_DEBUG_TEST_HTTP_PORT and
CDSE_DEBUG_TEST_HTTPS_PORT when set, or ports 18080 and 18443 by default.
The script rejects invalid, duplicate, or busy ports before launching the
debug executable. In full mode it also starts a held-open DEBUG web service
for each protocol and uses curl to authenticate, create a temporary
organization and storage resource, upload a CSV document, query a row and
column, upload a script.perl document, and execute that parser script
through both HTTP and HTTPS. The live flow also asserts negative
authentication cases for missing credentials on both protocols and missing or
mismatched client certificates on HTTPS. HTTPS uses a per-run client
certificate chain signed by the committed test CA fixture. The verifier also
checks that HTTP TLS-auth bypass is visible in DEBUG/test startup logs and that
release configure rejects the bypass flag.
CDSE_DEBUG_TEST_TIMEOUT controls the overall executable timeout and defaults
to 120s. Use --skip-web only when the local environment cannot bind test
ports. Live API runs also write
coverage artifacts under the log directory: live-api-coverage.csv for
machine-readable comparisons and live-api-coverage.txt for the fixed-width
summary table. Each row records protocol, feature name, inferred HTTP method,
expected and actual status, curl result, marker status, elapsed time, and
body/meta log paths.
Set CDSE_VERIFY_REDACT=1 for CI logs or AI-assisted debugging sessions. In
that mode the verifier masks orgKey, newOrgKey, selected credential-style
request parameters, and generated certificate/key file paths in summary.txt,
live request body/meta artifacts, the full DEBUG run log, component extract
logs, and live service logs. Redaction is disabled by default so local
debugging keeps full request details unless explicitly requested.
The --ci-smoke profile runs configure, build, install, component markers,
HTTP/HTTPS startup checks, and one live API protocol. It defaults to HTTP live
coverage; use --ci-smoke --web-protocol=https to select HTTPS instead.
For sanitizer coverage, use a DEBUG/test build with Clang and the explicit sanitizer configure option:
CC=clang ./configure --prefix=/tmp/cdse-sanitize --enable-DEBUG --enable-TESTDATABASE --enable-BYPASSTLSAUTHINHTTP --enable-SANITIZERS=address,undefined --disable-HARDENING
make
make check
make install
CDSE_VERIFY_PREFIX=/tmp/cdse-sanitize \
CDSE_VERIFY_LOG_DIR=/tmp/cdse-sanitize-components \
CDSE_VERIFY_REDACT=1 \
ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:strict_string_checks=1:detect_stack_use_after_return=1 \
UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1 \
TEST/run_debug_components.sh --skip-build --skip-web
--enable-SANITIZERS accepts address, undefined, or
address,undefined; yes selects both. Sanitizer builds are intentionally
DEBUG-only and do not change normal release or default DEBUG builds. The full
DEBUG verifier still exercises legacy embedded-Perl helper tests and currently
runs with LeakSanitizer disabled to avoid external interpreter noise; use
detect_leaks=1 only for focused C-only reproductions where the embedded
interpreter is not part of the signal.
Before promoting a change to master, run the checks that apply to its risk
and retain their redacted summaries with the pull request:
- Run
bash -n TEST/run_debug_components.sh, configure a DEBUG/test build, then runmake,make check, andCDSE_VERIFY_REDACT=1 TEST/run_debug_components.sh --skip-build --skip-web. - For HTTP(S) API, authentication, storage, or live-verifier changes, run both
CDSE_VERIFY_REDACT=1 TEST/run_debug_components.sh --live-only --web-protocol=httpand the corresponding--web-protocol=httpscommand. Reviewsummary.txt,live-api-coverage.csv, andlive-api-coverage.txtin the verifier log directory. - For C, memory-management, parser-runtime, or undefined-behavior changes,
run the Clang AddressSanitizer/UndefinedBehaviorSanitizer profile above,
including
make checkand the non-web verifier. - Confirm the pull request CI jobs have passed. Treat a CI web-smoke socket limitation as acceptable only when its non-web component verifier passed and the limitation is recorded in the job output. For code changes, also confirm the default release configure/build/check/install/distcheck job passed and its bypass-rejection guard succeeded.
- Do not attach unredacted logs, organization keys, private certificates, or generated key paths. State the commands run, pass/fail counts, skipped checks, and retained redacted artifact locations in the pull request.
Pull requests run the GitHub Actions workflow in .github/workflows/pr-ci.yml.
Documentation-only PRs run lightweight shell, OpenAPI route-contract, and
TODO-format checks. Code, build, test, workflow, and sample changes run the
DEBUG configure/build path, make, make check, standalone sample self-tests,
and the redacted DEBUG component verifier. The CI workflow also attempts TEST/run_debug_components.sh --ci-smoke; if a GitHub-hosted runner denies local socket creation or binding,
the web smoke step is treated as an environment limitation only after the
non-web component verifier has passed. A separate sanitizer job runs a Clang
DEBUG build with AddressSanitizer and UndefinedBehaviorSanitizer, then executes
make, make check, installation, and the non-web component verifier with
redacted logs. The sanitizer job fails on invalid accesses and undefined
behavior; LeakSanitizer is disabled for the full verifier profile because
legacy embedded-Perl helper tests are still part of that run.
The default release job separately configures without DEBUG/test switches,
builds, runs make check, stages an installation, and verifies that release
configuration rejects the HTTP TLS-auth bypass. It also runs make distcheck
against the generated source archive to verify a clean release build, check,
install, and uninstall. Verifier, sanitizer, and release-build diagnostics are
uploaded as short-lived PR artifacts.
.github/workflows/scheduled-live-api.yml runs every Monday at 03:17 UTC and
can also be started manually. It builds the DEBUG/test profile and requires
both HTTP and HTTPS live API flows to pass, including the HTTPS client-certificate
path. Its verifier artifacts are redacted and retained for 14 days.
The committed test database under TEST/testDB_opt_cdse uses
0CDBB9AF76AF43BDB72E095989E612CC as the EngineAdmin / EngineOrg
organization key in the DEBUG resource component tests and API examples.
Older history shows the previous fixture key
6DA74D788E0A33A0272252796EF0748A in TEST/testDB_opt_cdse/secureTmp/EngineOrg.key.txt
and older README examples; it is not valid for the current committed fixture.
The password1 value used by several component tests is a document/resource
fixture key for generated CSV and DB-browsing resources, not the default
EngineAdmin organization key.
For automated first-run release verification, the CaumeDSE binary accepts
optional startup parameters:
--admin-org-key KEY
--admin-key-confirmed
--https-port PORT
--admin-org-key sets the initial EngineAdmin organization key when the
system databases do not exist yet. --admin-key-confirmed skips the
interactive prompt that normally waits for the operator to confirm that the
generated key has been written down. --https-port runs HTTPS on a specific
port instead of the default 8443.
Temporary-file deletion uses one zero-fill overwrite pass by default. To
compile with additional overwrite passes, define
CDSE_SECURE_OVERWRITE_PASSES in CFLAGS, for example:
CFLAGS="-DCDSE_SECURE_OVERWRITE_PASSES=3" ./configure
CaumeDSE does not validate OAuth signatures, bearer tokens, refresh
tokens, scopes, expiry, or revocation internally. OAuth deployments must
use an external engine manager or reverse-proxy layer that owns those
protocol duties. After validating an OAuth grant, that layer should
create a delegated CaumeDSE scope, for example an organization named
<orgId>_OAUTH, with a manager-held organization key. It should then
create the delegated user, role-table entries, filter-list entries, and
resource copies or references that represent the OAuth grant.
Requests forwarded to CaumeDSE must use the delegated userId, orgId,
and orgKey parameters. When the OAuth grant expires or is revoked, the
manager must delete the delegated organization and associated resources.
CaumeDSE intentionally does not store organization keys or OAuth tokens,
so the manager must keep its own mapping from OAuth grants to delegated
CaumeDSE scopes.
For AI-agent integrations, prefer short-lived delegated tokens issued by that
external manager instead of exposing the CaumeDSE organization key to the
agent. A delegated token should be opaque to CaumeDSE and should include, at
the manager layer, a subject, narrow scopes, expiry, a revocation identifier,
and the delegated CaumeDSE user/organization binding. After the manager
validates the token, it forwards the request to CaumeDSE using the
manager-held delegated userId, orgId, and orgKey, while role-table and
filter-list rows enforce the same narrow permissions inside CaumeDSE. See
samples/delegated-token-broker/ for a standard-library Python sample and
offline allow/deny/expiry/revocation checks.
CaumeDSE does not automatically migrate protected SQLite values when an operator changes an organization key or storage crypto profile. Use an explicit re-protect workflow:
- Take an encrypted backup and record the target organization, storage, document, source profile, target profile, and operator-approved scope.
- Run a dry-run inventory for the selected ColumnFile DBs and review row counts, legacy AES rows, Herradura-framed rows, and unsupported MAC/sign metadata before mutation.
- Create an operator-held journal outside model-visible logs. Record one checkpoint per ColumnFile before mutation, after successful DB transaction, and after readback with the new key/profile. Do not include org keys.
- Re-protect through the DB-level helper so salts and profile metadata are updated in one SQLite transaction.
- Verify readback with the target key/profile before marking the checkpoint complete. If verification fails, restore from backup or keep the old checkpoint as the authoritative state.
Mixed AES/Herradura data is allowed during staged migration. MAC/sign protected metadata currently fails closed in the DB-level helper until a dedicated recomputation workflow updates those integrity columns with the new ciphertext and key material.
New internal SQLite databases include a schema_meta table with schemaClass,
schemaVersion, definitionsVersion, and migrationState=complete. Startup
validates ResourcesDB, RolesDB, and LogsDB before use; secure ColumnFile DBs can
be checked with the same DB helper when an explicit workflow needs to assert
storage compatibility.
Legacy databases without schema_meta remain readable when their required
tables and columns match a known compatible layout. Startup explicitly upgrades
legacy ResourcesDB lookup columns and legacy LogsDB transaction columns, then
writes current schema metadata. Databases with future schema versions, missing
metadata rows, migrationState values other than complete, missing required
columns, or wrong column types fail closed with schema diagnostics. Take an
encrypted backup before schema or key/profile migration and keep the old
checkpoint authoritative until readback succeeds after upgrade.
In release mode the software enters and infinite loop to answer connections; right now you need to kill the process to stop it).
If you want to bypass TLS authentication for testing purposes in DEBUG mode with HTTP (where client-certificate authentication cannot run), you may enable this feature (FOR TESTING PURPOSES ONLY) with:
--enable-BYPASSTLSAUTHINHTTP
The bypass is rejected unless --enable-DEBUG is also present, and DEBUG
startup logs print whether HTTP TLS-auth bypass is enabled. Do not use this
flag for deployable builds.
If you want to enable the use of the old Password Based Key Derivation
Function PBKDF1 (i.e. PKCS5v1.5, which is compatible with openssl's command
line tool but is NOT RECOMMENDED; it uses MD5+the default cipher algorithm
and a counter = 1) use the following configure switch.
--enable-OLDPBKDF1
The default Password Based Key Derivation Function standard is now PBKDF2
(PKCS5v2.0, using HMAC-SHA256 and cmeDefaultPBKDFCount iterations) for
newly protected data.
Data written with the earlier PBKDF2-HMAC-SHA1 profile and 2,000 iterations
remains readable through a decrypt-only legacy fallback. The older optimized
single-iteration path for sufficiently long hexadecimal binary keys is also
limited to that legacy fallback path so existing protected values can be read
without using it for new writes.
The default symmetric encryption algorithm used by CaumeDSE is AES-256-GCM.
You may override this at runtime with a configuration file, or by setting the
environment variable CDSE_DEFAULT_ENC_ALG to a supported storage encryption
profile. In default builds, supported profiles are OpenSSL cipher names such as
aes-256-cbc for AES in CBC mode. By default CaumeDSE reads caumedse.conf
from the configured data directory, and CDSE_CONFIG_FILE may point to an
alternate configuration file. The supported configuration keys for this setting
are defaultEncAlg, default_enc_alg, and CDSE_DEFAULT_ENC_ALG; the
environment variable takes precedence over the configuration file.
HerraduraKEx support is optional and scoped to internal data encryption at rest for protected values stored in SQLite-backed CaumeDSE databases and protected file parts. It does not change TLS channel encryption, HTTPS certificates, client-certificate authentication, or libmicrohttpd/GnuTLS behavior.
Builds that need HerraduraKEx must opt in explicitly:
./configure --enable-HERRADURAKEX --with-herradurakex=/path/to/HerraduraKExThe path may point to a HerraduraKEx repository root or include directory
containing herradura.h. Configure checks for 256-bit key constants and the
HSKE-NL AEAD entry points. Default builds do not enable Herradura algorithm names
and reject them instead of falling back to OpenSSL.
Recommended storage profiles:
herradura-hske-nla1-aead-256: initial PQC-oriented at-rest encryption candidate. Use this first when the deployment has passed the Herradura-enabled DEBUG and live verifier checks.herradura-hske-duplex-256: evaluation profile for variable-size SQLite fields when a direct arbitrary-length AEAD interface is preferred.herradura-hske-nla2-256: experimental only unless a specific reversible-permutation storage use case is documented.
Do not use hkex-rnl for direct SQLite value encryption. Treat it as a future
key-wrapping or offline key-establishment candidate. Treat hfscx-256 and
hfscx-256-ds as future Herradura-native hash/MAC candidates, not replacements
for the existing compatibility HMAC paths in the first implementation. Do not
use hpke-stern, hpke-stern-kem, or hpks-stern for production CaumeDSE
storage until upstream production decoder and round requirements are satisfied.
Herradura-protected values use a versioned CDSEHKX1 frame that records the
compact profile id, nonce, tag, and ciphertext. Existing AES-GCM rows are not
migrated automatically and remain readable. Herradura-protected rows require a
Herradura-enabled binary for readback or rollback.
All other keys are assumed to be human generated passwords or passphrases
which require key expansion with a slow function, in order to limit
dictionary and brute force attacks. Therefore, they are processed by the
PBKDF2 function using the defined number of iterations each time, which
slows down all the encryption and decryption processes.
The default key generated for EngineAdmin when creating a new database is now an hexadecimal representation of a binary key with the same langth as the default cipher key (i.e. cryptographic operations are much faster). However, the example database included has a shorter key for the default administrator account EngineAdmin; using the example database with the EngineAdmin account is therefore much slower.
By default, hardening parameters are included as parto of compiler and linker options. If you need to disable them (e.g. for testing) use:
--disable-HARDENING