Add Project DB per-domain role provisioning functions

[esoergel]

These three stacked PRs introduce a database-level safeguard that keeps a Project DB query from reading another domain's data, even if the query layer above it has a bug or is handed hostile SQL.

Part I: HQ Query Users (This PR)

Each domain gets its own postgres role, which has read-only access to only that domain's tables. Queries on behalf of that domain are authenticated as that user, so postgres enforces separation between tenants.

HQ provisions these query users using a constrained SECURITY DEFINER function called projectdb_provision_role which narrowly defines the action. This function is created during database provisioning. Passwords are derived from the domain name via django.utils.crypto.salted_hmac, never stored.

Part II: PgBouncer

PgBouncer connection pooling complicates things in two ways

Part IIa: PGBouncer Authentication (#6963)

PgBouncer authenticates queries itself, so it has to know user passwords. Currently, it validates against a file called userlist.txt which is managed by commcare-cloud. Adding a new user requires amending that file, which just isn't doable in this context. Instead, we give pgbouncer an auth_query it can run to get the username and password to authenticate a user that's not in userlist.txt.

To do this, we need a few things in place:

• A user_lookup function as described in the pgbouncer docs (https://www.pgbouncer.org/config.html), supporting the auth_query
• An auth_user pgbouncer can authenticate as when running the auth_query, and a password for that user
• A role for the pgbouncer auth user with only LOGIN permissions by default
  • This role must also be granted access to the user_lookup function

Part IIb: (#6965)

PgBouncer keys its pools by (user, database) pair, meaning that each domain will have it's own pool in pgbouncer. Right now, we allow up to 490 open connections per pool (default_pool_size). Since ProjectDB uses many pools, we can instead set a max_db_connections as a limit across all ProjectDB pools, and set a much lower pool_size to prevent one domain from claiming all that for itself.

Environments Affected

Staging, since it has project_db configured already. Should only affect other environments when that's enabled.

Announce New Release

No. Shouldn't require action by anyone but me.

[MartinRiese]

so {{ postgres_users.commcare.username }} is a superuser?

[esoergel]

No, the commcare user doesn't normally have permission to modify roles. Its permissions are limited, and granting CREATEROLE directly would open up potential privilege escalation paths:

Because the CREATEROLE privilege allows a user to grant or revoke membership even in roles to which it does not (yet) have any access, a CREATEROLE user can obtain access to the capabilities of every predefined role in the system, including highly privileged roles such as pg_execute_server_program and pg_write_server_files.

This was actually addressed in PostgreSQL 16, but we're still on 14. So instead, projectdb_provision_role is a SECURITY DEFINER function (meaning it runs with the permissions of the role which created it). By granting just that function to the commcare user, its ability to fiddle with roles is limited to creating and deleting roles starting with projectdb_. Further, roles are cluster wide, and not limited to just the projectdb logical database.

[millerdev]

Offlined with Ethan about concerns around connection pooling per-domain and how that could affect PG resource usage. It was already on his radar.

This looks like a solid start. However, there is a small part of me that wonders if we actually need it? On the con side, it may add significant inconvenience to debugging projectdb data. On the other hand, I can understand the rationale given that the plan is to execute (pre-parsed and re-rendered) user-supplied SQL against these tables. It's a belt-and-suspenders approach.

[millerdev]

With this change, projectdb_provision_role no longer reads pg_roles, so the pg_temp-shadowing concern no longer applies. Seems like pg_temp could be removed from the search path, although maybe it's safer to leave it in case this function changes in a way that needs it later?

[esoergel]

This looks like a solid start. However, there is a small part of me that wonders if we actually need it?

Absolutely. When I pitched the idea, it seemed like a pretty simple way to get a too-simple-to-fail security layer that makes this unassailable, whereas the other mechanisms are probably going to have some degree of brittleness. Not that I don't trust them - it's just like providing DB level schema guarantees vs application layer wrappers.

it may add significant inconvenience to debugging projectdb data

How so? There will still be a CommCareHQ application user with full read and write access to all tables. You and I already spoke about this offline, but recording here for others (and for posterity) - I think we might want to route any kind of complex querying through this user, but we can also use the regular CommCareHQ user for simple access patterns. One benefit of that is that if we GA a version of this, we don't necessarily need a user (and pgbouncer pool) for every domain - if this is accessed only through some Case List Explorer type report or something similarly controlled, we can say that the normal mechanisms of tenant separation are sufficient.

[millerdev]

There will still be a CommCareHQ application user with full read and write access to all tables. ... we can also use the regular CommCareHQ user for simple access patterns.

That makes sense. I had missed/misunderstood it in our original discussion. Thank for reiterating.