[{"data":1,"prerenderedAt":771},["ShallowReactive",2],{"/en-us/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions":3,"navigation-en-us":37,"banner-en-us":465,"footer-en-us":482,"Grzegorz Bizon-Stan Hu":727,"next-steps-en-us":750,"footer-source-/en-us/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/":765},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":27,"_id":30,"_type":31,"title":32,"_source":33,"_file":34,"_stem":35,"_extension":36},"/en-us/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"Why we spent the last month eliminating PostgreSQL subtransactions","How a mysterious stall in database queries uncovered a performance limitation with PostgreSQL.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749669470/Blog/Hero%20Images/nessie.jpg","https://about.gitlab.com/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"Why we spent the last month eliminating PostgreSQL subtransactions\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Grzegorz Bizon\"},{\"@type\":\"Person\",\"name\":\"Stan Hu\"}],\n        \"datePublished\": \"2021-09-29\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},[18,19],"Grzegorz Bizon","Stan Hu","2021-09-29","Since last June, we noticed the database on GitLab.com would\n\nmysteriously stall for minutes, which would lead to users seeing 500\n\nerrors during this time. Through a painstaking investigation over\n\nseveral weeks, we finally uncovered the cause of this: initiating a\n\nsubtransaction via the [`SAVEPOINT` SQL\nquery](https://www.postgresql.org/docs/current/sql-savepoint.html) while\n\na long transaction is in progress can wreak havoc on database\n\nreplicas. Thus launched a race, which we recently completed, to\n\neliminate all `SAVEPOINT` queries from our code. Here's what happened,\n\nhow we discovered the problem, and what we did to fix it.\n\n\n### The symptoms begin\n\n\nOn June 24th, we noticed that our CI/CD runners service reported a high\n\nerror rate:\n\n\n![runners\nerrors](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/ci-runners-errors.png)\n\n\nA quick investigation revealed that database queries used to retrieve\n\nCI/CD builds data were timing out and that the unprocessed builds\n\nbacklog grew at a high rate:\n\n\n![builds\nqueue](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/builds-queue.png)\n\n\nOur monitoring also showed that some of the SQL queries were waiting for\n\nPostgreSQL lightweight locks (`LWLocks`):\n\n\n![aggregated\nlwlocks](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/aggregated-lwlocks.png)\n\n\nIn the following weeks we had experienced a few incidents like this. We were\n\nsurprised to see how sudden these performance degradations were, and how\n\nquickly things could go back to normal:\n\n\n![ci queries\nlatency](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/ci-queries-latency.png)\n\n\n### Introducing Nessie: Stalled database queries\n\n\nIn order to learn more, we extended our observability tooling [to sample\n\nmore data from\n`pg_stat_activity`](https://gitlab.com/gitlab-cookbooks/gitlab-exporters/-/merge_requests/231).\nIn PostgreSQL, the `pg_stat_activity`\n\nvirtual table contains the list of all database connections in the system as\n\nwell as what they are waiting for, such as a SQL query from the\n\nclient. We observed a consistent pattern: the queries were waiting on\n\n`SubtransControlLock`. Below shows a graph of the URLs or jobs that were\n\nstalled:\n\n\n![endpoints\nlocked](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/endpoints-locked.png)\n\n\nThe purple line shows the sampled number of transactions locked by\n\n`SubtransControlLock` for the `POST /api/v4/jobs/request` endpoint that\n\nwe use for internal communication between GitLab and GitLab Runners\n\nprocessing CI/CD jobs.\n\n\nAlthough this endpoint was impacted the most, the whole database cluster\n\nappeared to be affected as many other, unrelated queries timed out.\n\n\nThis same pattern would rear its head on random days. A week would pass\n\nby without incident, and then it would show up for 15 minutes and\n\ndisappear for days. Were we chasing the Loch Ness Monster?\n\n\nLet's call these stalled queries Nessie for fun and profit.\n\n\n### What is a `SAVEPOINT`?\n\n\nTo understand `SubtransControlLock` ([PostgreSQL\n\n13](https://www.postgresql.org/docs/13/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)\n\nrenamed this to `SubtransSLRU`), we first must understand how\n\nsubtransactions work in PostgreSQL. In PostgreSQL, a transaction can\n\nstart via a `BEGIN` statement, and a subtransaction can be started with\n\na subsequent `SAVEPOINT` query. PostgreSQL assigns each of these a\n\ntransaction ID (XID for short) [when a transaction or a subtransaction\n\nneeds one, usually before a client modifies\ndata](https://gitlab.com/postgres/postgres/blob/a00c138b78521b9bc68b480490a8d601ecdeb816/src/backend/access/transam/README#L193-L198).\n\n\n#### Why would you use a `SAVEPOINT`?\n\n\nFor example, let's say you were running an online store and a customer\n\nplaced an order. Before the order is fullfilled, the system needs to\n\nensure a credit card account exists for that user. In Rails, a common\n\npattern is to start a transaction for the order and call\n\n[`find_or_create_by`](https://apidock.com/rails/v5.2.3/ActiveRecord/Relation/find_or_create_by).\nFor\n\nexample:\n\n\n```ruby\n\nOrder.transaction do\n  begin\n    CreditAccount.transaction(requires_new: true) do\n      CreditAccount.find_or_create_by(customer_id: customer.id)\n  rescue ActiveRecord::RecordNotUnique\n    retry\n  end\n  # Fulfill the order\n  # ...\nend\n\n```\n\n\nIf two orders were placed around the same time, you wouldn't want the\n\ncreation of a duplicate account to fail one of the orders. Instead, you\n\nwould want the system to say, \"Oh, an account was just created; let me\n\nuse that.\"\n\n\nThat's where subtransactions come in handy: the `requires_new: true`\n\ntells Rails to start a new subtransaction if the application already is\n\nin a transaction. The code above translates into several SQL calls that\n\nlook something like:\n\n```sql\n\n--- Start a transaction\n\nBEGIN\n\nSAVEPOINT active_record_1\n\n--- Look up the account\n\nSELECT * FROM credit_accounts WHERE customer_id = 1\n\n--- Insert the account; this may fail due to a duplicate constraint\n\nINSERT INTO credit_accounts (customer_id) VALUES (1)\n\n--- Abort this by rolling back\n\nROLLBACK TO active_record_1\n\n--- Retry here: Start a new subtransaction\n\nSAVEPOINT active_record_2\n\n--- Find the newly-created account\n\nSELECT * FROM credit_accounts WHERE customer_id = 1\n\n--- Save the data\n\nRELEASE SAVEPOINT active_record_2\n\nCOMMIT\n\n```\n\n\nOn line 7 above, the `INSERT` might fail if the customer account was\n\nalready created, and the database unique constraint would prevent a\n\nduplicate entry. Without the first `SAVEPOINT` and `ROLLBACK` block, the\n\nwhole transaction would have failed. With that subtransaction, the\n\ntransaction can retry gracefully and look up the existing account.\n\n\n### What is `SubtransControlLock`?\n\n\nAs we mentioned earlier, Nessie returned at random times with queries\n\nwaiting for `SubtransControlLock`. `SubtransControlLock` indicates that\n\nthe query is waiting for PostgreSQL to load subtransaction data from\n\ndisk into shared memory.\n\n\nWhy is this data needed? When a client runs a `SELECT`, for example,\n\nPostgreSQL needs to decide whether each version of a row, known as a\n\ntuple, is actually visible within the current transaction. It's possible\n\nthat a tuple has been deleted or has yet to be committed by another\n\ntransaction. Since only a top-level transaction can actually commit\n\ndata, PostgreSQL needs to map a subtransaction ID (subXID) to its parent\n\nXID.\n\n\nThis mapping of subXID to parent XID is stored on disk in the\n\n`pg_subtrans` directory. Since reading from disk is slow, PostgreSQL\n\nadds a simple least-recently used (SLRU) cache in front for each\n\nbackend process. The lookup is fast if the desired page is already\n\ncached. However, as [Laurenz Albe discussed in his blog\n\npost](https://www.cybertec-postgresql.com/en/subtransactions-and-performance-in-postgresql/),\n\nPostgreSQL may need to read from disk if the number of active\n\nsubtransactions exceeds 64 in a given transaction, a condition\n\nPostgreSQL terms `suboverflow`. Think of it as the feeling you might get\n\nif you ate too many Subway sandwiches.\n\n\nSuboverflowing (is that a word?) can bog down performance because as\n\nLaurenz said, \"Other transactions have to update `pg_subtrans` to\n\nregister subtransactions, and you can see in the perf output how they\n\nvie for lightweight locks with the readers.\"\n\n\n### Hunting for nested subtransactions\n\n\nLaurenz's blog post suggested that we might be using too many\n\nsubtransactions in one transaction. At first, we suspected we might be\n\ndoing this in some of our expensive background jobs, such as project\n\nexport or import. However, while we did see numerous `SAVEPOINT` calls\n\nin these jobs, we didn't see an unusual degree of nesting in local\n\ntesting.\n\n\nTo isolate the cause, we started by [adding Prometheus metrics to track\n\nsubtransactions as a Prometheus metric by\nmodel](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66477).\n\nThis led to nice graphs as the following:\n\n\n![subtransactions\nplot](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/subtransactions-plot.png)\n\n\nWhile this was helpful in seeing the rate of subtransactions over time,\n\nwe didn't see any obvious spikes that occurred around the time of the\n\ndatabase stalls. Still, it was possible that suboverflow was happening.\n\n\nTo see if that was happening, we [instrumented our application to track\n\nsubtransactions and log a message whenever we detected more than 32\n\n`SAVEPOINT` calls in a given\ntransaction](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67918).\nRails\n\nmakes it possible for the application to subscribe to all of its SQL\n\nqueries via `ActiveSupport` notifications. Our instrumentation looked\n\nsomething like this, simplified for the purposes of discussion:\n\n\n```ruby\n\nActiveSupport::Notifications.subscribe('sql.active_record') do |event|\n  sql = event.payload.dig(:sql).to_s\n  connection = event.payload[:connection]\n  manager = connection&.transaction_manager\n\n  context = manager.transaction_context\n  return if context.nil?\n\n  if sql.start_with?('BEGIN')\n    context.set_depth(0)\n  elsif cmd.start_with?('SAVEPOINT', 'EXCEPTION')\n    context.increment_savepoints\n  elsif cmd.start_with?('ROLLBACK TO SAVEPOINT')\n    context.increment_rollbacks\n  elsif cmd.start_with?('RELEASE SAVEPOINT')\n    context.increment_releases\n  elsif sql.start_with?('COMMIT', 'ROLLBACK')\n    context.finish_transaction\n  end\nend\n\n```\n\n\nThis code looks for the key SQL commands that initiate transactions and\n\nsubtransactions and increments counters when they occurred. After a\n\n`COMMIT,` we log a JSON message that contained the backtrace and the\n\nnumber of `SAVEPOINT` and `RELEASES` calls. For example:\n\n\n```json\n\n{\n  \"sql\": \"/*application:web,correlation_id:01FEBFH1YTMSFEEHS57FA8C6JX,endpoint_id:POST /api/:version/projects/:id/merge_requests/:merge_request_iid/approve*/ BEGIN\",\n  \"savepoints_count\": 1,\n  \"savepoint_backtraces\": [\n    [\n      \"app/models/application_record.rb:75:in `block in safe_find_or_create_by'\",\n      \"app/models/application_record.rb:75:in `safe_find_or_create_by'\",\n      \"app/models/merge_request.rb:1859:in `ensure_metrics'\",\n      \"ee/lib/analytics/merge_request_metrics_refresh.rb:11:in `block in execute'\",\n      \"ee/lib/analytics/merge_request_metrics_refresh.rb:10:in `each'\",\n      \"ee/lib/analytics/merge_request_metrics_refresh.rb:10:in `execute'\",\n      \"ee/app/services/ee/merge_requests/approval_service.rb:57:in `calculate_approvals_metrics'\",\n      \"ee/app/services/ee/merge_requests/approval_service.rb:45:in `block in create_event'\",\n      \"ee/app/services/ee/merge_requests/approval_service.rb:43:in `create_event'\",\n      \"app/services/merge_requests/approval_service.rb:13:in `execute'\",\n      \"ee/app/services/ee/merge_requests/approval_service.rb:14:in `execute'\",\n      \"lib/api/merge_request_approvals.rb:58:in `block (3 levels) in \u003Cclass:MergeRequestApprovals>'\",\n    ]\n  \"rollbacks_count\": 0,\n  \"releases_count\": 1\n}\n\n```\n\n\nThis log message contains not only the number of subtransactions via\n\n`savepoints_count`, but it also contains a handy backtrace that\n\nidentifies the exact source of the problem. The `sql` field also\n\ncontains [Marginalia comments](https://github.com/basecamp/marginalia)\n\nthat we tack onto every SQL query. These comments make it possible to\n\nidentify what HTTP request initiated the SQL query.\n\n\n### Taking a hard look at PostgreSQL\n\n\nThe new instrumentation showed that while the application regularly used\n\nsubtransactions, it never exceeded 10 nested `SAVEPOINT` calls.\n\n\nMeanwhile, [Nikolay Samokhvalov](https://gitlab.com/NikolayS), founder\n\nof [Postgres.ai](https://postgres.ai/), performed a battery of tests [trying\nto replicate the\nproblem](https://gitlab.com/postgres-ai/postgresql-consulting/tests-and-benchmarks/-/issues/20).\n\nHe replicated Laurenz's results when a single transaction exceeded 64\n\nsubtransactions, but that wasn't happening here.\n\n\nWhen the database stalls occurred, we observed a number of patterns:\n\n\n1. Only the replicas were affected; the primary remained unaffected.\n\n1. There was a long-running transaction, usually relating to\n\nPostgreSQL's autovacuuming, during the time. The stalls stopped quickly\nafter the transaction ended.\n\n\nWhy would this matter? Analyzing the PostgreSQL source code, Senior\n\nSupport Engineer [Catalin Irimie](https://gitlab.com/cat) [posed an\n\nintriguing question that led to a breakthrough in our\nunderstanding](https://gitlab.com/gitlab-org/gitlab/-/issues/338410#note_652056284):\n\n\n> Does this mean that, having subtransactions spanning more than 32 cache\npages, concurrently, would trigger the exclusive SubtransControlLock because\nwe still end up reading them from the disk?\n\n\n### Reproducing the problem with replicas\n\n\nTo answer this, Nikolay immediately modified his test [to involve replicas\nand long-running\ntransactions](https://gitlab.com/postgres-ai/postgresql-consulting/tests-and-benchmarks/-/issues/21#note_653453774).\nWithin a day, he reproduced the problem:\n\n\n![Nikolay\nexperiment](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/nikolay-experiment.png)\n\n\nThe image above shows that transaction rates remain steady around\n\n360,000 transactions per second (TPS). Everything was proceeding fine\n\nuntil the long-running transaction started on the primary. Then suddenly\n\nthe transaction rates plummeted to 50,000 TPS on the replicas. Canceling\n\nthe long transaction immediately caused the transaction rate to return.\n\n\n### What is going on here?\n\n\nIn his blog post, Nikolay called the problem [Subtrans SLRU\noverflow](https://v2.postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful#problem-4-subtrans-slru-overflow).\n\nIn a busy database, it's possible for the size of the subtransaction log\n\nto grow so large that the working set no longer fits into memory. This\n\nresults in a lot of cache misses, which in turn causes a high amount of\n\ndisk I/O and CPU as PostgreSQL furiously tries to load data from disk to\n\nkeep up with all the lookups.\n\n\nAs mentioned earlier, the subtransaction cache holds a mapping of the\n\nsubXID to the parent XID. When PostgreSQL needs to look up the subXID,\n\nit calculates in which memory page this ID would live, and then does a\n\nlinear search to find in the memory page. If the page is not in the\n\ncache, it evicts one page and loads the desired one into memory. The\n\ndiagram below shows the memory layout of the subtransaction SLRU.\n\n\n![Subtrans\nSLRU](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/subtrans-slru.png)\n\n\nBy default, each SLRU page is an 8K buffer holding 4-byte parent\n\nXIDs. This means 8192/4 = 2048 transaction IDs can be stored in each\n\npage.\n\n\nNote that there may be gaps in each page. PostgreSQL will cache XIDs as\n\nneeded, so a single XID can occupy an entire page.\n\n\nThere are 32 (`NUM_SUBTRANS_BUFFERS`) pages, which means up to 65K\n\ntransaction IDs can be stored in memory. Nikolay demonstrated that in a\n\nbusy system, it took about 18 seconds to fill up all 65K entries. Then\n\nperformance dropped off a cliff, making the database replicas unusable.\n\n\nTo our surprise, our experiments also demonstrated that a single\n\n`SAVEPOINT` during a long-transaction [could initiate this problem if\n\nmany writes also occurred\nsimultaneously](https://gitlab.com/gitlab-org/gitlab/-/issues/338865#note_655312474).\nThat\n\nis, it wasn't enough just to reduce the frequency of `SAVEPOINT`; we had\n\nto eliminate them completely.\n\n\n#### Why does a single `SAVEPOINT` cause problems?\n\n\nTo answer this question, we need to understand what happens when a\n\n`SAVEPOINT` occurs in one query while a long-running transaction is\n\nrunning.\n\n\nWe mentioned earlier that PostgreSQL needs to decide whether a given row\n\nis visible to support a feature called [multi-version concurrency\ncontrol](https://www.postgresql.org/docs/current/mvcc.html), or MVCC for\n\nshort. It does this by storing hidden columns, `xmin` and `xmax`, in\n\neach tuple.\n\n\n`xmin` holds the XID of when the tuple was created, and `xmax` holds the\n\nXID when it was marked as dead (0 if the row is still present). In\n\naddition, at the beginning of a transaction, PostgreSQL records metadata\n\nin a database snapshot. Among other items, this snapshot records the\n\noldest XID and the newest XID in its own `xmin` and `xmax` values.\n\n\nThis metadata helps [PostgreSQL determine whether a tuple is\nvisible](https://www.interdb.jp/pg/pgsql05.html).\n\nFor example, a committed XID that started before `xmin` is definitely\n\nvisible, while anything after `xmax` is invisible.\n\n\n### What does this have to do with long transactions?\n\n\nLong transactions are bad in general because they can tie up\n\nconnections, but they can cause a subtly different problem on a\n\nreplica. On the replica, a single `SAVEPOINT` during a long transaction\n\ncauses a snapshot to suboverflow. Remember that dragged down performance\n\nin the case where we had more than 64 subtransactions.\n\n\nFundamentally, the problem happens because a replica behaves differently\n\nfrom a primary when creating snapshots and checking for tuple\n\nvisibility. The diagram below illustrates an example with some of the\n\ndata structures used in PostgreSQL:\n\n\n![Diagram of subtransaction handling in\nreplicas](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/pg-replica-subtransaction-diagram.png)\n\n\nOn the top of this diagram, we can see the XIDs increase at the\n\nbeginning of a subtransaction: the `INSERT` after the `BEGIN` gets 1,\n\nand the subsequent `INSERT` in `SAVEPOINT` gets 2. Another client comes\n\nalong and performs a `INSERT` and `SELECT` at XID 3.\n\n\nOn the primary, PostgreSQL stores the transactions in progress in a\n\nshared memory segment. The process array (`procarray`) stores XID 1 with\n\nthe first connection, and the database also writes that information to\n\nthe `pg_xact` directory. XID 2 gets stored in the `pg_subtrans`\n\ndirectory, mapped to its parent, XID 1.\n\n\nIf a read happens on the primary, the snapshot generated contains `xmin`\n\nas 1, and `xmax` as 3. `txip` holds a list of transactions in progress,\n\nand `subxip` holds a list of subtransactions in progress.\n\n\nHowever, neither the `procarray` nor the snapshot are shared directly\n\nwith the replica. The replica receives all the data it needs from the\n\nwrite-ahead log (WAL).\n\n\nPlaying the WAL back one entry at time, the replica populates a shared data\n\nstructure called `KnownAssignedIds`. It contains all the transactions in\n\nprogress on the primary. Since this structure can only hold a limited number\nof\n\nIDs, a busy database with a lot of active subtransactions could easily fill\n\nthis buffer. PostgreSQL made a design choice to kick out all subXIDs from\nthis\n\nlist and store them in the `pg_subtrans` directory.\n\n\nWhen a snapshot is generated on the replica, notice how `txip` is\n\nblank. A PostgreSQL replica treats **all** XIDs as though they are\n\nsubtransactions and throws them into the `subxip` bucket. That works\n\nbecause if a XID has a parent XID, then it's a subtransaction. Otherwise,\nit's a normal transaction. [The code comments\n\nexplain the\nrationale](https://gitlab.com/postgres/postgres/blob/9f540f840665936132dd30bd8e58e9a67e648f22/src/backend/storage/ipc/procarray.c#L1665-L1681).\n\n\nHowever, this means the snapshot is missing subXIDs, and that could be\n\nbad for MVCC. To deal with that, the [replica also updates\n`lastOverflowedXID`](https://gitlab.com/postgres/postgres/blob/9f540f840665936132dd30bd8e58e9a67e648f22/src/backend/storage/ipc/procarray.c#L3176-L3182):\n\n\n```c\n * When we throw away subXIDs from KnownAssignedXids, we need to keep track of\n * that, similarly to tracking overflow of a PGPROC's subxids array.  We do\n * that by remembering the lastOverflowedXID, ie the last thrown-away subXID.\n * As long as that is within the range of interesting XIDs, we have to assume\n * that subXIDs are missing from snapshots.  (Note that subXID overflow occurs\n * on primary when 65th subXID arrives, whereas on standby it occurs when 64th\n * subXID arrives - that is not an error.)\n```\n\n\nWhat is this \"range of interesting XIDs\"? We can see this in [the code\nbelow](https://gitlab.com/postgres/postgres/blob/4bf0bce161097869be5a56706b31388ba15e0113/src/backend/storage/ipc/procarray.c#L1702-L1703):\n\n\n```c\n\nif (TransactionIdPrecedesOrEquals(xmin, procArray->lastOverflowedXid))\n    suboverflowed = true;\n```\n\n\nIf `lastOverflowedXid` is smaller than our snapshot's `xmin`, it means\n\nthat all subtransactions have completed, so we don't need to check for\n\nsubtransactions. However, in our example:\n\n\n1. `xmin` is 1 because of the transaction.\n\n2. `lastOverflowXid` is 2 because of the `SAVEPOINT`.\n\n\nThis means `suboverflowed` is set to `true` here, which tells PostgreSQL\n\nthat whenever a XID needs to be checked, check to see if it has a parent\n\nXID. Remember that this causes PostgreSQL to:\n\n\n1. Look up the subXID for the parent XID in the SLRU cache.\n\n1. If this doesn't exist in the cache, fetch the data from `pg_trans`.\n\n\nIn a busy system, the requested XIDs could span an ever-growing range of\n\nvalues, which could easily exhaust the 64K entries in the SLRU\n\ncache. This range will continue to grow as long as the transaction runs;\n\nthe rate of increase depends on how many updates are happening on the\n\nprmary. As soon as the transaction terminates, the `suboverflowed` state\n\ngets set to `false`.\n\n\nIn other words, we've replicated the same conditions as we saw with 64\n\nsubtransactions, only with a single `SAVEPOINT` and a long transaction.\n\n\n### What can we do about getting rid of Nessie?\n\n\nThere are three options:\n\n\n1. Eliminate `SAVEPOINT` calls completely.\n\n1. Eliminate all long-running transactions.\n\n1. Apply [Andrey Borodin's patches to PostgreSQL and increase the\nsubtransaction\ncache](https://www.postgresql.org/message-id/flat/494C5E7F-E410-48FA-A93E-F7723D859561%40yandex-team.ru#18c79477bf7fc44a3ac3d1ce55e4c169).\n\n\nWe chose the first option because most uses of subtransaction could be\n\nremoved fairly easily. There were a [number of\napproaches](https://gitlab.com/groups/gitlab-org/-/epics/6540) we took:\n\n\n1. Perform updates outside of a subtransaction. Examples:\n[1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68471),\n[2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68690)\n\n1. Rewrite a query to use a `INSERT` or an `UPDATE` with an `ON CONFLICT`\nclause to deal with duplicate constraint violations. Examples:\n[1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68433),\n[2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69240),\n[3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68509)\n\n1. Live with a non-atomic `find_or_create_by`. We used this approach\nsparingly. Example:\n[1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68649)\n\n\nIn addition, we added [an alert whenever the application used a a single\n`SAVEPOINT`](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/3881):\n\n\n![subtransaction\nalert](https://about.gitlab.com/images/blogimages/postgresql-subtransactions/subtransactions-alert-example.png)\n\n\nThis had the side benefit of flagging a [minor\nbug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70889).\n\n\n#### Why not eliminate all long-running transactions?\n\n\nIn our database, it wasn't practical to eliminate all long-running\n\ntransactions because we think many of them happened via [database\n\nautovacuuming](https://www.postgresql.org/docs/current/runtime-config-autovacuum.html),\n\nbut [we're not able to reproduce this\nyet](https://gitlab.com/postgres-ai/postgresql-consulting/tests-and-benchmarks/-/issues/21#note_669698320).\n\nWe are working on partitioning the tables and sharding the database, but\nthis is a much more time-consuming problem\n\nthan removing all subtransactions.\n\n\n#### What about the PostgreSQL patches?\n\n\nAlthough we tested Andrey's PostgreSQL patches, we did not feel comfortable\n\ndeviating from the official PostgreSQL releases. Plus, maintaining a\n\ncustom patched release over upgrades would add a significant maintenance\n\nburden for our infrastructure team. Our self-managed customers would\n\nalso not benefit unless they used a patched database.\n\n\nAndrey's patches do two main things:\n\n\n1. Allow administrators to change the SLRU size to any value.\n\n1. Adds an [associative cache](https://www.youtube.com/watch?v=A0vR-ks3hsQ).\n\nto make it performant to use a large cache value.\n\n\nRemember that the SLRU cache does a linear search for the desired\n\npage. That works fine when there are only 32 pages to search, but if you\n\nincrease the cache size to 100 MB the search becomes much more\n\nexpensive. The associative cache makes the lookup fast by indexing pages\n\nwith a bitmask and looking up the entry with offsets from the remaining\n\nbits. This mitigates the problem because a transaction would need to be\n\nseveral magnitudes longer to cause a problem.\n\n\nNikolay demonstrated that the `SAVEPOINT` problem disappeared as soon as\n\nwe increased the SLRU size to 100 MB with those patches. With a 100 MB\n\ncache, PostgreSQL can cache 26.2 million IDs (104857600/4), far more\n\nthan the measely 65K.\n\n\nThese [patches are currently awaiting\nreview](https://postgres.ai/blog/20210831-postgresql-subtransactions-considered-harmful#ideas-for-postgresql-development),\n\nbut in our opinion they should be given high priority for PostgreSQL 15.\n\n\n### Conclusion\n\n\nSince removing all `SAVEPOINT` queries, we have not seen Nessie rear her\n\nhead again. If you are running PostgreSQL with read replicas, we\n\nstrongly recommend that you also remove *all* subtransactions until\n\nfurther notice.\n\n\nPostgreSQL is a fantastic database, and its well-commented code makes it\n\npossible to understand its limitations under different configurations.\n\n\nWe would like to thank the GitLab community for bearing with us while we\n\niron out this production issue.\n\n\nWe are also grateful for the support from [Nikolay\n\nSamokhvalov](https://gitlab.com/NikolayS) and [Catalin\n\nIrimie](https://gitlab.com/cat), who contributed to understanding where our\n\nLoch Ness Monster was hiding.\n\n\nCover image by [Khadi\nGaniev](https://www.istockphoto.com/portfolio/Ganiev?mediatype=photography)\non [iStock](https://istock.com), licensed under [standard\nlicense](https://www.istockphoto.com/legal/license-agreement)\n","engineering",[24,25,26],"performance","contributors","frontend",{"slug":28,"featured":6,"template":29},"why-we-spent-the-last-month-eliminating-postgresql-subtransactions","BlogPost","content:en-us:blog:why-we-spent-the-last-month-eliminating-postgresql-subtransactions.yml","yaml","Why We Spent The Last Month Eliminating Postgresql Subtransactions","content","en-us/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions.yml","en-us/blog/why-we-spent-the-last-month-eliminating-postgresql-subtransactions","yml",{"_path":38,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"data":40,"_id":461,"_type":31,"title":462,"_source":33,"_file":463,"_stem":464,"_extension":36},"/shared/en-us/main-navigation","en-us",{"logo":41,"freeTrial":46,"sales":51,"login":56,"items":61,"search":392,"minimal":423,"duo":442,"pricingDeployment":451},{"config":42},{"href":43,"dataGaName":44,"dataGaLocation":45},"/","gitlab logo","header",{"text":47,"config":48},"Get free trial",{"href":49,"dataGaName":50,"dataGaLocation":45},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":52,"config":53},"Talk to sales",{"href":54,"dataGaName":55,"dataGaLocation":45},"/sales/","sales",{"text":57,"config":58},"Sign in",{"href":59,"dataGaName":60,"dataGaLocation":45},"https://gitlab.com/users/sign_in/","sign in",[62,106,203,208,313,373],{"text":63,"config":64,"cards":66,"footer":89},"Platform",{"dataNavLevelOne":65},"platform",[67,73,81],{"title":63,"description":68,"link":69},"The most comprehensive AI-powered DevSecOps Platform",{"text":70,"config":71},"Explore our Platform",{"href":72,"dataGaName":65,"dataGaLocation":45},"/platform/",{"title":74,"description":75,"link":76},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":77,"config":78},"Meet GitLab Duo",{"href":79,"dataGaName":80,"dataGaLocation":45},"/gitlab-duo/","gitlab duo ai",{"title":82,"description":83,"link":84},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":85,"config":86},"Learn more",{"href":87,"dataGaName":88,"dataGaLocation":45},"/why-gitlab/","why gitlab",{"title":90,"items":91},"Get started with",[92,97,102],{"text":93,"config":94},"Platform Engineering",{"href":95,"dataGaName":96,"dataGaLocation":45},"/solutions/platform-engineering/","platform engineering",{"text":98,"config":99},"Developer Experience",{"href":100,"dataGaName":101,"dataGaLocation":45},"/developer-experience/","Developer experience",{"text":103,"config":104},"MLOps",{"href":105,"dataGaName":103,"dataGaLocation":45},"/topics/devops/the-role-of-ai-in-devops/",{"text":107,"left":108,"config":109,"link":111,"lists":115,"footer":185},"Product",true,{"dataNavLevelOne":110},"solutions",{"text":112,"config":113},"View all Solutions",{"href":114,"dataGaName":110,"dataGaLocation":45},"/solutions/",[116,141,164],{"title":117,"description":118,"link":119,"items":124},"Automation","CI/CD and automation to accelerate deployment",{"config":120},{"icon":121,"href":122,"dataGaName":123,"dataGaLocation":45},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[125,129,133,137],{"text":126,"config":127},"CI/CD",{"href":128,"dataGaLocation":45,"dataGaName":126},"/solutions/continuous-integration/",{"text":130,"config":131},"AI-Assisted Development",{"href":79,"dataGaLocation":45,"dataGaName":132},"AI assisted development",{"text":134,"config":135},"Source Code Management",{"href":136,"dataGaLocation":45,"dataGaName":134},"/solutions/source-code-management/",{"text":138,"config":139},"Automated Software Delivery",{"href":122,"dataGaLocation":45,"dataGaName":140},"Automated software delivery",{"title":142,"description":143,"link":144,"items":149},"Security","Deliver code faster without compromising security",{"config":145},{"href":146,"dataGaName":147,"dataGaLocation":45,"icon":148},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[150,154,159],{"text":151,"config":152},"Application Security Testing",{"href":146,"dataGaName":153,"dataGaLocation":45},"Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":45,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":45},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":45},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":45,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":45,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":45,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":45,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":45,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":45,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":45,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":45},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":45},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":45},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":45,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":45},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":45},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":45},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":45},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":45},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":45},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":45},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":45},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":45},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":45},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":45},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":45},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":45},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":45},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":45},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":45},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":45},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":45},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":45},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":45},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":45},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":45},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":45},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":45},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":52,"config":380},{"href":54,"dataGaName":381,"dataGaLocation":45},"talk to sales",{"text":383,"config":384},"Support portal",{"href":385,"dataGaName":386,"dataGaLocation":45},"https://support.gitlab.com","support portal",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":45},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":59,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":74,"config":405},{"href":79,"dataGaName":74,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":126,"config":411},{"href":128,"dataGaName":126,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":87,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":50,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":79,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},{"freeTrial":452,"mobileIcon":457,"desktopIcon":459},{"text":453,"config":454},"Back to pricing",{"href":206,"dataGaName":455,"dataGaLocation":428,"icon":456},"back to pricing","GoBack",{"altText":430,"config":458},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":460},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":466,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"title":467,"button":468,"image":473,"config":477,"_id":479,"_type":31,"_source":33,"_file":480,"_stem":481,"_extension":36},"/shared/en-us/banner","is now in public beta!",{"text":469,"config":470},"Try the Beta",{"href":471,"dataGaName":472,"dataGaLocation":45},"/gitlab-duo/agent-platform/","duo banner",{"altText":474,"config":475},"GitLab Duo Agent Platform",{"src":476},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":478},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":483,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"data":484,"_id":723,"_type":31,"title":724,"_source":33,"_file":725,"_stem":726,"_extension":36},"/shared/en-us/main-footer",{"text":485,"source":486,"edit":492,"contribute":497,"config":502,"items":507,"minimal":715},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":487,"config":488},"View page source",{"href":489,"dataGaName":490,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":493,"config":494},"Edit this page",{"href":495,"dataGaName":496,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":498,"config":499},"Please contribute",{"href":500,"dataGaName":501,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":503,"facebook":504,"youtube":505,"linkedin":506},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[508,555,608,652,681],{"title":204,"links":509,"subMenu":524},[510,514,519],{"text":511,"config":512},"View plans",{"href":206,"dataGaName":513,"dataGaLocation":491},"view plans",{"text":515,"config":516},"Why Premium?",{"href":517,"dataGaName":518,"dataGaLocation":491},"/pricing/premium/","why premium",{"text":520,"config":521},"Why Ultimate?",{"href":522,"dataGaName":523,"dataGaLocation":491},"/pricing/ultimate/","why ultimate",[525],{"title":526,"links":527},"Contact Us",[528,531,533,535,540,545,550],{"text":529,"config":530},"Contact sales",{"href":54,"dataGaName":55,"dataGaLocation":491},{"text":383,"config":532},{"href":385,"dataGaName":386,"dataGaLocation":491},{"text":388,"config":534},{"href":390,"dataGaName":391,"dataGaLocation":491},{"text":536,"config":537},"Status",{"href":538,"dataGaName":539,"dataGaLocation":491},"https://status.gitlab.com/","status",{"text":541,"config":542},"Terms of use",{"href":543,"dataGaName":544,"dataGaLocation":491},"/terms/","terms of use",{"text":546,"config":547},"Privacy statement",{"href":548,"dataGaName":549,"dataGaLocation":491},"/privacy/","privacy statement",{"text":551,"config":552},"Cookie preferences",{"dataGaName":553,"dataGaLocation":491,"id":554,"isOneTrustButton":108},"cookie preferences","ot-sdk-btn",{"title":107,"links":556,"subMenu":564},[557,561],{"text":558,"config":559},"DevSecOps platform",{"href":72,"dataGaName":560,"dataGaLocation":491},"devsecops platform",{"text":130,"config":562},{"href":79,"dataGaName":563,"dataGaLocation":491},"ai-assisted development",[565],{"title":566,"links":567},"Topics",[568,573,578,583,588,593,598,603],{"text":569,"config":570},"CICD",{"href":571,"dataGaName":572,"dataGaLocation":491},"/topics/ci-cd/","cicd",{"text":574,"config":575},"GitOps",{"href":576,"dataGaName":577,"dataGaLocation":491},"/topics/gitops/","gitops",{"text":579,"config":580},"DevOps",{"href":581,"dataGaName":582,"dataGaLocation":491},"/topics/devops/","devops",{"text":584,"config":585},"Version Control",{"href":586,"dataGaName":587,"dataGaLocation":491},"/topics/version-control/","version control",{"text":589,"config":590},"DevSecOps",{"href":591,"dataGaName":592,"dataGaLocation":491},"/topics/devsecops/","devsecops",{"text":594,"config":595},"Cloud Native",{"href":596,"dataGaName":597,"dataGaLocation":491},"/topics/cloud-native/","cloud native",{"text":599,"config":600},"AI for Coding",{"href":601,"dataGaName":602,"dataGaLocation":491},"/topics/devops/ai-for-coding/","ai for coding",{"text":604,"config":605},"Agentic AI",{"href":606,"dataGaName":607,"dataGaLocation":491},"/topics/agentic-ai/","agentic ai",{"title":609,"links":610},"Solutions",[611,613,615,620,624,627,631,634,636,639,642,647],{"text":151,"config":612},{"href":146,"dataGaName":151,"dataGaLocation":491},{"text":140,"config":614},{"href":122,"dataGaName":123,"dataGaLocation":491},{"text":616,"config":617},"Agile development",{"href":618,"dataGaName":619,"dataGaLocation":491},"/solutions/agile-delivery/","agile delivery",{"text":621,"config":622},"SCM",{"href":136,"dataGaName":623,"dataGaLocation":491},"source code management",{"text":569,"config":625},{"href":128,"dataGaName":626,"dataGaLocation":491},"continuous integration & delivery",{"text":628,"config":629},"Value stream management",{"href":179,"dataGaName":630,"dataGaLocation":491},"value stream management",{"text":574,"config":632},{"href":633,"dataGaName":577,"dataGaLocation":491},"/solutions/gitops/",{"text":189,"config":635},{"href":191,"dataGaName":192,"dataGaLocation":491},{"text":637,"config":638},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":491},{"text":640,"config":641},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":491},{"text":643,"config":644},"Education",{"href":645,"dataGaName":646,"dataGaLocation":491},"/solutions/education/","education",{"text":648,"config":649},"Financial services",{"href":650,"dataGaName":651,"dataGaLocation":491},"/solutions/finance/","financial services",{"title":209,"links":653},[654,656,658,660,663,665,667,669,671,673,675,677,679],{"text":221,"config":655},{"href":223,"dataGaName":224,"dataGaLocation":491},{"text":226,"config":657},{"href":228,"dataGaName":229,"dataGaLocation":491},{"text":231,"config":659},{"href":233,"dataGaName":234,"dataGaLocation":491},{"text":236,"config":661},{"href":238,"dataGaName":662,"dataGaLocation":491},"docs",{"text":259,"config":664},{"href":261,"dataGaName":5,"dataGaLocation":491},{"text":254,"config":666},{"href":256,"dataGaName":257,"dataGaLocation":491},{"text":263,"config":668},{"href":265,"dataGaName":266,"dataGaLocation":491},{"text":276,"config":670},{"href":278,"dataGaName":279,"dataGaLocation":491},{"text":268,"config":672},{"href":270,"dataGaName":271,"dataGaLocation":491},{"text":281,"config":674},{"href":283,"dataGaName":284,"dataGaLocation":491},{"text":286,"config":676},{"href":288,"dataGaName":289,"dataGaLocation":491},{"text":291,"config":678},{"href":293,"dataGaName":294,"dataGaLocation":491},{"text":296,"config":680},{"href":298,"dataGaName":299,"dataGaLocation":491},{"title":314,"links":682},[683,685,687,689,691,693,695,699,704,706,708,710],{"text":321,"config":684},{"href":323,"dataGaName":316,"dataGaLocation":491},{"text":326,"config":686},{"href":328,"dataGaName":329,"dataGaLocation":491},{"text":334,"config":688},{"href":336,"dataGaName":337,"dataGaLocation":491},{"text":339,"config":690},{"href":341,"dataGaName":342,"dataGaLocation":491},{"text":344,"config":692},{"href":346,"dataGaName":347,"dataGaLocation":491},{"text":349,"config":694},{"href":351,"dataGaName":352,"dataGaLocation":491},{"text":696,"config":697},"Sustainability",{"href":698,"dataGaName":696,"dataGaLocation":491},"/sustainability/",{"text":700,"config":701},"Diversity, inclusion and belonging (DIB)",{"href":702,"dataGaName":703,"dataGaLocation":491},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":705},{"href":356,"dataGaName":357,"dataGaLocation":491},{"text":364,"config":707},{"href":366,"dataGaName":367,"dataGaLocation":491},{"text":369,"config":709},{"href":371,"dataGaName":372,"dataGaLocation":491},{"text":711,"config":712},"Modern Slavery Transparency Statement",{"href":713,"dataGaName":714,"dataGaLocation":491},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":716},[717,719,721],{"text":541,"config":718},{"href":543,"dataGaName":544,"dataGaLocation":491},{"text":546,"config":720},{"href":548,"dataGaName":549,"dataGaLocation":491},{"text":551,"config":722},{"dataGaName":553,"dataGaLocation":491,"id":554,"isOneTrustButton":108},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[728,740],{"_path":729,"_dir":730,"_draft":6,"_partial":6,"_locale":7,"content":731,"config":735,"_id":737,"_type":31,"title":18,"_source":33,"_file":738,"_stem":739,"_extension":36},"/en-us/blog/authors/grzegorz-bizon","authors",{"name":18,"config":732},{"headshot":733,"ctfId":734},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659488/Blog/Author%20Headshots/gitlab-logo-extra-whitespace.png","Grzegorz-Bizon",{"template":736},"BlogAuthor","content:en-us:blog:authors:grzegorz-bizon.yml","en-us/blog/authors/grzegorz-bizon.yml","en-us/blog/authors/grzegorz-bizon",{"_path":741,"_dir":730,"_draft":6,"_partial":6,"_locale":7,"content":742,"config":746,"_id":747,"_type":31,"title":19,"_source":33,"_file":748,"_stem":749,"_extension":36},"/en-us/blog/authors/stan-hu",{"name":19,"config":743},{"headshot":744,"ctfId":745},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659504/Blog/Author%20Headshots/stanhu-headshot.jpg","stanhu",{"template":736},"content:en-us:blog:authors:stan-hu.yml","en-us/blog/authors/stan-hu.yml","en-us/blog/authors/stan-hu",{"_path":751,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"header":752,"eyebrow":753,"blurb":754,"button":755,"secondaryButton":759,"_id":761,"_type":31,"title":762,"_source":33,"_file":763,"_stem":764,"_extension":36},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":47,"config":756},{"href":757,"dataGaName":50,"dataGaLocation":758},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":52,"config":760},{"href":54,"dataGaName":55,"dataGaLocation":758},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":766,"content":767,"config":770,"_id":30,"_type":31,"title":32,"_source":33,"_file":34,"_stem":35,"_extension":36},{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},{"title":9,"description":10,"authors":768,"heroImage":11,"date":20,"body":21,"category":22,"tags":769},[18,19],[24,25,26],{"slug":28,"featured":6,"template":29},1761814423348]