[{"data":1,"prerenderedAt":760},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change":3,"navigation-en-us":36,"banner-en-us":464,"footer-en-us":481,"Igor Wiedler":726,"next-steps-en-us":739,"footer-source-/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":754},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE's HAProxy Config Change: An Unexpected Journey","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...",[19],"Igor Wiedler","2021-01-14","This blog post was originally published on the GitLab Unfiltered\nblog. It was reviewed and republished on\n2021-02-12.\n\n{: .note .alert-info .text-center}\n\n\n## TL;DR\n\n\n- HAProxy has a `server-state-file` directive that persists some of its\nstate across restarts.\n\n- This state file contains the port of each backend server.\n\n- If an `haproxy.cfg` change modifies the port, the new port will be\noverwritten with the previous one from the state file.\n\n- A workaround is to change the backend server name, so that it is\nconsidered to be a separate server that does not match what is in the state\nfile.\n\n- This has implications for the rollout procedure we use on HAProxy.\n\n\n## Background\n\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\n\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\n\nThe rollout to staging involves changing the request flow from TCP\nproxying...\n\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n\n... to using the [PROXY\nprotocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n\n`fe-pages-01-lb-gstg` (note the port change):\n\n```diff\n\n-    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n\n## The brokenness\n\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests\nto\n\nthat node start failing.\n\n\nBy retrying a few times via `curl` on the command line, we see this error:\n\n```\n\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n\n*   Trying 35.229.69.78...\n\n* TCP_NODELAY set\n\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n\n* ALPN, offering h2\n\n* ALPN, offering http/1.1\n\n* successfully set certificate verify locations:\n\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n* Closing connection 0\n\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n```\n\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\n\nconnection. It turns out that `LibreSSL` does not give us much insight into\nthe\n\nunderlying issue here.\n\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n\n```\n\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n\n```\n\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and\npull\n\nthe dump down for further analysis. That `pcap` file can be opened in\nWireshark, and this allows the data to be\n\nexplored and filtered interactively. Here, the first really surprising thing\nhappens:\n\n\n**We do not see any traffic on port 2443.**\n\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to\nlook at what underlies the LibreSSL error, and what we find\n\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a\nTCP SYN/ACK, establishing the connection. Followed by the client\n\nsending a TLS \"hello\". After which the server closes the connection with a\nFIN.\n\n\nIn other words, the server is closing the connection on the client.\n\n\n## The early hypotheses\n\n\nSo here come the usual suspects:\n\n\n* Did we modify the correct place in the config file?\n\n* Did we catch all places we need to update in the config?\n\n* Did the HAProxy process parse th econfig successfully?\n\n* Did HAProxy actually reload?\n\n* Is there a difference between reload and restart?\n\n* Did we modify the correct config file?\n\n* Are there old lingering HAProxy processes on the box?\n\n* Are we actually sending traffic to this node?\n\n* Are backend health checks failing?\n\n* Is there anything in the HAProxy logs?\n\n\nNone of these gave any insights whatsoever.\n\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with\na\n\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise,\nthis\n\nworked correctly. I tested with different HAProxy versions. It worked\nlocally, but not on\n\n`fe-pages-01`.\n\n\nAt this point I'm stumped. The local config is not identical to gstg, but\nquite\n\nsimilar. What could possibly be the difference?\n\n\n## Digging deeper\n\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help\nwith the investigation.\n\n\nWe started off by repeating the experiment. We saw the same results:\n\n\n* Server closes connection after client sends TLS hello\n\n* No traffic from fe-pages to web-pages on port 2443\n\n* Traffic from fe-pages to web-pages on port 1443\n\n\nThe first lead was to look at the packets going to port 1443. What do they\n\ncontain? We see this:\n\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the\nstream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){:\n.shadow.center}\n\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the\nstream\n\n{: .note.text-center}\n\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the\nclient sent. And before that there is some really weird preamble:\n\n\n```\n\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n\n```\n\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the\nword\n\n\"QUIT.\" Nothing.\n\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\n\nreveals some hits, but none that explain this.\n\n\nSo this is a mystery. We leave it for now, and probe in a different\ndirection.\n\n\n## Honing in\n\n\nHow come we are sending traffic to port 1443, when that port is not\nmentioned in\n\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\n\nI suggested running `strace` on HAProxy startup, so that we can see which\nfiles\n\nare being `open`ed. This is a bit tricky to do though, because the process\nis\n\nsystemd-managed.\n\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc),\nwe\n\ncan actually listen on open events system-wide using the wonderful\n\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py).\nSo we run `opensnoop` and restart `haproxy`, and this is what we see,\nhighlighting the relevant bit:\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo\n/usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n\n...\n\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n\n...\n\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n\n...\n\n24.118333000  16702  haproxy             4   0\n/etc/haproxy/cloudflare_ips_v4.lst\n\n...\n\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n\n```\n\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly\nsuspicious.\n\nWhat could it possibly be? Let's see what this file contains.\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat\n/etc/haproxy/state/global\n\n\n1\n\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state\nsrv_uweight srv_iweight srv_time_since_last_change srv_check_status\nsrv_check_result srv_check_health srv_check_state srv_agent_state\nbk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0\n0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0\n0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n```\n\n\nIt appears we are storing some metadata for each backend server, including\nits old port number!\n\n\nNow, looking again in `haproxy.cfg`, we see:\n\n```\n\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\n\nSo we are using the\n\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\n\ndirective. This will persist server state across HAProxy restarts. That is\n\nuseful to keep metadata consistent, such as whether a server was marked as\n\nMAINT.\n\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\n\nThe suspected behavior is:\n\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`,\n`2443`, `send-proxy-v2`\n\n* HAProxy reload is initiated\n\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the\nold port of each backend server)\n\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new\nconfig: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the\nbackend server `web-pages-01-sv-gstg`, and overrides all values, including\nthe port!\n\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS\nport.\n\n\n## The workaround\n\n\nTo validate the theory and develop a potential workaround, we modify\n\n`haproxy.cfg` to use a different backend server name.\n\n\nThe new diff is:\n\n```diff\n\n-    server web-pages-01-sv-gstg        \nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg        \nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg-proxyv2\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg-proxyv2\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nWith this config change in place, we reload HAProxy and indeed, it is now\n\nserving traffic correctly. See [the merge request fixing\nit](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n\n## A follow-up on those `QUIT` bytes\n\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol?\nRemember, searching [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\n\nstring did not find any matches. However, Matt actually read the spec, and\nfound this section on version 2 of\n\nthe protocol:\n\n```\n\nThe binary header format starts with a constant 12 bytes block containing\nthe\n\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly\nless mnemonic than the header from text-based version 1 of the protocol:\n\n```\n\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\n\nWell, I suppose that explains it.\n\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[24,25],"production","inside GitLab",{"slug":27,"featured":6,"template":28},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":460,"_type":30,"title":461,"_source":32,"_file":462,"_stem":463,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":391,"minimal":422,"duo":441,"pricingDeployment":450},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,202,207,312,372],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":184},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,163],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[149,153,158],{"text":150,"config":151},"Application Security Testing",{"href":145,"dataGaName":152,"dataGaLocation":44},"Application security testing",{"text":154,"config":155},"Software Supply Chain Security",{"href":156,"dataGaLocation":44,"dataGaName":157},"/solutions/supply-chain/","Software supply chain security",{"text":159,"config":160},"Software Compliance",{"href":161,"dataGaName":162,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":164,"link":165,"items":170},"Measurement",{"config":166},{"icon":167,"href":168,"dataGaName":169,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[171,175,179],{"text":172,"config":173},"Visibility & Measurement",{"href":168,"dataGaLocation":44,"dataGaName":174},"Visibility and Measurement",{"text":176,"config":177},"Value Stream Management",{"href":178,"dataGaLocation":44,"dataGaName":176},"/solutions/value-stream-management/",{"text":180,"config":181},"Analytics & Insights",{"href":182,"dataGaLocation":44,"dataGaName":183},"/solutions/analytics-and-insights/","Analytics and insights",{"title":185,"items":186},"GitLab for",[187,192,197],{"text":188,"config":189},"Enterprise",{"href":190,"dataGaLocation":44,"dataGaName":191},"/enterprise/","enterprise",{"text":193,"config":194},"Small Business",{"href":195,"dataGaLocation":44,"dataGaName":196},"/small-business/","small business",{"text":198,"config":199},"Public Sector",{"href":200,"dataGaLocation":44,"dataGaName":201},"/solutions/public-sector/","public sector",{"text":203,"config":204},"Pricing",{"href":205,"dataGaName":206,"dataGaLocation":44,"dataNavLevelOne":206},"/pricing/","pricing",{"text":208,"config":209,"link":211,"lists":215,"feature":299},"Resources",{"dataNavLevelOne":210},"resources",{"text":212,"config":213},"View all resources",{"href":214,"dataGaName":210,"dataGaLocation":44},"/resources/",[216,249,271],{"title":217,"items":218},"Getting started",[219,224,229,234,239,244],{"text":220,"config":221},"Install",{"href":222,"dataGaName":223,"dataGaLocation":44},"/install/","install",{"text":225,"config":226},"Quick start guides",{"href":227,"dataGaName":228,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":230,"config":231},"Learn",{"href":232,"dataGaLocation":44,"dataGaName":233},"https://university.gitlab.com/","learn",{"text":235,"config":236},"Product documentation",{"href":237,"dataGaName":238,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":240,"config":241},"Best practice videos",{"href":242,"dataGaName":243,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":245,"config":246},"Integrations",{"href":247,"dataGaName":248,"dataGaLocation":44},"/integrations/","integrations",{"title":250,"items":251},"Discover",[252,257,261,266],{"text":253,"config":254},"Customer success stories",{"href":255,"dataGaName":256,"dataGaLocation":44},"/customers/","customer success stories",{"text":258,"config":259},"Blog",{"href":260,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":262,"config":263},"Remote",{"href":264,"dataGaName":265,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":267,"config":268},"TeamOps",{"href":269,"dataGaName":270,"dataGaLocation":44},"/teamops/","teamops",{"title":272,"items":273},"Connect",[274,279,284,289,294],{"text":275,"config":276},"GitLab Services",{"href":277,"dataGaName":278,"dataGaLocation":44},"/services/","services",{"text":280,"config":281},"Community",{"href":282,"dataGaName":283,"dataGaLocation":44},"/community/","community",{"text":285,"config":286},"Forum",{"href":287,"dataGaName":288,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":290,"config":291},"Events",{"href":292,"dataGaName":293,"dataGaLocation":44},"/events/","events",{"text":295,"config":296},"Partners",{"href":297,"dataGaName":298,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":300,"textColor":301,"text":302,"image":303,"link":307},"#2f2a6b","#fff","Insights for the future of software development",{"altText":304,"config":305},"the source promo card",{"src":306},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":308,"config":309},"Read the latest",{"href":310,"dataGaName":311,"dataGaLocation":44},"/the-source/","the source",{"text":313,"config":314,"lists":316},"Company",{"dataNavLevelOne":315},"company",[317],{"items":318},[319,324,330,332,337,342,347,352,357,362,367],{"text":320,"config":321},"About",{"href":322,"dataGaName":323,"dataGaLocation":44},"/company/","about",{"text":325,"config":326,"footerGa":329},"Jobs",{"href":327,"dataGaName":328,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":328},{"text":290,"config":331},{"href":292,"dataGaName":293,"dataGaLocation":44},{"text":333,"config":334},"Leadership",{"href":335,"dataGaName":336,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":338,"config":339},"Team",{"href":340,"dataGaName":341,"dataGaLocation":44},"/company/team/","team",{"text":343,"config":344},"Handbook",{"href":345,"dataGaName":346,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":348,"config":349},"Investor relations",{"href":350,"dataGaName":351,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":353,"config":354},"Trust Center",{"href":355,"dataGaName":356,"dataGaLocation":44},"/security/","trust center",{"text":358,"config":359},"AI Transparency Center",{"href":360,"dataGaName":361,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":363,"config":364},"Newsletter",{"href":365,"dataGaName":366,"dataGaLocation":44},"/company/contact/","newsletter",{"text":368,"config":369},"Press",{"href":370,"dataGaName":371,"dataGaLocation":44},"/press/","press",{"text":373,"config":374,"lists":375},"Contact us",{"dataNavLevelOne":315},[376],{"items":377},[378,381,386],{"text":51,"config":379},{"href":53,"dataGaName":380,"dataGaLocation":44},"talk to sales",{"text":382,"config":383},"Support portal",{"href":384,"dataGaName":385,"dataGaLocation":44},"https://support.gitlab.com","support portal",{"text":387,"config":388},"Customer portal",{"href":389,"dataGaName":390,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":392,"login":393,"suggestions":400},"Close",{"text":394,"link":395},"To search repositories and projects, login to",{"text":396,"config":397},"gitlab.com",{"href":58,"dataGaName":398,"dataGaLocation":399},"search login","search",{"text":401,"default":402},"Suggestions",[403,405,409,411,415,419],{"text":73,"config":404},{"href":78,"dataGaName":73,"dataGaLocation":399},{"text":406,"config":407},"Code Suggestions (AI)",{"href":408,"dataGaName":406,"dataGaLocation":399},"/solutions/code-suggestions/",{"text":125,"config":410},{"href":127,"dataGaName":125,"dataGaLocation":399},{"text":412,"config":413},"GitLab on AWS",{"href":414,"dataGaName":412,"dataGaLocation":399},"/partners/technology-partners/aws/",{"text":416,"config":417},"GitLab on Google Cloud",{"href":418,"dataGaName":416,"dataGaLocation":399},"/partners/technology-partners/google-cloud-platform/",{"text":420,"config":421},"Why GitLab?",{"href":86,"dataGaName":420,"dataGaLocation":399},{"freeTrial":423,"mobileIcon":428,"desktopIcon":433,"secondaryButton":436},{"text":424,"config":425},"Start free trial",{"href":426,"dataGaName":49,"dataGaLocation":427},"https://gitlab.com/-/trials/new/","nav",{"altText":429,"config":430},"Gitlab Icon",{"src":431,"dataGaName":432,"dataGaLocation":427},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":429,"config":434},{"src":435,"dataGaName":432,"dataGaLocation":427},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":437,"config":438},"Get Started",{"href":439,"dataGaName":440,"dataGaLocation":427},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":442,"mobileIcon":446,"desktopIcon":448},{"text":443,"config":444},"Learn more about GitLab Duo",{"href":78,"dataGaName":445,"dataGaLocation":427},"gitlab duo",{"altText":429,"config":447},{"src":431,"dataGaName":432,"dataGaLocation":427},{"altText":429,"config":449},{"src":435,"dataGaName":432,"dataGaLocation":427},{"freeTrial":451,"mobileIcon":456,"desktopIcon":458},{"text":452,"config":453},"Back to pricing",{"href":205,"dataGaName":454,"dataGaLocation":427,"icon":455},"back to pricing","GoBack",{"altText":429,"config":457},{"src":431,"dataGaName":432,"dataGaLocation":427},{"altText":429,"config":459},{"src":435,"dataGaName":432,"dataGaLocation":427},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":465,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":466,"button":467,"image":472,"config":476,"_id":478,"_type":30,"_source":32,"_file":479,"_stem":480,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":468,"config":469},"Try the Beta",{"href":470,"dataGaName":471,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":473,"config":474},"GitLab Duo Agent Platform",{"src":475},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":477},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":482,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":483,"_id":722,"_type":30,"title":723,"_source":32,"_file":724,"_stem":725,"_extension":35},"/shared/en-us/main-footer",{"text":484,"source":485,"edit":491,"contribute":496,"config":501,"items":506,"minimal":714},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":486,"config":487},"View page source",{"href":488,"dataGaName":489,"dataGaLocation":490},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":492,"config":493},"Edit this page",{"href":494,"dataGaName":495,"dataGaLocation":490},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":497,"config":498},"Please contribute",{"href":499,"dataGaName":500,"dataGaLocation":490},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":502,"facebook":503,"youtube":504,"linkedin":505},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[507,554,607,651,680],{"title":203,"links":508,"subMenu":523},[509,513,518],{"text":510,"config":511},"View plans",{"href":205,"dataGaName":512,"dataGaLocation":490},"view plans",{"text":514,"config":515},"Why Premium?",{"href":516,"dataGaName":517,"dataGaLocation":490},"/pricing/premium/","why premium",{"text":519,"config":520},"Why Ultimate?",{"href":521,"dataGaName":522,"dataGaLocation":490},"/pricing/ultimate/","why ultimate",[524],{"title":525,"links":526},"Contact Us",[527,530,532,534,539,544,549],{"text":528,"config":529},"Contact sales",{"href":53,"dataGaName":54,"dataGaLocation":490},{"text":382,"config":531},{"href":384,"dataGaName":385,"dataGaLocation":490},{"text":387,"config":533},{"href":389,"dataGaName":390,"dataGaLocation":490},{"text":535,"config":536},"Status",{"href":537,"dataGaName":538,"dataGaLocation":490},"https://status.gitlab.com/","status",{"text":540,"config":541},"Terms of use",{"href":542,"dataGaName":543,"dataGaLocation":490},"/terms/","terms of use",{"text":545,"config":546},"Privacy statement",{"href":547,"dataGaName":548,"dataGaLocation":490},"/privacy/","privacy statement",{"text":550,"config":551},"Cookie preferences",{"dataGaName":552,"dataGaLocation":490,"id":553,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"title":106,"links":555,"subMenu":563},[556,560],{"text":557,"config":558},"DevSecOps platform",{"href":71,"dataGaName":559,"dataGaLocation":490},"devsecops platform",{"text":129,"config":561},{"href":78,"dataGaName":562,"dataGaLocation":490},"ai-assisted development",[564],{"title":565,"links":566},"Topics",[567,572,577,582,587,592,597,602],{"text":568,"config":569},"CICD",{"href":570,"dataGaName":571,"dataGaLocation":490},"/topics/ci-cd/","cicd",{"text":573,"config":574},"GitOps",{"href":575,"dataGaName":576,"dataGaLocation":490},"/topics/gitops/","gitops",{"text":578,"config":579},"DevOps",{"href":580,"dataGaName":581,"dataGaLocation":490},"/topics/devops/","devops",{"text":583,"config":584},"Version Control",{"href":585,"dataGaName":586,"dataGaLocation":490},"/topics/version-control/","version control",{"text":588,"config":589},"DevSecOps",{"href":590,"dataGaName":591,"dataGaLocation":490},"/topics/devsecops/","devsecops",{"text":593,"config":594},"Cloud Native",{"href":595,"dataGaName":596,"dataGaLocation":490},"/topics/cloud-native/","cloud native",{"text":598,"config":599},"AI for Coding",{"href":600,"dataGaName":601,"dataGaLocation":490},"/topics/devops/ai-for-coding/","ai for coding",{"text":603,"config":604},"Agentic AI",{"href":605,"dataGaName":606,"dataGaLocation":490},"/topics/agentic-ai/","agentic ai",{"title":608,"links":609},"Solutions",[610,612,614,619,623,626,630,633,635,638,641,646],{"text":150,"config":611},{"href":145,"dataGaName":150,"dataGaLocation":490},{"text":139,"config":613},{"href":121,"dataGaName":122,"dataGaLocation":490},{"text":615,"config":616},"Agile development",{"href":617,"dataGaName":618,"dataGaLocation":490},"/solutions/agile-delivery/","agile delivery",{"text":620,"config":621},"SCM",{"href":135,"dataGaName":622,"dataGaLocation":490},"source code management",{"text":568,"config":624},{"href":127,"dataGaName":625,"dataGaLocation":490},"continuous integration & delivery",{"text":627,"config":628},"Value stream management",{"href":178,"dataGaName":629,"dataGaLocation":490},"value stream management",{"text":573,"config":631},{"href":632,"dataGaName":576,"dataGaLocation":490},"/solutions/gitops/",{"text":188,"config":634},{"href":190,"dataGaName":191,"dataGaLocation":490},{"text":636,"config":637},"Small business",{"href":195,"dataGaName":196,"dataGaLocation":490},{"text":639,"config":640},"Public sector",{"href":200,"dataGaName":201,"dataGaLocation":490},{"text":642,"config":643},"Education",{"href":644,"dataGaName":645,"dataGaLocation":490},"/solutions/education/","education",{"text":647,"config":648},"Financial services",{"href":649,"dataGaName":650,"dataGaLocation":490},"/solutions/finance/","financial services",{"title":208,"links":652},[653,655,657,659,662,664,666,668,670,672,674,676,678],{"text":220,"config":654},{"href":222,"dataGaName":223,"dataGaLocation":490},{"text":225,"config":656},{"href":227,"dataGaName":228,"dataGaLocation":490},{"text":230,"config":658},{"href":232,"dataGaName":233,"dataGaLocation":490},{"text":235,"config":660},{"href":237,"dataGaName":661,"dataGaLocation":490},"docs",{"text":258,"config":663},{"href":260,"dataGaName":5,"dataGaLocation":490},{"text":253,"config":665},{"href":255,"dataGaName":256,"dataGaLocation":490},{"text":262,"config":667},{"href":264,"dataGaName":265,"dataGaLocation":490},{"text":275,"config":669},{"href":277,"dataGaName":278,"dataGaLocation":490},{"text":267,"config":671},{"href":269,"dataGaName":270,"dataGaLocation":490},{"text":280,"config":673},{"href":282,"dataGaName":283,"dataGaLocation":490},{"text":285,"config":675},{"href":287,"dataGaName":288,"dataGaLocation":490},{"text":290,"config":677},{"href":292,"dataGaName":293,"dataGaLocation":490},{"text":295,"config":679},{"href":297,"dataGaName":298,"dataGaLocation":490},{"title":313,"links":681},[682,684,686,688,690,692,694,698,703,705,707,709],{"text":320,"config":683},{"href":322,"dataGaName":315,"dataGaLocation":490},{"text":325,"config":685},{"href":327,"dataGaName":328,"dataGaLocation":490},{"text":333,"config":687},{"href":335,"dataGaName":336,"dataGaLocation":490},{"text":338,"config":689},{"href":340,"dataGaName":341,"dataGaLocation":490},{"text":343,"config":691},{"href":345,"dataGaName":346,"dataGaLocation":490},{"text":348,"config":693},{"href":350,"dataGaName":351,"dataGaLocation":490},{"text":695,"config":696},"Sustainability",{"href":697,"dataGaName":695,"dataGaLocation":490},"/sustainability/",{"text":699,"config":700},"Diversity, inclusion and belonging (DIB)",{"href":701,"dataGaName":702,"dataGaLocation":490},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":353,"config":704},{"href":355,"dataGaName":356,"dataGaLocation":490},{"text":363,"config":706},{"href":365,"dataGaName":366,"dataGaLocation":490},{"text":368,"config":708},{"href":370,"dataGaName":371,"dataGaLocation":490},{"text":710,"config":711},"Modern Slavery Transparency Statement",{"href":712,"dataGaName":713,"dataGaLocation":490},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":715},[716,718,720],{"text":540,"config":717},{"href":542,"dataGaName":543,"dataGaLocation":490},{"text":545,"config":719},{"href":547,"dataGaName":548,"dataGaLocation":490},{"text":550,"config":721},{"dataGaName":552,"dataGaLocation":490,"id":553,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[727],{"_path":728,"_dir":729,"_draft":6,"_partial":6,"_locale":7,"content":730,"config":734,"_id":736,"_type":30,"title":19,"_source":32,"_file":737,"_stem":738,"_extension":35},"/en-us/blog/authors/igor-wiedler","authors",{"name":19,"config":731},{"headshot":732,"ctfId":733},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":735},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":740,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":741,"eyebrow":742,"blurb":743,"button":744,"secondaryButton":748,"_id":750,"_type":30,"title":751,"_source":32,"_file":752,"_stem":753,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":745},{"href":746,"dataGaName":49,"dataGaLocation":747},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":749},{"href":53,"dataGaName":54,"dataGaLocation":747},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":755,"content":756,"config":759,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},{"title":17,"description":10,"authors":757,"heroImage":11,"date":20,"body":21,"category":22,"tags":758},[19],[24,25],{"slug":27,"featured":6,"template":28},1761814421060]