[{"data":1,"prerenderedAt":760},["ShallowReactive",2],{"/en-us/blog/learning-python-with-a-little-help-from-ai-code-suggestions":3,"navigation-en-us":37,"banner-en-us":465,"footer-en-us":482,"Michael Friedrich":726,"next-steps-en-us":739,"footer-source-/en-us/blog/learning-python-with-a-little-help-from-ai-code-suggestions/":754},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":27,"_id":30,"_type":31,"title":32,"_source":33,"_file":34,"_stem":35,"_extension":36},"/en-us/blog/learning-python-with-a-little-help-from-ai-code-suggestions","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"Learning Python with a little help from AI","Use this guided tutorial, along with GitLab Duo Code Suggestions, to learn a new programming language.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749663918/Blog/Hero%20Images/aipower.jpg","https://about.gitlab.com/blog/learning-python-with-a-little-help-from-ai-code-suggestions","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"Learning Python with a little help from AI\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Michael Friedrich\"}],\n        \"datePublished\": \"2023-11-09\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Michael Friedrich","2023-11-09","Learning a new programming language can help broaden your software\ndevelopment expertise, open career opportunities, or create fun challenges.\nHowever, it can be difficult to decide on one specific approach to learning\na new language. Artificial intelligence (AI) can help. In this tutorial,\nyou'll learn how to leverage AI-powered GitLab Duo Code Suggestions for a\nguided experience in learning the Python programming language with a\npractical hands-on example.\n\n\n- [Preparations](#preparations)\n  - [VS Code](#vs-code)\n  - [Code Suggestions](#code-suggestions)\n- [Learning a new programming language:\nPython](#learning-a-new-programming-language-python)\n    - [Development environment for Python](#development-environment-for-python)\n    - [Hello, World](#hello-world)\n- [Start learning Python with a practical\nexample](#start-learning-python-with-a-practical-example)\n    - [Define variables and print them](#define-variables-and-print-them)\n    - [Explore variable types](#explore-variable-types)\n- [File I/O: Read and print a log file](#file-io-read-and-print-a-log-file)\n\n- [Flow control](#flow-control)\n    - [Loops and lists to collect files](#loops-and-lists-to-collect-files)\n    - [Conditionally collect files](#conditionally-collect-files)\n- [Functions](#functions)\n    - [Start with a simple log format](#start-with-a-simple-log-format)\n    - [String and data structure operations](#string-and-data-structure-operations)\n    - [Parse log files using regular expressions](#parse-log-files-using-regular-expressions)\n    - [Advanced log format: auth.log](#advanced-log-format-authlog)\n    - [Parsing more types: Structured logging](#parsing-more-types-structured-logging)\n- [Printing results and formatting](#printing-results-and-formatting)\n\n- [Dependency management and continuous\nverification](#dependency-management-and-continuous-verification)\n    - [Pip and pyenv: Bringing structure into Python](#pip-and-pyenv-bringing-structure-into-python)\n    - [Automation: Configure CI/CD pipeline for Python](#automation-configure-cicd-pipeline-for-python)\n- [What is next](#what-is-next)\n    - [Async learning exercises](#async-learning-exercises)\n    - [Share your feedback](#share-your-feedback)\n\n## Preparations \n\n\nChoose your [preferred and supported\nIDE](https://docs.gitlab.com/ee/user/project/repository/code_suggestions.html#enable-code-suggestions-in-other-ides-and-editors),\nand follow the documentation to enable Code Suggestions for [GitLab.com\nSaaS](https://docs.gitlab.com/ee/user/project/repository/code_suggestions.html#enable-code-suggestions-on-gitlab-saas)\nor [GitLab self-managed\ninstances](https://docs.gitlab.com/ee/user/project/repository/code_suggestions.html#enable-code-suggestions-on-self-managed-gitlab).\n\n\nProgramming languages can require installing the language interpreter\ncommand-line tools or compilers that generate binaries from source code to\nbuild and run the application.\n\n\n**Tip:** You can also use [GitLab Remote Development\nworkspaces](/blog/quick-start-guide-for-gitlab-workspaces/) to\ncreate your own cloud development environments, instead of local development\nenvironments. This blog post focuses on using VS Code and the GitLab Web\nIDE. \n\n\n### VS Code\n\n\n[Install VS Code](https://code.visualstudio.com/download) on your client,\nand open it. Navigate to the `Extensions` menu and search for `gitlab\nworkflow`. Install the [GitLab Workflow extension for VS\nCode](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow).\nVS Code will also detect the programming languages, and offer to install\nadditional plugins for syntax highlighting and development experience. For\nexample, install the [Python\nextension](https://marketplace.visualstudio.com/items?itemName=ms-python.python).\n\n\n### Code Suggestions\n\n\nFamiliarize yourself with suggestions before actually verifying the\nsuggestions. GitLab Duo Code Suggestions are provided as you type, so you do\nnot need use specific keyboard shortcuts. To accept a code suggestion, press\nthe `tab` key. Also note that writing new code works more reliably than\nrefactoring existing code. AI is non-deterministic, which means that the\nsame suggestion may not be repeated after deleting the code suggestion.\nWhile Code Suggestions is in Beta, we are working on improving the accuracy\nof generated content overall. Please review the [known\nlimitations](https://docs.gitlab.com/ee/user/project/repository/code_suggestions.html#known-limitations),\nas this could affect your learning experience.\n\n\n**Tip:** The latest release of Code Suggestions supports multiline\ninstructions. You can refine the specifications to your needs to get better\nsuggestions. We will practice this method throughout the blog post.\n\n\n## Learning a new programming language: Python  \n\n\nNow, let's dig into learning Python, which is one of the [supported\nlanguages in Code\nSuggestions](https://docs.gitlab.com/ee/user/project/repository/code_suggestions.html#supported-languages). \n\n\nBefore diving into the source code, make sure to set up your development\nenvironment.\n\n\n### Development environment for Python \n\n\n1) Create a new project `learn-python-ai` in GitLab, and clone the project\ninto your development environment. All code snippets are available in this\n[\"Learn Python with AI\"\nproject](https://gitlab.com/gitlab-de/use-cases/ai/learn-with-ai/learn-python-ai).\n\n\n```shell\n\ngit clone https://gitlab.com/NAMESPACE/learn-python-ai.git\n\n\ncd learn-python-ai\n\n\ngit status\n\n```\n\n\n2) Install Python and the build toolchain. Example on macOS using Homebrew:\n\n\n```\n\nbrew install python\n\n```\n\n\n3) Consider adding a `.gitignore` file for Python, for example this\n[.gitignore template for\nPython](https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/gitignore/Python.gitignore?ref_type=heads). \n\n\nYou are all set to learn Python! \n\n\n### Hello, World\n\n\nStart your learning journey in the [official\ndocumentation](https://www.python.org/about/gettingstarted/), and review the\nlinked resources, for example, the [Python\ntutorial](https://docs.python.org/3/tutorial/index.html). The\n[library](https://docs.python.org/3/library/index.html) and [language\nreference](https://docs.python.org/3/reference/index.html) documentation can\nbe helpful, too. \n\n\n**Tip:** When I touched base with Python in 2005, I did not have many use\ncases except as a framework to test Windows 2000 drivers. Later, in 2016, I\nrefreshed my knowledge with the book \"Head First Python, 2nd Edition,\"\nproviding great practical examples for the best learning experience – two\nweeks later, I could explain the differences between Python 2 and 3. You do\nnot need to worry about Python 2 – it has been deprecated some years ago,\nand we will focus only on Python 3 in this blog post. In August 2023, \"[Head\nFirst Python, 3rd\nEdition](https://www.oreilly.com/library/view/head-first-python/9781492051282/)\"\nwas published. The book provides a great learning resource, along with the\nexercises shared in this blog post. \n\n\nCreate a new file `hello.py` in the root directory of the project and start\nwith a comment saying `# Hello world`. Review and accept the suggestion by\npressing the `tab` key and save the file (keyboard shortcut: cmd s). \n\n\n```\n\n# Hello world\n\n```\n\n\nCommit the change to the Git repository. In VS Code, use the keyboard\nshortcut `ctrl shift G`, add a commit message, and hit `cmd enter` to\nsubmit. \n\n\nUse the command palette (`cmd shift p`) and search for `create terminal` to\nopen a new terminal. Run the code with the Python interpreter. On macOS, the\nbinary from Homebrew is called `python3`, other operating systems and\ndistributions might use `python` without the version.\n\n\n```shell\n\npython3 hello.py\n\n```\n\n\n![Hello World, hello GitLab Duo Code\nSuggestions](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_python_code_sugestions_hello_world.png)\n\n\n**Tip:** Adding code comments in Python starting with the `#` character\nbefore you start writing a function or algorithm will help Code Suggestions\nwith more context to provide better suggestions. In the example above, we\ndid that with `# Hello world`, and will continue doing so in the next\nexercises.\n\n\nAdd `hello.py` to Git, commit all changes and push them to your GitLab\nproject.\n\n\n```shell\n\ngit add hello.py\n\n\ngit commit -avm \"Initialize Python\"\n\n\ngit push\n\n```\n\n\nThe source code for all exercises in this blog post is available in this\n[\"Learn Python with AI\"\nproject](https://gitlab.com/gitlab-de/use-cases/ai/learn-with-ai/learn-python-ai).\n\n\n## Start learning Python with a practical example \n\n\nThe learning goal in the following sections involves diving into the\nlanguage datatypes, variables, flow control, and functions. We will also\nlook into file operations, string parsing, and data structure operations for\nprinting the results. The exercises will help build a command-line\napplication that reads different log formats, works with the data, and\nprovides a summary. This will be the foundation for future projects that\nfetch logs from REST APIs, and inspire more ideas such as rendering images,\ncreating a web server, or adding Observability metrics.\n\n\n![Parsing log files into structured objects, example result after following\nthe\nexercises](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_terminal_parsing_logs_and_pretty_print_results.png)\n\n\nAs an experienced admin, you can put the script into production and use\nreal-world log format exmples. Parsing and analyzing logs in stressful\nproduction incidents can be time-consuming. A local CLI tool is sometimes\nfaster than a log management tool.\n\n\nLet's get started: Create a new file called `log_reader.py` in the directory\nroot, add it to Git, and create a Git commit.\n\n\n### Define variables and print them\n\n\nAs a first step, we need to define the log files location, and the expected\nfile suffix. Therefore, let's create two variables and print them. Actually,\nask Code Suggestions to do that for you by writing only the code comments\nand accepting the suggestions. Sometimes, you need to experiment with\nsuggestions and delete already accepted code blocks. Do not worry – the\nquality of the suggestions will improve over time as the model generates\nbetter suggestions with more context.\n\n\n![Define log path and file suffix\nvariables](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_log_reader_variables_01.png){:\n.shadow}\n\n\n![Print the variables to\nverify](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_log_reader_variables_02.png){:\n.shadow}\n\n\n```python\n\n# Specify the path and file suffix in variables\n\npath = '/var/log/'\n\nfile_suffix = '.log'\n\n\n# Print the variables \n\n\nprint(path)\n\nprint(file_suffix)\n\n```\n\n\nNavigate into the VS Code terminal and run the Python script:\n\n\n```shell\n\npython3 log_reader.py\n\n```\n\n\n![VS Code terminal, printing the\nvariables](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_terminal_print_variables.png)\n\n\nPython supports many different types in the [standard\nlibrary](https://docs.python.org/3/library/index.html). Most common types\nare: Numeric (int, float, complex), Boolean (True, False), and String (str).\nData structures include support for lists, tuples, and dictionaries. \n\n\n### Explore variable types \n\n\nTo practice different variable types, let's define a limit of log files to\nread as a variable with the `integer` type.\n\n\n![Log file\nvariable](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_log_reader_variables_03.png){:\n.shadow}\n\n\n```python\n\n# Define log file limit variable \n\nlog_file_limit = 1024 \n\n```\n\n\nCreate a Boolean variable that forces to read all files in the directory, no\nmatter the log file suffix. \n\n\n```python\n\n# Define boolean variable whether to read all files recursively\n\nread_all_files_recursively = True\n\n```\n\n\n## File I/O: Read and print a log file\n\n\nCreate a directory called `log-data` in your project tree. You can copy all\nfile examples from the [log-data directory in the example\nproject](https://gitlab.com/gitlab-de/use-cases/ai/learn-with-ai/learn-python-ai/-/tree/main/log-data?ref_type=heads).  \n\n\nCreate a new file `sample.log` with the following content, or any other two\nlines that provide a different message at the end.\n\n\n```\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: dpkg-db-backup.service: Deactivated\nsuccessfully.\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: Finished Daily dpkg database backup\nservice.\n\n```\n\n\nInstruct Code Suggestions to read the file `log-data/sample.log` and print\nthe content. \n\n\n![Code Suggestions: Read log file and print\nit](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_read_log_file_and_print.png){:\n.shadow}\n\n\n```python\n\n# Read the file in log-data/sample.log and print its content\n\nwith open('log-data/sample.log', 'r') as f:\n    print(f.read())\n```\n\n\n**Tip:** You will notice the indent here. The `with open() as f:` statement\nopens a new scope where `f` is available as stream. This flow requires\nindenting )`tab`) the code block, and perform actions in this scope, calling\n`f.read()` to read the file contents, and passing the immediate value as\nparameter into the `print()` function.\n\n\nNavigate into the terminal, and run the script again with `python3\nlog_reader.py`. You will see the file content shown in the VS Code editor,\nalso printed into the terminal.\n\n\n![VS Code terminal: Read log file, and print\nit](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_terminal_print_logfile_show_file_sample.png)\n\n\n## Flow control \n\n\nReading one log file is not enough – we want to analyze all files in a given\ndirectory recursively. For the next exercise, we instruct Code Suggestions\nto create an index of all files. \n\n\nPrepare the `log-data` directory with more example files from the [log-data\ndirectory in the example\nproject](https://gitlab.com/gitlab-de/use-cases/ai/learn-with-ai/learn-python-ai/-/tree/main/log-data?ref_type=heads).\nThe directory tree should look as follows:\n\n\n```shell\n\ntree log-data                                                             ─╯\n\nlog-data\n\n├── sample.log\n\n└── var\n    └── log\n        ├── auth.log\n        ├── syslog.log\n        └── syslog_structured.log\n\n3 directories, 4 files\n\n```\n\n\n### Loops and lists to collect files \n\n\nModify the `path` variable to use the value `log-data/`. \n\n\n```python\n\n# Specify the path and file suffix in variables\n\npath = 'log-data/'\n\nfile_suffix = '.log'\n\n```\n\n\nTell Code Suggestions to read all file paths in the directory into a list.\nAfter the collection loop, print the list of file paths. \n\n\n```python\n\n# Read all file paths in the directory into a list\n\n\n# Print the list of log file paths\n\n```\n\n\n![Code Suggestion, collect file\npaths](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_collect_files.png){:\n.shadow}\n\n\nAn example of a suggestion could look like this:\n\n\n```python\n\n# Read all file paths in the directory into a list\n\nimport os\n\n\n# Read all file paths in the directory into a list\n\nlog_files = []\n\nfor root, directories, files in os.walk(path):\n    for file in files:\n        if file.endswith(file_suffix):\n            log_files.append(os.path.join(root, file))\n\n# Print the list of log file paths\n\n\nprint(log_files)\n\n```\n\n\nLet's analyze what happens here: The `import os` statement is required to\nbring the `os` library into the current scope, and we are able to call\n`os.walk()` later. The `log_files`\n[list](https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range)\nis initialized as an empty list using empty brackets. `os.walk()` expects a\npath as directory tree to start searching for files. The function returns a\ngenerator object, which will walk the directory tree recursively. The\nreturned tuples include the collected files, which we want to access in a\nloop iteraor, using the `for` loop notation. When the file ends with the\ndefined suffix, its full path is appended to the list. This is an optimized\nand complex example. Do not worry about understanding everything just yet -\nwe will revisit this later. \n\n\n### Conditionally collect files \n\n\nModify the instructions and add the Boolean flag evaluation for\n`read_all_files_recursively` as an input to the glob() function. \n\n\n```python\n\n# Read all file paths in the top level directory into a list\n\n# If read_all_files_recursively is True, read all files recursively\n\n```\n\n\n![Code Suggestions, read files recursively\nconditionally](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_read_files_recursively_conditionally.png){:\n.shadow}\n\n\n```\n\n# Read all file paths in the top level directory \n\nimport os\n\n\n# Read all file paths in the top level directory into a list\n\n# If read_all_files_recursively is True, read all files recursively\n\n\nlog_files = []\n\nfor file in os.listdir(path):\n    if file.endswith(file_suffix):\n        log_files.append(os.path.join(path, file))\n\nif read_all_files_recursively:\n    for root, directories, files in os.walk(path):\n        for file in files:\n            if file.endswith(file_suffix):               \n                log_files.append(os.path.join(root, file))\n\n# Print log_files\n\nprint(log_files)\n\n```\n\n\nThe result is not optimal yet because it always executes the first loop, and\noptionally the second loop. This flow leads to duplicated results when the\nscript is executed.\n\n\n```\n\npython3 log_reader.py\n\n\n['log-data/sample.log', 'log-data/sample.log', 'log-data/var/log/auth.log']\n\n```\n\n\nExperiment with Code Suggestions instructions to get a solution for the\nproblem. There are different approaches you can take: \n\n\n1) A potential solution is to wrap the source code into an if-then-else\nblock, and move the `os.listdir()` loop into the else-block. \n\n\n```python\n\nif read_all_files_recursively:\n    for root, directories, files in os.walk(path):\n        for file in files:\n            if file.endswith(file_suffix):               \n                log_files.append(os.path.join(root, file))\nelse:\n    for file in os.listdir(path):\n        if file.endswith(file_suffix):\n            log_files.append(os.path.join(path, file))  \n\n```\n\n\n2) Alternatively, do not use `append()` to always add a new list entry, but\ncheck if the item exists in the list first. \n\n\n```python\n\nfor file in os.listdir(path):\n    if file.endswith(file_suffix):\n        # check if the entry exists in the list already\n        if os.path.isfile(os.path.join(path, file)):\n            log_files.append(os.path.join(path, file))\n\nif read_all_files_recursively:\n    for root, directories, files in os.walk(path):\n        for file in files:\n            if file.endswith(file_suffix):\n                # check if the entry exists in the list already\n                if file not in log_files:\n                    log_files.append(os.path.join(root, file))\n```\n\n\n3) Or, we could eliminate duplicate entries after collecting all items.\nPython allows converting lists into\n[sets](https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset),\nwhich hold unique entries. After applying `set()`, you can again convert the\nset back into a list. Code Suggestions knows about this possibility, and\nwill help with the comment `# Ensure that only unique file paths are in the\nlist` \n\n\n![Code Suggestions, converting a list to unique\nitems](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_unique_list.png){:\n.shadow}\n\n\n```python\n\n# Ensure that only unique file paths are in the list\n\n\nlog_files = list(set(log_files))\n\n```\n\n\n4) Take a step back and evaluate whether the variable\nread_all_files_recursively makes sense. Maybe the default behavior should\njust be reading all files recursively?\n\n\n**Tip for testing different paths in VS Code:** Select the code blocks, and\npress [`cmd /` on\nmacOS](https://code.visualstudio.com/docs/getstarted/keybindings) to comment\nout the code. \n\n\n## Functions \n\n\nLet's create a function called `parse_log_file` that parses a log file, and\nreturns the extracted data. We will define the expected log format and\ncolumns to extract, following the [syslog format\nspecification](https://en.wikipedia.org/wiki/Syslog). There are different\nlog format types and also customized formats by developers that need to be\ntaken into account – exercise for later. \n\n\n### Start with a simple log format \n\n\nInspect a running Linux VM, or use the following example log file example\nfor additional implementation.\n\n\n```\n\nless /var/log/syslog | grep -v docker \n\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: Starting Daily dpkg database backup\nservice...\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: Starting Rotate log files...\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: dpkg-db-backup.service: Deactivated\nsuccessfully.\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: Finished Daily dpkg database backup\nservice.\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: logrotate.service: Deactivated\nsuccessfully.\n\nOct 17 00:00:04 ebpf-chaos systemd[1]: Finished Rotate log files.\n\nOct 17 00:17:01 ebpf-chaos CRON[727495]: (root) CMD (   cd / && run-parts\n--report /etc/cron.hourly)\n\n```\n\n\nWe can create an algorithm to split each log line by whitespaces, and then\njoin the results again. Let's ask Code Suggestions for help. \n\n\n```python\n\n# Split log line \"Oct 17 00:00:04 ebpf-chaos systemd[1]: Finished Rotate log\nfiles.\" by whitespaces and save in a list\n\n\nlog_line = \"Oct 17 00:00:04 ebpf-chaos systemd[1]: Finished Rotate log\nfiles.\"\n\nlog_line_split = log_line.split(\" \")\n\nprint(log_line_split)\n\n```\n\n\nRun the script again to verify the result.\n\n\n```shell\n\npython3 log_reader.py\n\n\n['Oct', '17', '00:00:04', 'ebpf-chaos', 'systemd[1]:', 'Finished', 'Rotate',\n'log', 'files.']\n\n```\n\n\nThe first three items are part of the datetime string, followed by the host,\nservice, and remaining log message items. Let's practice string operations\nin Python as the next step. \n\n\n### String and data structure operations\n\n\nLet's ask Code Suggestions for help with learning to join strings, and\nperform list operations.\n\n\n1. Join the first three items with a whitespace again. \n\n2. Keep host and service. \n\n3. Join the remaining variable item count into a string, separated with\nwhitespaces, again. \n\n4. Store the identified column keys, and their respective values in a new\ndata structure:\n[dictionary](https://docs.python.org/3/library/stdtypes.html#mapping-types-dict). \n\n\n![Code suggestions for list items with string\noperations](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_list_items_string_join_extract.png){:\n.shadow}\n\n\n```shell \n\npython3 log_reader.py\n\n\n# Array\n\n['Oct', '17', '00:00:04', 'ebpf-chaos', 'systemd[1]:', 'Finished', 'Rotate',\n'log', 'files.']\n\n\n# Dictionary \n\n{'datetime': 'Oct 17 00:00:04', 'host': 'ebpf-chaos', 'service':\n'systemd[1]:', 'message': ' ebpf-chaos systemd[1]: Finished Rotate log\nfiles.'}\n\n```\n\n\nA working suggestion can look like the following:\n\n\n```python\n\n# Initialize results dictionary with empty values for datetime, host,\nservice, message\n\n# Loop over log line split \n\n# Join the first three list items as date string\n\n# Item 4: host \n\n# Item 5: service\n\n# Join the remaining items into a string, separated with whitespaces \n\n# Print the results after the loop \n\n\nresults = {'datetime': '', 'host': '', 'service': '', 'message': ''}\n\n\nfor item in log_line_split:\n\n    if results['datetime'] == '':\n        results['datetime'] = ' '.join(log_line_split[0:3])\n\n    elif results['host'] == '':\n        results['host'] = log_line_split[3]\n\n    elif results['service'] == '':\n        results['service'] = log_line_split[4]\n\n    else:\n        results['message'] += ' ' + item\n\nprint(results)\n\n\n```\n\n\nThe suggested algorithm loops over all log line items, and applies the same\noperation for the first three items. `log_line_split[0:3]` extracts a slice\nof three items into a new list. Calling `join()` on a separator character\nand passing the array as an argument joins the items into a string. The\nalgorithm continues to check for not initialized values for host (Item 4)\nand service (Item 5)and concludes with the remaining list items appended\ninto the message string. To be honest, I would have used a slightly\ndifferent algorithm, but it is a great learning curve to see other\nalgorithms, and ways to implement them. Practice with different\ninstructions, and data structures, and continue printing the data sets. \n\n\n**Tip:** If you need to terminate a script early, you can use `sys.exit()`.\nThe remaining code will not be executed. \n\n\n```python\n\nimport sys \n\nsys.exit(1)\n\n```\n\n\nImagine doing these operations for different log formats, and message types\n– it can get complicated and error-prone very quickly. Maybe there is\nanother approach. \n\n\n### Parse log files using regular expressions\n\n\nThere are different syslog format RFCs – [RFC\n3164](https://datatracker.ietf.org/doc/html/rfc3164) is obsolete but still\nfound in the wild as default configuration (matching the pattern above),\nwhile [RFC 5424](https://datatracker.ietf.org/doc/html/rfc5424) is more\nmodern, including datetime with timezone information. Parsing this format\ncan be tricky, so let's ask Code Suggestions for advice. \n\n\nIn some cases, the suggestions include regular expressions. They might not\nmatch immediately, making the code more complex to debug, with trial and\nerrors. A good standalone resource to text and explain regular expressions\nis [regex101.com](https://regex101.com/).  \n\n\n**Tip:** You can skip diving deep into regular expressions using the\nfollowing code snippet as a quick cheat. The next step involves instructing\nCode Suggestions to use these log patterns, and help us extract all valuable\ncolumns. \n\n\n```python\n\n# Define the syslog log format regex in a dictionary\n\n# Add entries for RFC3164, RFC5424\n\nregex_log_pattern = {\n    'rfc3164': '([A-Z][a-z][a-z]\\s{1,2}\\d{1,2}\\s\\d{2}[:]\\d{2}[:]\\d{2})\\s([\\w][\\w\\d\\.@-]*)\\s(.*)$',\n    'rfc5424': '(?:(\\d{4}[-]\\d{2}[-]\\d{2}[T]\\d{2}[:]\\d{2}[:]\\d{2}(?:\\.\\d{1,6})?(?:[+-]\\d{2}[:]\\d{2}|Z)?)|-)\\s(?:([\\w][\\w\\d\\.@-]*)|-)\\s(.*)$;'\n}\n\n```\n\n\nWe know what the function should do, and its input parameters – the file\nname, and a log pattern to match. The log lines should be split by this\nregular expression, returning a key-value dictionary for each log line. The\nfunction should return a list of dictionaries. \n\n\n```python\n\n# Create a function that parses a log file\n\n# Input parameter: file path\n\n# Match log line against regex_log_pattern\n\n# Return the results as dictionary list: log line, pattern, extracted\ncolumns\n\n```\n\n\n![Code suggestion based on a multiline comment instruction to get a function\nthat parses a log file based on regex\npatterns](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_log_format_regex_function_instructions_01.png){:\n.shadow}\n\n\nRemember the indent for opening a new scope? The same applies for functions\nin Python. The `def` identifier requires a function name, and a list of\nparameters, followed by an opening colon. The next lines of code require the\nindent. VS Code will help with live-linting wrong indent, before the script\nexecution fails, or the CI/CD pipelines. \n\n\nContinue with Code Suggestions – it might already know that you want to\nparse all log files, and parse them using the newly created function. \n\n\n![Code suggestion to parse all log files, and print the result\nset](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_log_format_regex_function_instructions_02.png){:\n.shadow}\n\n\nA full working example can look like this: \n\n\n```\n\nimport os\n\n\n# Specify the path and file suffix in variables\n\npath = 'log-data/'\n\nfile_suffix = '.log'\n\n\n# Read all file paths in the directory into a list\n\nlog_files = []\n\nfor root, directories, files in os.walk(path):\n    for file in files:\n        if file.endswith(file_suffix):\n            log_files.append(os.path.join(root, file))\n\n# Define the syslog log format regex in a dictionary\n\n# Add entries for RFC3164, RFC5424\n\nregex_log_pattern = {\n    'rfc3164': '([A-Z][a-z][a-z]\\s{1,2}\\d{1,2}\\s\\d{2}[:]\\d{2}[:]\\d{2})\\s([\\w][\\w\\d\\.@-]*)\\s(.*)$',\n    'rfc5424': '(?:(\\d{4}[-]\\d{2}[-]\\d{2}[T]\\d{2}[:]\\d{2}[:]\\d{2}(?:\\.\\d{1,6})?(?:[+-]\\d{2}[:]\\d{2}|Z)?)|-)\\s(?:([\\w][\\w\\d\\.@-]*)|-)\\s(.*)$;'\n}\n\n\n# Create a function that parses a log file\n\n# Input parameter: file path\n\n# Match log line against regex_log_pattern\n\n# Return the results as dictionary list: log line, pattern name, extracted\ncolumns\n\nimport re\n\n\ndef parse_log_file(file_path):\n    # Read the log file\n    with open(file_path, 'r') as f:\n        log_lines = f.readlines()\n\n    # Create a list to store the results\n    results = []\n\n    # Iterate over the log lines\n    for log_line in log_lines:\n        # Match the log line against the regex pattern\n        for pattern_name, pattern in regex_log_pattern.items():\n            match = re.match(pattern, log_line)\n\n            # If the log line matches the pattern, add the results to the list\n            if match:\n                extracted_columns = match.groups()\n                results.append({\n                    'log_line': log_line,\n                    'pattern_name': pattern_name,\n                    'extracted_columns': extracted_columns,\n                    'source_file': file_path\n                })\n\n    # Return the results\n    return results\n\n# Parse all files and print results\n\nfor log_file in log_files:\n    results = parse_log_file(log_file)\n    print(results)\n```\n\n\nLet's unpack what the `parse_log_file()` function does:\n\n\n1. Opens the file from `file_path` parameter. \n\n2. Reads all lines into a new variable `log_lines`. \n\n3. Creates a results list to store all items. \n\n4. Iterates over the log lines. \n\n5. Matches against all regex patterns configured in regex_log_pattern. \n\n6. If a match is found, extracts the matching column values.\n\n7. Creates a results item, including the values for the keys `log_line`,\n`pattern_name`, `extracted_colums`, `source_file`. \n\n8. Appends the results item to the results list.\n\n9. Returns the results list. \n\n\nThere are different variations to this – especially for the returned result\ndata structure. For this specific case, log lines come as list already.\nAdding a dictionary object instead of a raw log line allows function callers\nto extract the desired information in the next step. Once a working example\nhas been implemented, you can refactor the code later, too. \n\n\n### Advanced log format: auth.log\n\n\nParsing the syslog on a Linux distribution might not unveil the necessary\ndata to analyze. On a virtual machine that exposes port 22 (SSH) to the\nworld, the authentication log is much more interesting – plenty of bots and\nmalicious actors testing default password combinations and often brute force\nattacks.\n\n\nThe following snippet from `/var/log/auth.log` on one of my private servers\nshows the authentication log format and the random attempts from bots using\ndifferent usernames, etc. \n\n\n```\n\nOct 15 00:00:19 ebpf-chaos sshd[3967944]: Failed password for invalid user\nubuntu from 93.254.246.194 port 48840 ssh2\n\nOct 15 00:00:20 ebpf-chaos sshd[3967916]: Failed password for root from\n180.101.88.227 port 44397 ssh2\n\nOct 15 00:00:21 ebpf-chaos sshd[3967944]: Received disconnect from\n93.254.246.194 port 48840:11: Bye Bye [preauth]\n\nOct 15 00:00:21 ebpf-chaos sshd[3967944]: Disconnected from invalid user\nubuntu 93.254.246.194 port 48840 [preauth]\n\nOct 15 00:00:24 ebpf-chaos sshd[3967916]: Failed password for root from\n180.101.88.227 port 44397 ssh2\n\nOct 15 00:00:25 ebpf-chaos sshd[3967916]: Received disconnect from\n180.101.88.227 port 44397:11:  [preauth]\n\nOct 15 00:00:25 ebpf-chaos sshd[3967916]: Disconnected from authenticating\nuser root 180.101.88.227 port 44397 [preauth]\n\nOct 15 00:00:25 ebpf-chaos sshd[3967916]: PAM 2 more authentication\nfailures; logname= uid=0 euid=0 tty=ssh ruser= rhost=180.101.88.227 \nuser=root\n\nOct 15 00:00:25 ebpf-chaos sshd[3967998]: Invalid user teamspeak from\n185.218.20.10 port 33436\n\n```\n\n\n**Tip for intrusion prevention:** Add a firewall setup, and use\n[fail2ban](https://en.wikipedia.org/wiki/Fail2ban) to block invalid auth\nlogins. \n\n\nThe next exercise is to extend the logic to understand the free form log\nmessage parts, for example `Failed password for invalid user ubuntu from\n93.254.246.194 port 48840 ssh2`. The task is to store the data in an\noptional dictionary with key value pairs. \n\n\nCreate a new function that takes the previously parsed log line results as\ninput, and specifically parses the last list item for each line.\n\n\n1. Count the number of `Failed password` and `Invalid user` messages.\n\n2. Return the results with count, log file, pattern \n\n\n![Code suggestions for a log file message parser to count auth.log\nfailures](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_parse_log_message_auth_log.png){:\n.shadow}\n\n\nA working suggestion can look like the following code:\n\n\n```python\n\n# Create a function that parses a log file message from the last\nextracted_columns entry \n\n# Input: Parsed log lines results list \n\n# Loop over all log lines in the list, and extract the last list item as\nmessage \n\n# Count failure strings in the message: Failed password, Invalid user \n\n# Return the results if failure count greater 0: log_file, count, failure\nstring\n\ndef parse_log_file_message(results):\n    failure_results = []\n\n    # Iterate over the log lines\n    for result in results:\n        # Extract the message from the last list item\n        message = result['extracted_columns'][-1]\n\n        # Count the number of failure strings in the message\n        failure_count = message.count('Failed password') + message.count('Invalid user')\n\n        # If the failure count is greater than 0, add the results to the list\n        if failure_count > 0:\n            failure_results.append({\n                'log_file': result['source_file'],\n                'count': failure_count,\n                'failure_string': message\n            })\n\n    # Return the results\n    return failure_results\n\n# Parse all files and print results\n\nfor log_file in log_files:\n    results = parse_log_file(log_file)\n    failure_results = parse_log_file_message(results)\n    print(failure_results)\n```\n\n\nThe algorithm follows the previous implementations: First, create a results\narray to store matching data. Then, iterate over the already parsed\nlog_lines in the list. Each log line contains the `extracted_columns` key,\nwhich holds the free-form message string at the end. The next step is to\ncall the string object function `count()` to count how many times a given\ncharacter sequence is contained in a string. The returned numbers are added\nup to the `failure_count` variable. If it is greater than zero, the result\nis added to the results list, including the `log_file`, `count` and\n`failure_string` key-value pairs. After returning the parsed log message\nresults, loop through all log files, parse them, and print the results\nagain. \n\n\nExecute the script to inspect the detected matches. Note that the data\nstructure can be optimized in future learning steps.\n\n\n```\n\npython3 log_reader.py\n\n\n[{'log_file': 'log-data/var/log/auth.log', 'count': 1, 'failure_string':\n'sshd[3967944]: Failed password for invalid user ubuntu from 93.254.246.194\nport 48840 ssh2'}, {'log_file': 'log-data/var/log/auth.log', 'count': 1,\n'failure_string': 'sshd[3967916]: Failed password for root from\n180.101.88.227 port 44397 ssh2'}, {'log_file': 'log-data/var/log/auth.log',\n'count': 1, 'failure_string': 'sshd[3967916]: Failed password for root from\n180.101.88.227 port 44397 ssh2'}, {'log_file': 'log-data/var/log/auth.log',\n'count': 1, 'failure_string': 'sshd[3967998]: Invalid user teamspeak from\n185.218.20.10 port 33436'}, {'log_file': 'log-data/var/log/auth.log',\n'count': 1, 'failure_string': 'sshd[3967998]: Failed password for invalid\nuser teamspeak from 185.218.20.10 port 33436 ssh2'}, {'log_file':\n'log-data/var/log/auth.log', 'count': 1, 'failure_string': 'sshd[3968077]:\nInvalid user mcserver from 218.211.33.146 port 50950'}]\n\n\n```\n\n\n### Parsing more types: Structured logging\n\n\nApplication developers can use the structured logging format to help machine\nparsers to extract the key value pairs. Prometheus provides this information\nin the following structure in syslog:\n\n\n```\n\nOct 17 19:00:10 ebpf-chaos prometheus[594]: ts=2023-10-17T19:00:10.425Z\ncaller=compact.go:519 level=info component=tsdb m\n\nsg=\"write block\" mint=1697558404661 maxt=1697565600000\nulid=01HCZG4ZX51GTH8H7PVBYDF4N6 duration=148.675854ms\n\nOct 17 19:00:10 ebpf-chaos prometheus[594]: ts=2023-10-17T19:00:10.464Z\ncaller=head.go:1213 level=info component=tsdb msg\n\n=\"Head GC completed\" caller=truncateMemory duration=6.845245ms\n\nOct 17 19:00:10 ebpf-chaos prometheus[594]: ts=2023-10-17T19:00:10.467Z\ncaller=checkpoint.go:100 level=info component=tsd\n\nb msg=\"Creating checkpoint\" from_segment=2308 to_segment=2309\nmint=1697565600000\n\nOct 17 19:00:10 ebpf-chaos prometheus[594]: ts=2023-10-17T19:00:10.517Z\ncaller=head.go:1185 level=info component=tsdb msg\n\n=\"WAL checkpoint complete\" first=2308 last=2309 duration=50.052621ms\n\n```\n\n\nThis format is easier to parse for scripts, because the message part can be\nsplit by whitespaces, and the assignment character `=`. Strings that contain\nwhitespaces are guaranteed to be enclosed with quotes. The downside is that\nnot all programming language libraries provide ready-to-use structured\nlogging libraries, making it harder for developers to adopt this format. \n\n\nPractice following the previous example to parse the `auth.log` format with\nadditional information. Tell Code Suggestions that you are expecting\nstructured logging format with key-value pairs, and which returned data\nstructure would be great:\n\n\n```python\n\n# Create a function that parses a log file message from the last\nextracted_columns entry \n\n# Input: Parsed log lines results list \n\n# Loop over all log lines in the list, and extract the last list item as\nmessage \n\n# Parse structured logging key-value pairs into a dictionary\n\n# Return results: log_file, dictionary \n\n```\n\n\n![Code suggestions for parsing structured logging format in the log file\nmessage\npart](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_propose_structured_logging_message_parser.png){:\n.shadow}\n\n\n### Printing results and formatting\n\n\nMany of the examples used the `print()` statement to print the content on\nthe terminal. Python objects in the standard library support text\nrepresentation, and for some types it makes more sense (string, numbers),\nothers cannot provide much details (functions, etc.). \n\n\nYou can also pretty-print almost any data structure (lists, sets,\ndictionaries) in Python. The JSON library can format data structures in a\nreadable format, and use a given spaces indent to draw the JSON structure on\nthe terminal. Note that we use the `import` statement here to bring\nlibraries into the current scope, and access their methods, for example\n`json.dumps`. \n\n\n```python\n\nimport json \n\nprint(json.dumps(structured_results, indent=4))\n\n```\n\n\n![Parsing log files into structured objects, example result after following\nthe\nexercises](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_terminal_parsing_logs_and_pretty_print_results.png)\n\n\nPractice with modifying the existing source code, and replace the code\nsnippets where appropriate. Alternatively, create a new function that\nimplements pretty printing.\n\n\n```python\n\n# Create a pretty print function with indent 4 \n\n```\n\n\n![Code suggestions for pretty-print\nfunction](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/vs_code_code_suggestions_pretty_print.png){:\n.shadow}\n\n\nThis idea works in a similar fashion with creating your own logger\nfunctions...but we have to stop learning and take a break. Before we\nconclude the first blog post in the learning series, let's ensure that CI/CD\nand dependencies are set up properly for future exercises and async\npractice. \n\n\n## Dependency management and continuous verification  \n\n\n### Pip and pyenv: Bringing structure into Python \n\n\nDependencies can be managed in the [`requirements.txt`\nfile](https://pip.pypa.io/en/stable/reference/requirements-file-format/),\nincluding optional version dependencies. Using `requirements.txt` file also\nhas the advantage of being the single source of truth for local development\nenvironments and running continuous builds with GitLab CI/CD. They can use\nthe same installation command:\n\n\n```shell\n\npip install -r requirements.txt\n\n```\n\n\nSome Linux distributions do not install the pip package manager by default,\nfor example, Ubuntu/Debian require to install the `python3-pip` package. \n\n\nYou can manage different virtual environments using\n[venv](https://docs.python.org/3/library/venv.html). This workflow can be\nbeneficial to install Python dependencies into the virtual environment,\ninstead of globally into the OS path which might break on upgrades. \n\n\n```shell\n\npip install virtualenv\n\nvirtualenv venv\n\nsource venv/bin/activate \n\n```\n\n\n### Automation: Configure CI/CD pipeline for Python\n\n\nThe [CI/CD pipeline](https://docs.gitlab.com/ee/ci/) should continuously\nlint, test, and build the code. You can mimic the steps from the local\ndevelopment, and add testing more environments and versions: \n\n\n1. Lint the source code and check for formatting errors. The example uses\n[Pyflakes](https://pypi.org/project/pyflakes/), a mature linter, and\n[Ruff](https://docs.astral.sh/ruff/ ), a fast linter written in Rust. \n\n2. Cache dependencies installed using the pip package manager, following the\ndocumentation for [Python caching in GitLab\nCI/CD](https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies).\nThis saves time and resources on repeated CI/CD pipeline runs.\n\n3. Use parallel matrix builds to test different Python versions, based on\nthe available container images on Docker Hub and their tags. \n\n\n```yaml\n\nstages:\n  - lint\n  - test\n\ndefault:\n  image: python:latest\n  cache:                      # Pip's cache doesn't store the python packages\n    paths:                    # https://pip.pypa.io/en/stable/topics/caching/\n      - .cache/pip\n  before_script:\n    - python -V               # Print out python version for debugging\n    - pip install virtualenv\n    - virtualenv venv\n    - source venv/bin/activate\n\nvariables:  # Change pip's cache directory to be inside the project\ndirectory since we can only cache local items.\n  PIP_CACHE_DIR: \"$CI_PROJECT_DIR/.cache/pip\"\n\n# lint template\n\n.lint-tmpl:\n  script:\n    - echo \"Linting Python version $VERSION\"\n  parallel:\n    matrix:\n      - VERSION: ['3.9', '3.10', '3.11', '3.12']   # https://hub.docker.com/_/python\n\n# Lint, using Pyflakes: https://pypi.org/project/pyflakes/ \n\nlint-pyflakes:\n  extends: [.lint-tmpl]\n  script:\n    - pip install -r requirements.txt\n    - find . -not -path './venv' -type f -name '*.py' -exec sh -c 'pyflakes {}' \\;\n\n# Lint, using Ruff (Rust): https://docs.astral.sh/ruff/ \n\nlint-ruff:\n  extends: [.lint-tmpl]\n  script:\n    - pip install -r requirements.txt\n    - ruff .\n```\n\n\n![GitLab CI/CD Python lint job view, part of matrix\nbuilds](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/gitlab_cicd_python_lint_job_log_view.png)\n\n\n## What is next \n\n\nFun fact: GitLab Duo Code Suggestions also helped writing this blog post in\nVS Code, knowing about the context. In the screenshot, I just wanted to add\na tip about [regex101](https://regex101.com/), and GitLab Duo already knew. \n\n\n![Writing the GitLab blog post in VS Code with support from GitLab Duo Code\nSuggestions](https://about.gitlab.com/images/blogimages/learn-python-with-ai-code-suggestions-getting-started/gitlab_duo_code_suggestions_helping_write_the_learning_python_ai_blog_post.png)\n\n\nIn an upcoming blog, we will look into advanced learning examples with more\npractical (log) filtering and parallel operations, how to fetch logs from\nAPI endpoints (CI/CD job logs for example), and more data analytics and\nobservability. Until then, here are a few recommendations for practicing\nasync.\n\n\n### Async learning exercises\n\n\n- Implement the missing `log_file_limit` variable check. \n\n- Print a summary of the results in Markdown, not only JSON format. \n\n- Extend the script to accept a search filter as environment variable.\nPrint/count only filtered results. \n\n- Extend the script to accept a date range. It might require parsing the\ndatetime column in a time object to compare the range. \n\n- Inspect a GitLab CI/CD pipeline job log, and download the raw format.\nExtend the log parser to parse this specific format, and print a summary. \n\n\n### Share your feedback\n\n\nWhich programming language are you learning or considering learning? Start a\nnew topic on our [community](/community/) forum or Discord and share your\nexperience.\n\n\nWhen you use [GitLab Duo](/gitlab-duo/) Code Suggestions, please share your\nthoughts and feedback [in the feedback\nissue](https://gitlab.com/gitlab-org/gitlab/-/issues/405152).\n","ai-ml",[23,24,25,26],"DevSecOps platform","tutorial","workflow","AI/ML",{"slug":28,"featured":6,"template":29},"learning-python-with-a-little-help-from-ai-code-suggestions","BlogPost","content:en-us:blog:learning-python-with-a-little-help-from-ai-code-suggestions.yml","yaml","Learning Python With A Little Help From Ai Code Suggestions","content","en-us/blog/learning-python-with-a-little-help-from-ai-code-suggestions.yml","en-us/blog/learning-python-with-a-little-help-from-ai-code-suggestions","yml",{"_path":38,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"data":40,"_id":461,"_type":31,"title":462,"_source":33,"_file":463,"_stem":464,"_extension":36},"/shared/en-us/main-navigation","en-us",{"logo":41,"freeTrial":46,"sales":51,"login":56,"items":61,"search":392,"minimal":423,"duo":442,"pricingDeployment":451},{"config":42},{"href":43,"dataGaName":44,"dataGaLocation":45},"/","gitlab logo","header",{"text":47,"config":48},"Get free trial",{"href":49,"dataGaName":50,"dataGaLocation":45},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":52,"config":53},"Talk to sales",{"href":54,"dataGaName":55,"dataGaLocation":45},"/sales/","sales",{"text":57,"config":58},"Sign in",{"href":59,"dataGaName":60,"dataGaLocation":45},"https://gitlab.com/users/sign_in/","sign in",[62,106,203,208,313,373],{"text":63,"config":64,"cards":66,"footer":89},"Platform",{"dataNavLevelOne":65},"platform",[67,73,81],{"title":63,"description":68,"link":69},"The most comprehensive AI-powered DevSecOps Platform",{"text":70,"config":71},"Explore our Platform",{"href":72,"dataGaName":65,"dataGaLocation":45},"/platform/",{"title":74,"description":75,"link":76},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":77,"config":78},"Meet GitLab Duo",{"href":79,"dataGaName":80,"dataGaLocation":45},"/gitlab-duo/","gitlab duo ai",{"title":82,"description":83,"link":84},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":85,"config":86},"Learn more",{"href":87,"dataGaName":88,"dataGaLocation":45},"/why-gitlab/","why gitlab",{"title":90,"items":91},"Get started with",[92,97,102],{"text":93,"config":94},"Platform Engineering",{"href":95,"dataGaName":96,"dataGaLocation":45},"/solutions/platform-engineering/","platform engineering",{"text":98,"config":99},"Developer Experience",{"href":100,"dataGaName":101,"dataGaLocation":45},"/developer-experience/","Developer experience",{"text":103,"config":104},"MLOps",{"href":105,"dataGaName":103,"dataGaLocation":45},"/topics/devops/the-role-of-ai-in-devops/",{"text":107,"left":108,"config":109,"link":111,"lists":115,"footer":185},"Product",true,{"dataNavLevelOne":110},"solutions",{"text":112,"config":113},"View all Solutions",{"href":114,"dataGaName":110,"dataGaLocation":45},"/solutions/",[116,141,164],{"title":117,"description":118,"link":119,"items":124},"Automation","CI/CD and automation to accelerate deployment",{"config":120},{"icon":121,"href":122,"dataGaName":123,"dataGaLocation":45},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[125,129,133,137],{"text":126,"config":127},"CI/CD",{"href":128,"dataGaLocation":45,"dataGaName":126},"/solutions/continuous-integration/",{"text":130,"config":131},"AI-Assisted Development",{"href":79,"dataGaLocation":45,"dataGaName":132},"AI assisted development",{"text":134,"config":135},"Source Code Management",{"href":136,"dataGaLocation":45,"dataGaName":134},"/solutions/source-code-management/",{"text":138,"config":139},"Automated Software Delivery",{"href":122,"dataGaLocation":45,"dataGaName":140},"Automated software delivery",{"title":142,"description":143,"link":144,"items":149},"Security","Deliver code faster without compromising security",{"config":145},{"href":146,"dataGaName":147,"dataGaLocation":45,"icon":148},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[150,154,159],{"text":151,"config":152},"Application Security Testing",{"href":146,"dataGaName":153,"dataGaLocation":45},"Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":45,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":45},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":45},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":45,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":45,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":45,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":45,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":45,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":45,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":45,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":45},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":45},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":45},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":45,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":45},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":45},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":45},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":45},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":45},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":45},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":45},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":45},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":45},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":45},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":45},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":45},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":45},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":45},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":45},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":45},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":45},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":45},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":45},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":45},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":45},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":45},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":45},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":45},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":52,"config":380},{"href":54,"dataGaName":381,"dataGaLocation":45},"talk to sales",{"text":383,"config":384},"Support portal",{"href":385,"dataGaName":386,"dataGaLocation":45},"https://support.gitlab.com","support portal",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":45},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":59,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":74,"config":405},{"href":79,"dataGaName":74,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":126,"config":411},{"href":128,"dataGaName":126,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":87,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":50,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":79,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},{"freeTrial":452,"mobileIcon":457,"desktopIcon":459},{"text":453,"config":454},"Back to pricing",{"href":206,"dataGaName":455,"dataGaLocation":428,"icon":456},"back to pricing","GoBack",{"altText":430,"config":458},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":460},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":466,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"title":467,"button":468,"image":473,"config":477,"_id":479,"_type":31,"_source":33,"_file":480,"_stem":481,"_extension":36},"/shared/en-us/banner","is now in public beta!",{"text":469,"config":470},"Try the Beta",{"href":471,"dataGaName":472,"dataGaLocation":45},"/gitlab-duo/agent-platform/","duo banner",{"altText":474,"config":475},"GitLab Duo Agent Platform",{"src":476},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":478},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":483,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"data":484,"_id":722,"_type":31,"title":723,"_source":33,"_file":724,"_stem":725,"_extension":36},"/shared/en-us/main-footer",{"text":485,"source":486,"edit":492,"contribute":497,"config":502,"items":507,"minimal":714},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":487,"config":488},"View page source",{"href":489,"dataGaName":490,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":493,"config":494},"Edit this page",{"href":495,"dataGaName":496,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":498,"config":499},"Please contribute",{"href":500,"dataGaName":501,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":503,"facebook":504,"youtube":505,"linkedin":506},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[508,555,607,651,680],{"title":204,"links":509,"subMenu":524},[510,514,519],{"text":511,"config":512},"View plans",{"href":206,"dataGaName":513,"dataGaLocation":491},"view plans",{"text":515,"config":516},"Why Premium?",{"href":517,"dataGaName":518,"dataGaLocation":491},"/pricing/premium/","why premium",{"text":520,"config":521},"Why Ultimate?",{"href":522,"dataGaName":523,"dataGaLocation":491},"/pricing/ultimate/","why ultimate",[525],{"title":526,"links":527},"Contact Us",[528,531,533,535,540,545,550],{"text":529,"config":530},"Contact sales",{"href":54,"dataGaName":55,"dataGaLocation":491},{"text":383,"config":532},{"href":385,"dataGaName":386,"dataGaLocation":491},{"text":388,"config":534},{"href":390,"dataGaName":391,"dataGaLocation":491},{"text":536,"config":537},"Status",{"href":538,"dataGaName":539,"dataGaLocation":491},"https://status.gitlab.com/","status",{"text":541,"config":542},"Terms of use",{"href":543,"dataGaName":544,"dataGaLocation":491},"/terms/","terms of use",{"text":546,"config":547},"Privacy statement",{"href":548,"dataGaName":549,"dataGaLocation":491},"/privacy/","privacy statement",{"text":551,"config":552},"Cookie preferences",{"dataGaName":553,"dataGaLocation":491,"id":554,"isOneTrustButton":108},"cookie preferences","ot-sdk-btn",{"title":107,"links":556,"subMenu":563},[557,560],{"text":23,"config":558},{"href":72,"dataGaName":559,"dataGaLocation":491},"devsecops platform",{"text":130,"config":561},{"href":79,"dataGaName":562,"dataGaLocation":491},"ai-assisted development",[564],{"title":565,"links":566},"Topics",[567,572,577,582,587,592,597,602],{"text":568,"config":569},"CICD",{"href":570,"dataGaName":571,"dataGaLocation":491},"/topics/ci-cd/","cicd",{"text":573,"config":574},"GitOps",{"href":575,"dataGaName":576,"dataGaLocation":491},"/topics/gitops/","gitops",{"text":578,"config":579},"DevOps",{"href":580,"dataGaName":581,"dataGaLocation":491},"/topics/devops/","devops",{"text":583,"config":584},"Version Control",{"href":585,"dataGaName":586,"dataGaLocation":491},"/topics/version-control/","version control",{"text":588,"config":589},"DevSecOps",{"href":590,"dataGaName":591,"dataGaLocation":491},"/topics/devsecops/","devsecops",{"text":593,"config":594},"Cloud Native",{"href":595,"dataGaName":596,"dataGaLocation":491},"/topics/cloud-native/","cloud native",{"text":598,"config":599},"AI for Coding",{"href":600,"dataGaName":601,"dataGaLocation":491},"/topics/devops/ai-for-coding/","ai for coding",{"text":603,"config":604},"Agentic AI",{"href":605,"dataGaName":606,"dataGaLocation":491},"/topics/agentic-ai/","agentic ai",{"title":608,"links":609},"Solutions",[610,612,614,619,623,626,630,633,635,638,641,646],{"text":151,"config":611},{"href":146,"dataGaName":151,"dataGaLocation":491},{"text":140,"config":613},{"href":122,"dataGaName":123,"dataGaLocation":491},{"text":615,"config":616},"Agile development",{"href":617,"dataGaName":618,"dataGaLocation":491},"/solutions/agile-delivery/","agile delivery",{"text":620,"config":621},"SCM",{"href":136,"dataGaName":622,"dataGaLocation":491},"source code management",{"text":568,"config":624},{"href":128,"dataGaName":625,"dataGaLocation":491},"continuous integration & delivery",{"text":627,"config":628},"Value stream management",{"href":179,"dataGaName":629,"dataGaLocation":491},"value stream management",{"text":573,"config":631},{"href":632,"dataGaName":576,"dataGaLocation":491},"/solutions/gitops/",{"text":189,"config":634},{"href":191,"dataGaName":192,"dataGaLocation":491},{"text":636,"config":637},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":491},{"text":639,"config":640},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":491},{"text":642,"config":643},"Education",{"href":644,"dataGaName":645,"dataGaLocation":491},"/solutions/education/","education",{"text":647,"config":648},"Financial services",{"href":649,"dataGaName":650,"dataGaLocation":491},"/solutions/finance/","financial services",{"title":209,"links":652},[653,655,657,659,662,664,666,668,670,672,674,676,678],{"text":221,"config":654},{"href":223,"dataGaName":224,"dataGaLocation":491},{"text":226,"config":656},{"href":228,"dataGaName":229,"dataGaLocation":491},{"text":231,"config":658},{"href":233,"dataGaName":234,"dataGaLocation":491},{"text":236,"config":660},{"href":238,"dataGaName":661,"dataGaLocation":491},"docs",{"text":259,"config":663},{"href":261,"dataGaName":5,"dataGaLocation":491},{"text":254,"config":665},{"href":256,"dataGaName":257,"dataGaLocation":491},{"text":263,"config":667},{"href":265,"dataGaName":266,"dataGaLocation":491},{"text":276,"config":669},{"href":278,"dataGaName":279,"dataGaLocation":491},{"text":268,"config":671},{"href":270,"dataGaName":271,"dataGaLocation":491},{"text":281,"config":673},{"href":283,"dataGaName":284,"dataGaLocation":491},{"text":286,"config":675},{"href":288,"dataGaName":289,"dataGaLocation":491},{"text":291,"config":677},{"href":293,"dataGaName":294,"dataGaLocation":491},{"text":296,"config":679},{"href":298,"dataGaName":299,"dataGaLocation":491},{"title":314,"links":681},[682,684,686,688,690,692,694,698,703,705,707,709],{"text":321,"config":683},{"href":323,"dataGaName":316,"dataGaLocation":491},{"text":326,"config":685},{"href":328,"dataGaName":329,"dataGaLocation":491},{"text":334,"config":687},{"href":336,"dataGaName":337,"dataGaLocation":491},{"text":339,"config":689},{"href":341,"dataGaName":342,"dataGaLocation":491},{"text":344,"config":691},{"href":346,"dataGaName":347,"dataGaLocation":491},{"text":349,"config":693},{"href":351,"dataGaName":352,"dataGaLocation":491},{"text":695,"config":696},"Sustainability",{"href":697,"dataGaName":695,"dataGaLocation":491},"/sustainability/",{"text":699,"config":700},"Diversity, inclusion and belonging (DIB)",{"href":701,"dataGaName":702,"dataGaLocation":491},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":704},{"href":356,"dataGaName":357,"dataGaLocation":491},{"text":364,"config":706},{"href":366,"dataGaName":367,"dataGaLocation":491},{"text":369,"config":708},{"href":371,"dataGaName":372,"dataGaLocation":491},{"text":710,"config":711},"Modern Slavery Transparency Statement",{"href":712,"dataGaName":713,"dataGaLocation":491},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":715},[716,718,720],{"text":541,"config":717},{"href":543,"dataGaName":544,"dataGaLocation":491},{"text":546,"config":719},{"href":548,"dataGaName":549,"dataGaLocation":491},{"text":551,"config":721},{"dataGaName":553,"dataGaLocation":491,"id":554,"isOneTrustButton":108},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[727],{"_path":728,"_dir":729,"_draft":6,"_partial":6,"_locale":7,"content":730,"config":734,"_id":736,"_type":31,"title":18,"_source":33,"_file":737,"_stem":738,"_extension":36},"/en-us/blog/authors/michael-friedrich","authors",{"name":18,"config":731},{"headshot":732,"ctfId":733},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659879/Blog/Author%20Headshots/dnsmichi-headshot.jpg","dnsmichi",{"template":735},"BlogAuthor","content:en-us:blog:authors:michael-friedrich.yml","en-us/blog/authors/michael-friedrich.yml","en-us/blog/authors/michael-friedrich",{"_path":740,"_dir":39,"_draft":6,"_partial":6,"_locale":7,"header":741,"eyebrow":742,"blurb":743,"button":744,"secondaryButton":748,"_id":750,"_type":31,"title":751,"_source":33,"_file":752,"_stem":753,"_extension":36},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":47,"config":745},{"href":746,"dataGaName":50,"dataGaLocation":747},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":52,"config":749},{"href":54,"dataGaName":55,"dataGaLocation":747},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":755,"content":756,"config":759,"_id":30,"_type":31,"title":32,"_source":33,"_file":34,"_stem":35,"_extension":36},{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},{"title":9,"description":10,"authors":757,"heroImage":11,"date":19,"body":20,"category":21,"tags":758},[18],[23,24,25,26],{"slug":28,"featured":6,"template":29},1761814423397]