Search Gitlab For Sensitive Data and Credentials using GitLab Watchman

GitLab Watchman is an application that uses the GitLab API to Search GitLab for sensitive data and credentials exposed internally – this includes code, commits, wiki pages and more.

Features

It searches GitLab for sensitive data and credentials for internally shared projects and looks at:

• Code
• Commits
• Wiki pages
• Issues
• Merge requests
• Milestones

For the following data:

• GCP keys and service account files
• AWS keys
• Azure keys and service account files
• Google API keys
• Slack API tokens & webhooks
• Private keys (SSH, PGP, any other misc private key)
• Exposed tokens (Bearer tokens, access tokens, client_secret etc.)
• S3 config files
• Tokens for services such as Heroku, PayPal and more
• Passwords in plaintext
• and more

Time based searching

You can run GitLab Watchman to look for results going back as far as:

• 24 hours
• 7 days
• 30 days
• All time

This means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.

Installation

Install via pip

pip install gitlab-watchman

Or via source

Rules

GitLab Watchman uses custom YAML rules to detect matches in GitLab.

---
filename:
enabled: #[true|false]
meta:
name:
author:
date:
description: #what the search should find#
severity: #rating out of 100#
scope: #what to search, any combination of the below#
- blobs
- commits
- milestones
- wiki_blobs
- issues
- merge_requests
test_cases:
match_cases:
- #test case that should match the regex#
fail_cases:
- #test case that should not match the regex#
strings:
- #search query to use in GitLab#
pattern: #Regex pattern to filter out false positives#

There are Python tests to ensure rules are formatted properly and that the Regex patterns work in the tests dir. More information about rules, and how you can add your own, is in the file docs/rules.md.

Search Gitlab For Sensitive Data

GitLab Watchman will be installed as a global command, use as follows:

usage: gitlab-watchman [-h] --timeframe {d,w,m,a} --output
{file,stdout,stream} [--version] [--all] [--blobs]
[--commits] [--wiki-blobs] [--issues] [--merge-requests]
[--milestones] [--comments]

Monitoring GitLab for sensitive data shared publicly

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
--all Find everything
--blobs Search code blobs
--commits Search commits
--wiki-blobs Search wiki blobs
--issues Search issues
--merge-requests Search merge requests
--milestones Search milestones
--comments Search comments

required arguments:
--timeframe {d,w,m,a}
How far back to search: d = 24 hours w = 7 days, m =
30 days, a = all time
--output {file,stdout,stream}
Where to send results

You can run GitLab Watchman to look for everything, and output to default Stdout:

gitlab-watchman –timeframe a –all

Or arguments can be grouped together to search more granularly. This will look for commits and milestones for the last 30 days, and output the results to a TCP stream:

gitlab-watchman –timeframe m –commits –milestones –output stream

Other Watchman apps

You may be interested in some of the other apps in the Watchman family:

• Slack Watchman
• GitHub Watchman