Building an Open-Source Cloud Asset Inventory with CloudQuery and Grafana
November 15, 2021
- Name
- Yevgeny Pats
- @yevgenypats
In this blog post you will learn how to build an open-source cloud asset inventory with CloudQuery and Grafana.
General architecture:
- CloudQuery will take care of extracting, transforming and loading all your assets, across cloud and SaaS apps to PostgreSQL.
- Grafana will be used to query, visualize, monitor, and alert.
This is what you will get:
- All your assets configuration across cloud providers and SaaS apps in one database
- Vanilla PostgreSQL
- Reuse your current (assuming you use Grafana) visualization, monitoring and alerting workflows - send reports and alerts via email, slack.
- 3 out-of-the-box filterable asset inventory dashboards for AWS and GCP including security & compliance dashboards.
Running
To try out the below dashboards (or build a custom one) you will need to connect the same PostgreSQL used by CloudQuery to Grafana as a datasource.
You can run out CloudQuery locally or in your cloud environment. For production deployment see terraform-aws-cloudquery and/or helm-charts](https://github.com/cloudquery/helm-charts).
Importing Dashboards
You can try out some of our pre-made dashboards by importing them straight from our GitHub: AWS, GCP
Dashboard Examples and Use Cases
Asset (Resource) Search
One of the most common but a lot of times non-trivial tasks is to find a specific when the only identifier you have is one piece of information across accounts, regions and resource types. Just a few examples (as number of real-world scenarios is really indefinite ):
- Finding an asset across accounts/regions by name/ARN: In AWS specifically it might involve either clicking through 30+ regions, if you know in which account it is located or even more if not.
- Finding an ec2 instance by its public/private IP: This will also either involve click-ops, or bash sorcery.
Some of those can be also solved by AWS Config but has the following limitations:
- AWS Only - Works only on AWS resources (can’t ingest data from other services/cloud-providers).
- As this is using a proprietary subset of SQL, it can’t be integrated to your current Visualization, Monitoring Alerts workflows, such as Grafana.
This is why we created open-source “basic inventory” Grafana dashboards that you are free to use, customize or build completely new ones (feel free to share back or suggest other):
Filterable AWS dashboards by accounts and regions
Here is snippet from our AWS Asset Inventory Dashboard:
Filterable GCP dashboards across projects
Similar challenge exists in GCP though the situation is a bit better in some sense as you have a single view for each resource/asset type per project. Though, If you want to have a single view of all types of assets across multiple projects (which is common), this would be ClickOps or bash magic again.
Here is an example of GCP Compute Asset inventory dashboard:
Some of that can be solved with the in-house GCP Cloud Asset Inventory but has similar limitations:
- GCP Only - Works only on GCP resources (can’t ingest data from other services/cloud-providers).
- Custom Query Language (not SQL) - You will need to learn a new query engine that might be also more limited then SQL.
- Can’t integrate with best-in-class visualization, monitoring alerting systems such as Grafana.
- Cannot integrate across different organization accounts
Security
You can create your own security views and dashboards that you can then monitor and alert. Each company has its own security and compliance policies but we will share a basic one in this blog (more is coming…) we found useful.
AWS Public/Private EC2 Instances
Filterable dashboards by VPN, subnet, region including public, private ec2 instances.
Summary
We are excited for the future of open-source cloud asset inventory and are looking for your feedback, either on GitHub or Discord. Also, feel free to contribute back or request additional Grafana dashboards.
