3 results found

1. BigQuery near real-time cost anomaly for on-demand pricing model

   BigQuery provides native views such as INFORMATIONSCHEMA.JOBSBY_ORGANIZATION(https://cloud.google.com/bigquery/docs/information-schema-jobs-by-organization) which contain near real-time metadata about all jobs submitted in the organization or project.
   Late last year, I developed a custom solution for a local customer who had a huge cost spike because of an application bug.

   The idea is to run a Cloud Function job that check the cost of BQ jobs in the past hour (can be user defined). If it exceeds a threadshold (let's say $100), it will write a log to Cloud Logging, which then trigger an alert based on a custom Cloud Logging metric.

   This solution is doable because Google charges on-demand BQ users $6.25 per TiB data scanned. We can calculate it like this: ROUND(SUM(totalbytesbilled) * 6.25 / POWER(2, 40), 2) as estimated_cost.

   Details and discussion on Slack
   https://doitintl.slack.com/archives/C010V5SE3T5/p1738026192438959

   BigQuery provides native views such as INFORMATIONSCHEMA.JOBSBY_ORGANIZATION(https://cloud.google.com/bigquery/docs/information-schema-jobs-by-organization) which contain near real-time metadata about all jobs submitted in the organization or project.
   Late last year, I developed a custom solution for a local customer who had a huge cost spike because of an application bug.

   The idea is to run a Cloud Function job that check the cost of BQ jobs in the past hour (can be user defined). If it exceeds a threadshold (let's say $100), it will write a log to Cloud Logging, which then trigger an alert based on a custom Cloud Logging metric.

   … more

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   New  ·  1 comment  ·  Cost Anomalies  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

2. Automatically add the Cluster ID to RDS reports

   This is specific to how AWS RDS specifies the ARN of the instance vs the ARN of the cluster.

   When creating a Report in the console for RDS Aurora the cluster gets an ARN made up not of the cluster name, but of the cluster ID. The cluster ID is not obvious and must be searched in the AWS console or gotten via 2 API calls.
   It would be great if the DoiT console could automatically include the cluster ARN whenever an instance ARN is given, since the storage and backup SKUs (at least) are under the cluster ID ARN instead of the instance ARN.

   See for example this report:
   https://console.doit.com/customers/yplJpzAvfOabLT98x41E/analytics/reports/n9MCbG19DoLYUM9lkN6S

   To get the cluster ID from the Instance name you need to run 2 APIs
   - describe-db-instances
   - describe-db-clusters

   for example:
   ```

   Get DbClusterResourceId shown in Resource in Doit report from DB instance name

   INSTANCE_NAME=reader1

   CLUSTERID=$(aws rds describe-db-instances --db-instance-identifier $INSTANCENAME --query 'DBInstances[*].DBClusterIdentifier' --output text)

   aws rds describe-db-clusters --db-cluster-identifier $CLUSTER_ID --query 'DBClusters[*].DbClusterResourceId' --output text
   ```
   output:

   cluster-G2AKAT4P3JCOQDWPH6CSFTQPKQ

   With this information in the console, we could provide more "Insights" Eg “switch this cluster to I/O optimized”

   See also the thread: https://doitintl.slack.com/archives/C010V5SE3T5/p1733133308618059

   This is specific to how AWS RDS specifies the ARN of the instance vs the ARN of the cluster.

   When creating a Report in the console for RDS Aurora the cluster gets an ARN made up not of the cluster name, but of the cluster ID. The cluster ID is not obvious and must be searched in the AWS console or gotten via 2 API calls.
   It would be great if the DoiT console could automatically include the cluster ARN whenever an instance ARN is given, since the storage and backup SKUs (at least) are under the cluster ID ARN… more

   4 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   New  ·  0 comments  ·  Cloud Analytics & Reports  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

3. Allow permissions to be granted at the folder level rather than the org-level.

   Some companies have compliance/security concerns with granting organization-level-permissions, so I'm proposing that DoiT allows customers to be able to grant the necessary permissions for the DoiT Console at the folder-level. This would allow for additional flexibility for those customers that would like to leverage the full functionality of BQ Lens for specific projects while maintaining their organization's established security/compliance policies.

   2 votes

   Vote Vote Vote

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close

   Vote

   We’ll send you updates on this idea

   New  ·  0 comments  ·  BigQuery Lens  ·  Delete…  ·  Admin →
   How important is this to you?

   Not at all You must login first! Important You must login first! Critical You must login first!

   We're glad you're here

   Please sign in to leave feedback

   Signed in as (Sign out)

   Close

   Close