Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CSM O11y test] Filter out time series with no valid data point #32

Merged

Conversation

stanley-cheung
Copy link
Contributor

@stanley-cheung stanley-cheung commented Feb 7, 2024

  • From a prod test failure, we observed that there may be time series data in a metric that contain no valid data points. It's unclear why they come to be at the moment
  • This PR attempts to not count time series when there is no valid data point in the time series

Can't really do the filtering at the query level because the monitoring API does not support filtering by data points: https://cloud.google.com/monitoring/api/v3/filters

- From a prod test failure, we observed that there may be time series data in a metric that contain no valid data points. It's unclear why they come to be at the moment
- This PR attempts to not count time seires when there is no valid data point in the time series
@stanley-cheung
Copy link
Contributor Author

stanley-cheung commented Feb 7, 2024

Note that this did not really address the root cause as in why are there an extra time series with no valid data point.

[ Edit: #33 will attempt to dump the GMP output during test ]

- some pythonistic fixes
Copy link
Member

@sergiitk sergiitk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good - will enable PR auto-merge, but block on the unresolved conversation below.

@sergiitk sergiitk enabled auto-merge (squash) February 7, 2024 23:37
@sergiitk sergiitk merged commit a80dd7f into grpc:main Feb 8, 2024
6 checks passed
@stanley-cheung stanley-cheung deleted the filter-out-time-series-with-no-data-point branch February 8, 2024 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants