grafana issue introduced with 5.1.1-1

[rhclopes]

After the update to 5.1.1-1, a dashboard lost access to the host info data. You can see it here

https://ds2-london.perf.ja.net/grafana/d/eb96563b-0d93-4910-be3c-5be331a00339/perfsonar-host-metrics?orgId=1

This was working perfectly two weeks ago. And it hasn't been touched in several months.

The logs show an authentication error every time that page is reloaded

logger=context userId=0 orgId=1 uname= t=2024-07-16T08:14:14.718483139+01:00 level=info msg="Request Completed" method=GET path=/api/live/ws status=400 remote_addr=2a02:8010:65d4:0:97ec:14be:b952:5c1 time_ms=1 duration=1.439603ms size=12 referer= handler=/api/live/ws status_source=server
logger=authn.service t=2024-07-16T08:14:19.528405891+01:00 level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2024-07-16T08:14:19.96055322+01:00 level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"

Raul

[rhclopes]

That grafana dashboard now showing "No data" in all dashboards. And there is an error in logstash

[2024-07-26T08:43:36,776][WARN ][logstash.outputs.opensearch][prometheus][f352b446b29edba5f56d6ac2d100edf5642e0057a1259ec6eb8f652c6d6c5d12] Could not index event to OpenSearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"prometheus_node", :routing=>nil}, {"values"=>{"promhttp_metric_handler_requests_total"=>{"rate"=>0.0, "val"=>0.0, "delta"=>0.0}}, "meta"=>{"labels"=>{"code"=>"503"}, "id"=>"localhost", "os"=>{"variant"=>"", "image_version"=>"", "id"=>"almalinux", "version_id"=>"9.4", "build_id"=>"", "version"=>"9.4 (Seafoam Ocelot)", "pretty_name"=>"AlmaLinux 9.4 (Seafoam Ocelot)", "image_id"=>"", "name"=>"AlmaLinux", "version_codename"=>"", "id_like"=>"rhel centos fedora", "variant_id"=>""}, "uname"=>{"sysname"=>"Linux", "machine"=>"x86_64", "nodename"=>"ds2-london.perf.ja.net", "release"=>"5.14.0-427.26.1.el9_4.x86_64", "version"=>"#1 SMP PREEMPT_DYNAMIC Wed Jul 17 15:51:13 EDT 2024", "domainname"=>"(none)"}}, "type"=>"node", "@version"=>"1", "is_local"=>"true", "@timestamp"=>2024-07-26T07:43:36.485218380Z}], :response=>{"create"=>{"_index"=>".ds-prometheus_node-000001", "_id"=>"X03-7ZABa2NBmkoSyqvV", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Number of documents in shard [.ds-prometheus_node-000001][0] exceeds the limit of [2147483519] documents per shard"}}}}

Also in /var/log/perfsonar/psconfig-grafana-agent.log

2024-05-23 16:31:58 ERROR pid=517306 prog=_gf_update_datasource line=613 guid=222b1dfd-75ea-40bc-b5ee-e3cd78188cf7 config_src=remote config_url=https://ds2-london.perf.ja.net/psconfig/ds2-psconfig.json msg=Unable to update datasource for https://ds2-london.perf.ja.net/opensearch: HTTP Error - 403 Client Error: Forbidden for url: https://localhost/grafana/api/datasources/uid/cdl3zlgb9rg8we

and in /var/log/perfsonar/psconfig-grafana-agent-transactions.log

2024-05-23 15:31:54 ERROR guid=c23d4400-73e2-4c50-9ae0-47b802a2c305 config_src=remote config_url=https://ds2-london.perf.ja.net/psconfig/ds2-psconfig.json grafana_url=https://localhost/grafana/api/datasources/uid/cdl3zlgb9rg8we action=update_datasource.error data={"type": "grafana-opensearch-datasource", "url": "https://ds2-london.perf.ja.net/opensearch", "user": "", "database": "", "basicAuth": false, "isDefault": false, "access": "proxy", "jsonData": {"database": "pscheduler*", "flavor": "opensearch", "maxConcurrentShardRequests": 5, "pplEnabled": true, "timeField": "pscheduler.start_time", "tlsAuth": false, "tlsSkipVerify": true, "version": "2.13.0"}, "readOnly": false, "name": "pSConfig pScheduler - https://ds2-london.perf.ja.net/opensearch", "uid": "cdl3zlgb9rg8we"} msg=HTTP Error - 403 Client Error: Forbidden for url: https://localhost/grafana/api/datasources/uid/cdl3zlgb9rg8we

Raul

[rhclopes]

This was solved after 5.1.2