Splunk Observability Cloud (SignalFx)

Splunk Observability Cloud (formerly known as SignalFx) is a monitoring and analytics platform that allows customers to evaluate, visualize, automate and alert on metrics. By integrating Splunk Observability Cloud with Cortex, you can drive insights into SLOs.

Setup and configuration

Getting started

In order to connect Cortex to your Splunk Observability Cloud instance, you’ll need to create an access token.

Configuration

Once you've created an access token, go to Splunk Observability Cloud settings in Cortex and add the following information:

• Realm: Your Splunk Observability Cloud realm, which you can find in the URL for your instance (e.g. https://app..signalfx.com/#/metrics).

• Access token: The access token you created in Splunk.

If you’ve set everything up correctly, you’ll see the option to Remove Integration in settings.

You can also use the Test configuration button to confirm that the configuration was successful. If your configuration is valid, you’ll see a banner that says “Configuration is valid. If you see issues, please see documentation or reach out to Cortex support.”

Registration

Entity descriptor

You can pull in data about SLOs and manage them in Cortex by defining relevant SLIs through queries, along with a threshold and a comparator. Cortex will pull data from Splunk Observability Cloud and roll up the query with a specified rollup function.

x-cortex-slos:
  signalfx:
    - query: sf_metric:"jvm.memory.max" AND area:"nonheap"
      rollup: AVERAGE
      target: 5120000
      lookback: P1Y
      operation: "<="

Field	Description	Required
query	Elasticsearch query for your metric: Use sf_metric to filter by the given metric and add additional dimensions to narrow the searchQueries resulting in multiple datasets will be rolled up according to rollup	✓
rollup	SUM or AVERAGE	✓
target	Target number for the SLO	✓
lookback	ISO-8601 duration (P[n]Y[n]M[n]DT[n]H[n]M[n]S)	✓
operation	>, <, =, =	✓

Expected results

Entity pages

When an SLO is defined in an entity's descriptor, you'll see high-level information in the Monitoring block on an entity details page's overview.

Monitoring

You can access more detailed information about your SLO(s) in Monitoring > Splunk Observability Cloud in the entity's sidebar.

In addition to the query, target(s), and current value for each SLO, you'll find a graph of its performance over time.

Event timeline

Scorecards and CQL

With the Splunk Observability Cloud integration, you can create Scorecard rules and write CQL queries based on Splunk Observability Cloud SLOs.

See more examples in the CQL Explorer in Cortex.

SLOs

SLOs associated with the entity via ID or tags. You can use this data to check whether an entity has SLOs associated with it, and if those SLOs are passing.

Definition: slos: List<SLO>

Example

In a Scorecard, you can use this expression to make sure an entity is passing its SLOs:

slos().all((slo) => slo.passing) == true

Use this expression to make sure latency Service Level Indicator (SLI) value is above 99.99%:

slos().filter((slo) => slo.name.matchesIn("latency") and slo.sliValue >= 0.9999).length > 0

Still need help?​

The following options are available to get assistance from the Cortex Customer Engineering team:

• Email: [e-mail], or open a support ticket in the in app Resource Center

• Chat: Available in the Resource Center

• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.