Splunk Regex Cheat Sheet



@saranyaa21, based on the sample data provided, seems like you are traversing JSON data.Ideally for the sourcetype=serverlog, if you are only interested in JSON data, you should try either INDEXEDEXTRACTIONS=JSON or else KVMODE=json in your props.conf. But not both.In order to test the same at Search time you can try the following run anywhere search with spath to reveal all the.

  • LINEBREAKER = regular expression for event breaks TRUNCATE = 999999 (always a high number). Splunk® Data Onboarding Cheat Sheet (v2.5) v2.5.2. Review The Data.
  • Splunk-cheat-sheet AND,OR operator in splunk search Splunk Top command wildcards in splunk search dedup command head and tail stats eval Splunk Search book README.md Splunk-cheat-sheet.
  • Matches regex (2) regex: matches regex: In Splunk, regex is an operator. In Azure Monitor, it's a relational operator. Searchmatch In Splunk, searchmatch allows searching for the exact string. Random: rand rand(n) Splunk's function returns a number from zero to 2 31-1. Azure Monitor' returns a number between 0.0 and 1.0, or if a parameter.
  • So the next question popping in your mind is should i need to lean regex for using splunk.then answer is it depends.For using and operating splunk you do not need to learn regex in detail - basic knowledge will be ok.But if you want to become a skilled splunk admin then learning regex is necessary.

< Back |Home| Next >

Splunk regex tutorial | field extraction using regex

Regular expressions are extremely useful in extracting information from text such as code, log files, spreadsheets, or even documents.Regular expressions or regex is a specialized language for defining pattern matching rules .Regular expressions match patterns of characters in text. They have their own grammar and syntax rules.splunk uses regex for identifying interesting fields in logs like username,credit card number,ip address etc.By default splunk automatically extracts interesting fields and display them at left column is search result -only condition is log must contain key value pairs which means logs should contains field name and its value - like for username it should appear in log like usename=x or user:x.Extracted fields can be used later for sorting data,making specialized reports,creating valueable dashboards.But if logs do not contain field name in key value pair- like username or other fileds appears in log at random place then splunk will not detect the username automatically.In this condition regex comes for your help.You have to teach splunk to extract the field using regex.
So basically regex used to identify fields and list them in proper manner which later can be used for reporting,sorting and dashboard.Below image will help you in understanding the scenario
so the next question popping in your mind is should i need to lean regex for using splunk..then answer is it depends.For using and operating splunk you do not need to learn regex in detail - basic knowledge will be ok.But if you want to become a skilled splunk admin then learning regex is necessary.
Why Regex?

Regex is helpful in transforming your horrible looking machine logs into beautiful human understandable reports and dashboard -Easy to understand and use.Shown as below.

How to regex?

Splunk automatically identifies any fields that match its key/value pair intelligence, which can be found to the left of the search results as below. This can often allow you to start putting together useful data visualizations right out of the box.In below screenshot splunk has automatically extracted host,timestamp etc values.We can use these values for reporting,statistical analysis and creating dashboards.Splunk has inbuilt regex extractor called IFX (Interactive field extractor).By using IFX splunk autodetects useful fields and list them at left side.Splunk IFX can extract fields automatically which are in standard key value pair format i.e. key=value format like username=john etc.But if logs are not in key value pair format then you have to teach splunk to extract fields which you wan using regex.

We’re going to extract data that Splunk doesn’t recognize right away. There are a few of ways to do this, including using Splunk’s Interactive Field Extractor (IFX), or you can write your own regex (which I prefer)
How to extract fields using regex?

  • Good regex sites to help with Splunk
  • https://regex101.com/ - Great for general regex stuff and capture groups.
  • http://www.regexe.com/ - Great for dealing with capture groups in the way that Splunk likes them for anonymising data.
  • http://regexr.com/ - Classic website for quick PoC regexs.

Below are few commonly used regex notations while using extracting keywords using regex manually:

Splunk Regex Cheat Sheet Pdf


Use of special notations in regex:

Regex Cheat Sheet Pdf

Splunk Regex Cheat Sheet
Regex usage example
-->

This article is intended to assist users who are familiar with Splunk learn the Kusto Query Language to write log queries with Kusto. Direct comparisons are made between the two to highlight key differences and similarities, so you can build on your existing knowledge.

Structure and concepts

The following table compares concepts and data structures between Splunk and Kusto logs:

C# Regex Cheat Sheet

ConceptSplunkKustoComment
deployment unitclusterclusterKusto allows arbitrary cross-cluster queries. Splunk does not.
data cachesbucketscaching and retention policiesControls the period and caching level for the data. This setting directly affects the performance of queries and the cost of the deployment.
logical partition of dataindexdatabaseAllows logical separation of the data. Both implementations allow unions and joining across these partitions.
structured event metadataN/AtableSplunk doesn't expose the concept of event metadata to the search language. Kusto logs have the concept of a table, which has columns. Each event instance is mapped to a row.
data recordeventrowTerminology change only.
data record attributefieldcolumnIn Kusto, this setting is predefined as part of the table structure. In Splunk, each event has its own set of fields.
typesdatatypedatatypeKusto data types are more explicit because they're set on the columns. Both have the ability to work dynamically with data types and roughly equivalent set of datatypes, including JSON support.
query and searchsearchqueryConcepts essentially are the same between Kusto and Splunk.
event ingestion timesystem timeingestion_time()In Splunk, each event gets a system timestamp of the time the event was indexed. In Kusto, you can define a policy called ingestion_time that exposes a system column that can be referenced through the ingestion_time() function.

Functions

The following table specifies functions in Kusto that are equivalent to Splunk functions.

SplunkKustoComment
strcatstrcat()(1)
splitsplit()(1)
ififf()(1)
tonumbertodouble()
tolong()
toint()
(1)
upper
lower
toupper()
tolower()
(1)
replacereplace()(1)
Also note that although replace() takes three parameters in both products, the parameters are different.
substrsubstring()(1)
Also note that Splunk uses one-based indices. Kusto notes zero-based indices.
tolowertolower()(1)
touppertoupper()(1)
matchmatches regex(2)
regexmatches regexIn Splunk, regex is an operator. In Kusto, it's a relational operator.
searchmatchIn Splunk, searchmatch allows searching for the exact string.
randomrand()
rand(n)
Splunk's function returns a number between zero to 231-1. Kusto's returns a number between 0.0 and 1.0, or if a parameter is provided, between 0 and n-1.
nownow()(1)
relative_timetotimespan()(1)
In Kusto, Splunk's equivalent of relative_time(datetimeVal, offsetVal) is datetimeVal + totimespan(offsetVal).
For example, search | eval n=relative_time(now(), '-1d@d') becomes ... | extend myTime = now() - totimespan('1d').

(1) In Splunk, the function is invoked by using the eval operator. In Kusto, it's used as part of extend or project.
(2) In Splunk, the function is invoked by using the eval operator. In Kusto, it can be used with the where operator.

Operators

The following sections give examples of how to use different operators in Splunk and Kusto.

Note

In the following examples, the Splunk field rule maps to a table in Kusto, and Splunk's default timestamp maps to the Logs Analytics ingestion_time() column.

Search

In Splunk, you can omit the search keyword and specify an unquoted string. In Kusto, you must start each query with find, an unquoted string is a column name, and the lookup value must be a quoted string.

ProductOperatorExample
Splunksearchsearch Session.Id='c8894ffd-e684-43c9-9125-42adc25cd3fc' earliest=-24h
Kustofindfind Session.Id'c8894ffd-e684-43c9-9125-42adc25cd3fc' and ingestion_time()> ago(24h)

Filter

Kusto log queries start from a tabular result set in which filter is applied. In Splunk, filtering is the default operation on the current index. You also can use the where operator in Splunk, but we don't recommend it.

ProductOperatorExample
SplunksearchEvent.Rule='330009.2' Session.Id='c8894ffd-e684-43c9-9125-42adc25cd3fc' _indextime>-24h
KustowhereOffice_Hub_OHubBGTaskError
| where Session_Id 'c8894ffd-e684-43c9-9125-42adc25cd3fc' and ingestion_time() > ago(24h)

Get n events or rows for inspection

Kusto log queries also support take as an alias to limit. In Splunk, if the results are ordered, head returns the first n results. In Kusto, limit isn't ordered, but it returns the first n rows that are found.

ProductOperatorExample
SplunkheadEvent.Rule=330009.2
| head 100
KustolimitOffice_Hub_OHubBGTaskError
| limit 100

Get the first n events or rows ordered by a field or column

For the bottom results, in Splunk, you use tail. In Kusto, you can specify ordering direction by using asc.

ProductOperatorExample
SplunkheadEvent.Rule='330009.2'
| sort Event.Sequence
| head 20
KustotopOffice_Hub_OHubBGTaskError
| top 20 by Event_Sequence

Extend the result set with new fields or columns

Regex

Splunk has an eval function, but it's not comparable to the eval operator in Kusto. Both the eval operator in Splunk and the extend operator in Kusto support only scalar functions and arithmetic operators.

ProductOperatorExample
SplunkevalEvent.Rule=330009.2
| eval state= if(Data.Exception = '0', 'success', 'error')
KustoextendOffice_Hub_OHubBGTaskError
| extend state = iif(Data_Exception 0,'success' ,'error')
Splunk

Rename

Kusto uses the project-rename operator to rename a field. In the project-rename operator, a query can take advantage of any indexes that are prebuilt for a field. Splunk has a rename operator that does the same.

ProductOperatorExample
SplunkrenameEvent.Rule=330009.2
| rename Date.Exception as execption
Kustoproject-renameOffice_Hub_OHubBGTaskError
| project-rename exception = Date_Exception

Format results and projection

Splunk doesn't appear to have an operator that's similar to project-away. You can use the UI to filter out fields.

ProductOperatorExample
SplunktableEvent.Rule=330009.2
| table rule, state
Kustoproject
project-away
Office_Hub_OHubBGTaskError
| project exception, state

Aggregation

See the list of aggregations functions that are available.

ProductOperatorExample
Splunkstatssearch (Rule=120502.*)
| stats count by OSEnv, Audience
KustosummarizeOffice_Hub_OHubBGTaskError
| summarize count() by App_Platform, Release_Audience

Join

join in Splunk has substantial limitations. The subquery has a limit of 10,000 results (set in the deployment configuration file), and a limited number of join flavors are available.

ProductOperatorExample
SplunkjoinEvent.Rule=120103* &#124; stats by Client.Id, Data.Alias
| join Client.Id max=0 [search earliest=-24h Event.Rule='150310.0' Data.Hresult=-2147221040]
Kustojoincluster('OAriaPPT').database('Office PowerPoint').Office_PowerPoint_PPT_Exceptions
| where Data_Hresult -2147221040
| join kind = inner (Office_System_SystemHealthMetadata
| summarize by Client_Id, Data_Alias)on Client_Id

Sort

In Splunk, to sort in ascending order, you must use the reverse operator. Kusto also supports defining where to put nulls, either at the beginning or at the end.

ProductOperatorExample
SplunksortEvent.Rule=120103
| sort Data.Hresult
| reverse
Kustoorder byOffice_Hub_OHubBGTaskError
| order by Data_Hresult, desc

Multivalue expand

The multivalue expand operator is similar in both Splunk and Kusto.

Regex

Splunk Regex Cheat Sheet Pdf

ProductOperatorExample
Splunkmvexpandmvexpand solutions
Kustomv-expandmv-expand solutions

Result facets, interesting fields

In Log Analytics in the Azure portal, only the first column is exposed. All columns are available through the API.

ProductOperatorExample
SplunkfieldsEvent.Rule=330009.2
| fields App.Version, App.Platform
KustofacetsOffice_Excel_BI_PivotTableCreate
| facet by App_Branch, App_Version

Deduplicate

In Kusto, you can use summarize arg_min() to reverse the order of which record is chosen.

Splunk Regex Filter

ProductOperatorExample
SplunkdedupEvent.Rule=330009.2
| dedup device_id sortby -batterylife
Kustosummarize arg_max()Office_Excel_BI_PivotTableCreate
| summarize arg_max(batterylife, *) by device_id

Next steps

  • Walk through a tutorial on the Kusto Query Language.