![]() Some examples of these commands include dedup, table, and stats. The more work you can push out to the indexers, the better the search performance, because it’s distributed amongst more systems. ![]() Centralized streaming and transforming commands run in the reduce part of the process (on the search head), and force any SPL that comes after them in the pipeline to do the same. Splunk achieves a good deal of its performance by distributing part of the search out to the indexers, and then aggregating the results on the search head. These commands are basically the arch nemesis of mapreduce. Two of those shapes are “centralized streaming” and “transforming”. Splunk commands come in lots of shapes and sizes. This is disk I/O that could be better spent on just about anything else. ![]() Unfortunately, it means “look at all of my events and discard the ones that don’t start with rsa”. “sourcetype=rsa*” doesn’t mean “look at my list of sourcetypes, get the ones that start with rsa, and search for those”. Splunk lets you use wildcards, but it doesn’t use them very efficiently. Rule #2: Avoid wildcards like the plague: Think of these fields like the fields in an “index” on a database table - using them is a really fast way to access your data. However, generally most fields are “search time” fields. Sometimes, other fields are “indexed” fields as well, depending on the source (structured logs such as CSV or JSON are often configured as “indexed” fields). Other useful fields are “sourcetype”, “source” and “host” - these are “indexed” fields, meaning that they are actually stored to disk when the data is received, rather than being calculated at search time. As it stands, this search will basically run forever when subjected to “All Time” on 13 months / 14TB of data:Īdding “index=” to your search is the single best thing you can do to improve your search - I was able to convert a never-ending search to one that completed in less than 60 seconds by simply adding an index restriction to it. But, what do these primal instincts look like? I’ve boiled them down into a few simple rules for turning an “All Time” search over index=wineventlog from a “Nightmare on SPL Street” into “Done in 5600 seconds” (are these movie puns doing anything for you? We should hang out more).Īs I go through these rules, think about how you would apply them to this search in order to improve its performance. It’s times like this when it becomes important to harken back to your primal roots, and forego all modern conveniences - in an attempt to get back to the basics of searching - like field extractions and tags. Cases where even the most seasoned of SPL ( Search Processing Language) authors run screaming. However, as with any situation, there are edge cases… Cases where you need to search through 13 months of WinEventLog data, totalling over 14TB. There are plenty of tutorials out there that explain how to optimize your Splunk search, and for the most part they do a really good job. Let’s take a moment to venture back to our primal roots. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |