Community:WindowsCharacterEncoding

From Splunk Wiki

Jump to: navigation, search

< Back to Best Practices

What to do if Windows logs show up as x00 or not at all

Problem

If you are trying to input data from Windows and Splunk is

a) refusing to index the data

b) showing x00 instead of actual characters

This may be behavior resulting from current issues handling UTF-16 encoded text data. You may need to manually specify that the events are encoded in UTF-16.

This is the normal (and only, afaik) character set encoding for mssql 2005 (also sharepoint 2007 and probably all other newer-vintage microsoft applications).

Solution

Verify that the data is in UTF-16 format first. Nulls can occur for other reasons.

Note: For UTF-16 input, your splunk instances should be on 3.4.9 or later. As usual, receiving hosts should be on the same or a newer version than the forwarder. For Lightweight Forwarders, you should be on 3.4.10 or later.

a) Splunk may be classifying the file as 'binary', so you need to disable this check.

If you run this command from the $SPLUNK_HOME\bin, you'll see how Splunk is/should be classifying your file -

splunk cmd classify <filename>

b) We can't change the way SQL Server formats its own error log file, but we can change how Splunk is reading it by specifying a different character set in $SPLUNK_HOME/etc/system/local/props.conf.

[source::<path\\to\\files>]
CHARSET=UTF-16LE
NO_BINARY_CHECK = true

Note: currently it is insufficient to set the CHARSET at the sourcetype level, it must be set on a source basis.

Here are some links to more details on the subject.

3.4.8 and earlier:

Although this should work for you, there may be a point at which it breaks and you end up with malformed data in your index. If that happens, you may find yourself in the position of having to clean data from your index. To make this as painless as possible, we suggest that you use a separate index, or Splunk instance for your SQL Error logs, and back up your indexed data regularly.

Personal tools
Hot Wiki Topics


About Splunk >
  • Search and navigate IT data from applications, servers and network devices in real-time.
  • Download Splunk