Great introduction to Splunk Configuration files and the precedence of the configuraiton files at run time. SO good graphics help describe the order that the configuration files are processed by Splunk.
If you change your configuration in Splunk Web, that modification is written to a copy of the configuration file for that setting. Splunk generates a copy of this configuration file and then writes the change to that copy, and adds it to a directory under the $SPLUNK_HOME/etc/…. The directory that the new file is added to will depend on a number of reasons. The most used Splunk directory is $SPLUNK_HOME/etc/system/local.
Always remember that you should not change, copy, or move the configuration files that are in the default directory. Default files must remain intact and in their original location. When you upgrade your Splunk software, the default directory is overwritten. Any changes that you make in the default directory are lost when you upgrade to a newer version of the software. Changes that you make in non-default configuration directories persist when you upgrade.
To change settings for a particular configuration file, you must first create a new version of the file in a non-default directory and then add the settings that you want to change. When you first create this new version of the file, start with an empty file. Do not start from a copy of the file in the default directory. For information on the directories where you can manually change configuration files, see Configuration file directories.