Cody’s Data Cleaning Techniques Using SAS® Software

Checking a Range Using an Algorithm Based on Standard Deviation

One way of deciding what constitutes reasonable cutoffs for low and high data values is to use an algorithm based on the data values themselves. For example, you could decide to flag all values more than two standard deviations from the mean. However, if you had some severe data errors, the standard deviation could be so badly inflated that obviously incorrect data values might lie within two standard deviations. A possible workaround for this would be to compute the standard deviation after removing some of the highest and lowest values. For example, you could compute a standard deviation of the middle 50% of your data and use this to decide on outliers. Another popular alternative ...