Remove excessive whitespace in user input field

Question

In my controller method for handling a (potentially hostile) user input field I have the following code:

string tmptext = comment.Replace(System.Environment.NewLine, "{break was here}"); //marks line breaks for later re-insertion
tmptext = Encoder.HtmlEncode(tmptext);
//other sanitizing goes in here 
tmptext = tmptext.Replace("{break was here}", "<br />");

var regex = new Regex("(<br /><br />)\\1+");
tmptext = regex.Replace(tmptext, "$1");

My goal is to preserve line breaks for typical non-malicious use and display user input in safe, htmlencoded strings. I take the user input, parse it for newline characters and place a delimiter at the line breaks. I perform the HTML encoding and reinsert the breaks. (i will likely change this to reinserting paragraphs as p tags instead of br, but for now i'm using br)

Now actually inserting real html breaks opens me up to a subtle vulnerability: the enter key. The regex.replace code is there to strip out a malicious user just standing on the enter key and filling the page with crap.

This is a fix for big crap floods of just white but still leaves me open to abuse like entering one character, two line breaks, one character, two line breaks all down the page.

My question is for a method of determining that this is abusive and failing it on validation. I'm scared that there might not be a simple procedural method to do it and instead will need heuristic techniques or bayesian filters. Hopefully, someone has an easier, better way.

EDIT: perhaps I wasn't clear in the problem description, the regex handles seeing multiple line breaks in a row and converting them to just one or two. That problem is solved. The real problem is distinguishing legitimate text from crap flood like this:

a

a

a

...imagine 1000 of these...

a

a

a

a

Answer 1

It sounds like you're tempted to try something "clever" with a regex, but IMO the simplest approach is to just loop through the characters of the string copying them to a StringBuilder, filtering as you go.

Any that fail a char.IsWhiteSpace() test are not copied. (If one of these is a newline, then insert a <br/> and don't allow any more <br/>'s to be added until you have hit a non-whitespace character).

edit

If you want to stop the user entering any old crap, give up now. You will never find a way filtering that a user can't find a way around in less than a minute, if they really want to.

You will be much better off putting a limit on the number of newlines, or the total number of characters, in the input.

Think of how much effort it will take to do something clever to sanitise "bad input", and then consider how likely it is that this will happen. Probbaly there is no point. Probably all the sanitisation you really need is to ensure the data is legal (not too large for your system to handle, all dangerous characters stripped or escaped, etc). (This is exactly why forums have human moderators who can filter the posts based on whatever criteria are approriate).

Answer 2

I would HttpUtility.HtmlEncode the string, then convert newline characters to <br/> .

HttpUtility.HtmlEncode(subject).Replace("\r\n", "<br/>").Replace("\r", "<br/>").Replace("\n", "<br/>");

Also you should perform this logic when you are outputting to the user, not when saving in the database. The only validation I do on the database is make sure it's properly escaped (other than normal business rules that is).

EDIT : To fix the actual problem however, you can use Regex to replace multiple newlines with a single newline beforehand.

subject = Regex.Replace(@"(\r\n|\r|\n)+", @"\n", RegexOptions.Singleline);

I'm not sure if you would need RegexOptions.Singleline .

Answer 3

Rather than attempting to replace the newlines with filtered text and then attempting to use regular expressions on that, why not sanitize your data before inserting the <br /> tags? Don't forget to sanitize the input with HttpUtility.HtmlEncode first.

In an attempt to take care of multiple short lines in a row, here's my best attempt:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

class Program {
  static void Main() {
    // Arbirary cutoff used to join short strings.
    const int Cutoff = 6;

    string input =
      "\r\n\r\n\n\r\r\r\n\nthisisatest\r\nstring\r\nwith\nsome\r\n" + 
      "unsanatized\r\nbreaks\r\nand\ra\nsh\nor\nt\r\n\na\na\na\na" +
      "\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na";
    input = (input ?? String.Empty).Trim(); // Don't forget to HtmlEncode it.
    StringBuilder temp = new StringBuilder();
    List<string> result = new List<string>();
    var items = input.Split(
                        new[] { '\r', '\n' },
                        StringSplitOptions.RemoveEmptyEntries)
                     .Select(i => new { i.Length, Value = i });

    foreach (var item in items) {
      if (item.Length > Cutoff) {
        if (temp.Length > 0) {
          result.Add(temp.ToString());
          temp.Clear();
        }

        result.Add(item.Value);
        continue;
      }

      if (temp.Length > 0) { temp.Append(" "); }
      temp.Append(item.Value);
    }

    if (temp.Length > 0) {
      result.Add(temp.ToString());
    }

    Console.WriteLine(String.Join("<br />", result));
  }
}

Produces the following output:

thisisatest<br />string with some<br />unsanatized<br />breaks and a sh or t a a
 a a a a a a a a a a a a a a a a a a a

I'm sure you've already come up with this solution but unfortunately what you're asking for isn't very straight forward.

For those interested, here's my first attempt:

using System;
using System.Text.RegularExpressions;

class Program {
  static void Main() {
    string input = "\r\n\r\n\n\r\r\r\n\nthisisatest\r\nstring\r\nwith\nsome" +
                   "\r\nunsanatized\r\nbreaks\r\n\r\n";
    input = (input ?? String.Empty).Trim().Replace("\r", String.Empty);
    string output = Regex.Replace(
                      input,
                      "\\\n+",
                      "<br />",
                      RegexOptions.Multiline);
    Console.WriteLine(output);
  }
}

producing the following output:

thisisatest<br />string<br />with<br />some<br />unsanatized<br />breaks

Answer 4

This is not the most efficient way of handling this, nor the smartest (disclaimer),
but if your text is not too big it doesn't matter much and short of any smarter algorithms (note: it's hard to detect something like char\\nchar\\nchar\\n... though you could set a limit on the line len)

You could just Split on white characters (add any you can think of, short of \\n) - then Join with just one space and then split on \\n (to get lines) - join with <br /> . While joining the lines you can test for line.Length > 2 eg or something.

To make this faster you can iterate with a more efficient algorithm, char by char, using IndexOf etc..

Again not the most efficient or perfect way of handling this but would give you something fast.

EDIT: to filter 'same lines' - you could use eg DistinctUntilChanged - that's from the Ix - Interactive extensions (see NuGet Ix-experimental I think) which should filter 'same lines' consecutive + you could add line test for those.

Answer 5

受到slashdot.org的评论过滤器启发的随机建议：使用System.IO.Compression.DeflateStream压缩用户输入，并且如果与原始输入相比太小（您必须做一些实验才能找到有用的截止）拒绝它。