Ruby's String#gsub, unicode, and non-word characters

Question

As part of a larger series of operations, I'm trying to take tokenized chunks of a larger string and get rid of punctuation, non-word gobbledygook, etc. My initial attempt used String#gsub and the \\W regexp character class, like so:

my_str = "Hello,"
processed = my_str.gsub(/\W/,'')
puts processed # => Hello

Super, super, super simple. Of course, now I'm extending my program to deal with non-Latin characters, and all heck's broken loose. Ruby's \\W seems to be something like [^A-Za-z0-9_] , which, of course, excludes stuff with diacritics (ü, í, etc.). So, now my formerly-simple code crashes and burns in unpleasent ways:

my_str = "Quística."
processed = my_str.gsub(/\W/,'')
puts processed # => Qustica

Notice that gsub() obligingly removed the accented "í" character. One way I've thought of to fix this would be to extend Ruby's \\W whitelist to include higher Unicode code points, but there are an awful lot of them, and I know I'd miss some and cause problems down the line (and let's not even start thinking about non-Latin languages...). Another solution would be to blacklist all the stuff I want to get rid of (punctuation, $/%/&/™, etc.), but, again, there's an awful lot of that and I really don't want to start playing blacklist-whack-a-mole.

Has anybody out there found a principled solution to this problem? Is there some hidden, Unicode-friendly version of \\W that I haven't discovered yet? Thanks!

Answer 1

You need to run ruby with the "-Ku" option to make it use UTF-8. See the documentation for command-line options . This is what happens when I do this with irb:

% irb -Ku
irb(main):001:0> my_str = "Quística."
=> "Quística."
irb(main):002:0> processed = my_str.gsub(/\W/,'')
=> "Quística"
irb(main):003:0> 

You can also put it on the #! line in your ruby script:

#!/usr/bin/ruby -Ku

Answer 2

I would just like to add that in 1.9.1 it works by default.

$ irb
ruby-1.9.1-p243 > my_str = "Quística."
=> "Quística."
ruby-1.9.1-p243 > processed = my_str.gsub(/\W/,'')
=> "Quística"
ruby-1.9.1-p243 > processed.encoding
=> #<Encoding:UTF-8>

PS. Nothing beats rvm for trying out different versions of Ruby. DS.