在Ruby中随机生成有效的unicode字符

Question

How can I generate a random unicode string consisting of a given number of unicode characters in Ruby?

The following works, but includes control characters (0x00-0x1F, etc.) for instance:

20.times.map{ Random.rand(0xFFFF).chr('UTF-8')}.join

Answer 1

A lot of the characters in that range are not printable (as you've noted) or they are surrogate, custom, or otherwise invalid characters. The best approach (that I can think of) is to generate a sequence of characters, test each to make sure it's valid and printable, and then take the first 20 of them.

A few notes. We want to do rand(0x10000) in this case, not rand(0xFFFF) , because Random#rand and Kernel#rand will return a number less than its argument, and you want to include U+FFFF in your sampling. We should also give ourselves some flexibility to do one-byte, two-byte, three-byte, or four-byte UTF-8.

Let's start with a basic sequence generator, called an Enumerator in Ruby. This object yields values, one a time, and can represent a finite or infinite sequence. In this case, we want to enumerate an infinite sequence of random, three-byte UTF-8 characters, skipping invalid characters as we go.

random_utf8 = Enumerator.new do |yielder|
  loop do
    yielder << rand(0x10000).chr('UTF-8')
  rescue RangeError
  end
end

You can pull values off of the Enumerator with #next to see it in action:

irb(main):007:0> random_utf8.next
=> "\u9FEB"
irb(main):008:0> random_utf8.next
=> "槇"
irb(main):009:0> random_utf8.next
=> "엛"

(You'll notice that one of them didn't "render" because it's not a printable character. This illustrates why we need to filter the values before selecting 20 of them.)

Now we can take characters off this sequence and check each one to see if it's printable. The only catch is that we want to do this lazily , to avoid checking every character in the infinite sequence (which is impossible) before moving on to the next step in the chain. Finally, we'll take the first 20 printable characters and join them together into a string.

random_utf8
  .lazy
  .grep(/[[:print:]]/) # or [[:alpha:]] or \p{L} or whatever test you want here
  .first(20)
  .join # => "醸긍ᅋꝇ꼏捁㨃농鳹䝛ㆅ⇂擒璝缀챼砶"

Now let's abstract this into a method so we can parameterize some things. Ruby gives us a neat way to return an Enumerator from a method that yields values by returning Object#enum_for (aka Object#to_enum ) with the method symbol and any other arguments passed to the function.

def random_utf8(mb=3)
  return enum_for(__callee__, mb) unless block_given?

  # determine the maximum codepoint based on the number of UTF-8 bytes
  max = [0x80, 0x800, 0x10000, 0x110000][mb.pred]

  loop do
    yield rand(max).chr('UTF-8') # note the `yield` here
  rescue RangeError
  end
end

We can use this method exactly the same way we used our Enumerator above, optionally passing in the number of UTF-8 bytes desired.

This approach also gives us the option to call our method with a block instead of chaining operations off of it:

random_utf8(2) do |char|
  next unless char.match?(/[[:print:]]/)

  puts "Got >#{char}<!"

  break # don't loop infinitely
end

Which, admittedly, is not very useful in this particular case.

One additional note about the implementation of this solution: You could easily move the printable check into the method body, or move the RangeError exception handling out of the method body. You can also have the method return a lazy Enumerator by default. It's really up to you to design the method around your application requirements.

def lazy_printable_random_utf8(mb=3)
  return enum_for(__callee__, mb).lazy unless block_given?

  # determine the maximum codepoint based on the number of UTF-8 bytes
  max = [0x80, 0x800, 0x10000, 0x110000][mb.pred]

  loop do
    char = rand(max).chr('UTF-8')

    yield char if char.match?(/[[:print:]]/)
  rescue RangeError
  end
end