C# Char permutation with repetition on large set of chars

Question

Hello Im trying to get all possible combinations with repetitions of given char array. Char array consists of alphabet letters(only lower) and I need to generate strings with length of 30 or more chars.

I tried with method of many for-loops,but when I try to get all combinations of char in char array with length of string more then 5 I get out of Memory Exception.

So I created similar Method that takes only first 200000 strings,then next 2000000 and so on this was proven sucessfull but only with smaller length strings.

This was my method with length of 7 chars:

public static int Progress = 0;
public static ArrayList CreateRngUrl7()
        {

            ArrayList AllCombos = new ArrayList();
            int passed = 0;
            int Too = Progress + 200000;

            char[] alpha = "ABCDEFGHIJKLMNOPQRSTUVWXYZ".ToLower().ToCharArray();

            for (int i = 0; i < alpha.Length; i++)
                for (int i1 = 0; i1 < alpha.Length; i1++)
                    for (int i2 = 0; i2 < alpha.Length; i2++)
                        for (int i3 = 0; i3 < alpha.Length; i3++)
                            for (int i4 = 0; i4 < alpha.Length; i4++)
                                for (int i5 = 0; i5 < alpha.Length; i5++)
                                    for (int i6 = 0; i6 < alpha.Length; i6++)
                                {
                                    if (passed > (Too - 200000) && passed < Too)
                                    {
                                        string word = new string(new char[] { alpha[i], alpha[i1], alpha[i2], alpha[i3], alpha[i4], alpha[i5],alpha[i6] });
                                        AllCombos.Add(word);
                                    }

                                    passed++;
                                }
            if (Too >= passed)
            {
                MessageBox.Show("All combinations of RNG7 were returned");
            }
            Progress = Too;
            return AllCombos;
        }

I tried adding 30 for-loops with in way described above so i Would get strings with lenghts of 30 but application just hangs.Is there any better way to do this? All answers would be much appreciated. Thank you in advance!

Can someone please just post method how it is done with larger legth strings I just want to see an example? I don't have to store that data,I just need to compare it with something and release it from memory. I used alphabet for example I don't need whole alphabet.Question was not how long it would take or how much combinations would it be!!!!!

Answer 1

You get an OutOfMemoryException because inside the loop you allocate a string and store it in an ArrayList . The strings have to stay in memory until the ArrayList is garbage collected and your loop creates more strings than you will be able to store.

If you simply want to check the string for a condition you should put the check inside the loop:

for ( ... some crazy loop ...) {
  var word = ... create word ...
  if (!WordPassesTest(word)) {
    Console.WriteLine(word + " failed test.");
    return false;
  }
}
return true;

Then you only need storage for a single word. Of course, if the loop is crazy enough, it will not terminate before the end of the universe as we know it.

If you need to execute many nested but similar loops you can use recursion to simplify the code. Here is an example that is not incredible efficient, but at least it is simple:

Char[] chars = "ABCD".ToCharArray(); 

IEnumerable<String> GenerateStrings(Int32 length) {
  if (length == 0) {
    yield return String.Empty;
    yield break;
  }
  var strings = chars.SelectMany(c => GenerateStrings(length - 1), (c, s) => c + s);
  foreach (var str in strings)
    yield return str;
}

Calling GenerateStrings(3) will generate all strings of length 3 using lazy evaluation (so no additional storage is required for the strings).

Building on top of an IEnumerable generating your strings you can create primites to buffer and process buffers of strings. An easy solution is to using Reactive Extensions for .NET. Here you already have a Buffer primitive:

  GenerateStrings(3)
    .ToObservable()
    .Buffer(10)
    .Subscribe(list => ... ship the list to another computer and process it ...);

The lambda in Subscribe will be called with a List<String> with at most 10 strings (the parameter provided in the call to Buffer ).

Unless you have an infinte number of computers you will still have to pull the computers from a pool and only recycle them back to the pool when they have finished the computation.

It should be obvious from the comments on this question that you will not be able to process 26^30 strings even if you have multiple computers at your disposal.

Answer 2

I don't have time right now to write some code but essentially if you are running out of RAM use disk. I'm thinking along the lines of one thread running an algorithm to find the combinations and another persisting the results to disk and releasing the RAM.