If L and L complement are Recursively enumerable then why can't L be a Regular language?

Question

Below question was asked in GATE 2008 paper :

If L and L' (L complement) are Recursively enumerable then L is ?

a) Regular
b) CFL
c) CSL
d) Recursive

Correct option was option (d) and I accept that it's true. But my question is why can't it be regular or CSL ?

Because I think if we consider L is regular, then L' is also regular (As Regular languages are closed under complementation) . And now as L' is regular so according to 'Chomsky hierarchy' L' is also Recursively enumerable. As even L after being regular, it fits into the question statement then why option (a) is not a correct option ? Same goes for CSL, so why option (c) is also not a correct option?

Answer 1

A quick review of language classes -- we know that these 5 language classes are all (strict) subsets of each other:

regular ⊂ CFL ⊂ CSL ⊂ recursive ⊂ recursively enumerable

The question is asking, if we know a language L is recursively enumerable AND we know it's complement L' is also recursively enumerable, what can we say for certain which smaller class L is in?

The answer is equivalent to saying that if a language L is recursively enumerable and NOT recursive, then L' is NOT recursively enumerable. That statement is true, but the equivalent statement for any of the other language classes is not.

Answer 2

If L and L' are both recursively enumerable, then

a) L may be regular (indeed, if L is regular, then L' is regular as well, and all regular languages are recursively enumerable)... but there are non-regular languages whose complements are recursively enumerable

b) L may be a CFL (there are CFLs whose complements are also CFLs, as well as CFLs whose complements are not CFLs)... but there are non-context-free languages whose complements are recursively enumerable

c) L may be a CSL (there are CSLs whose complements are CSLs) ... but there are non-context-sensitive languages whose complements are recursively enumerable

d) L must be recursive because, by virtue of both L and L' being recursively enumerable, we have an effectively computable procedure for deciding whether or not any given string is in L: begin enumerating strings in each language, interleaving the enumerations (so you give the next string in L, then the next string in L', then back to L, etc.). Continuing this process will eventually find the target string either in L or L', at which point you can return true (if it was enumerated in L) or false (if enumerated in L').

Therefore, while it's true that L could be regular, CFL or CSL, it is also true that it might not be any of those; but it must absolutely be recursive. Therefore, that is the "best" answer and the only one that is generally correct in all cases.