[英]How can I randomize the lines in a file using standard tools on Red Hat Linux?
How can I randomize the lines in a file using standard tools on Red Hat Linux? 如何使用Red Hat Linux上的标准工具随机化文件中的行?
I don't have the shuf
command, so I am looking for something like a perl
or awk
one-liner that accomplishes the same task. 我没有
shuf
命令,所以我正在寻找像perl
或awk
one-liner这样的东西来完成同样的任务。
嗯,别忘了
sort --random-sort
shuf
is the best way. shuf
是最好的方式。
sort -R
is painfully slow. sort -R
非常缓慢。 I just tried to sort 5GB file. 我只是尝试排序5GB文件。 I gave up after 2.5 hours.
我放弃了2.5小时后。 Then
shuf
sorted it in a minute. 然后
shuf
在一分钟内对它进行了分类。
And a Perl one-liner you get! 你得到一个Perl单线程!
perl -MList::Util -e 'print List::Util::shuffle <>'
It uses a module, but the module is part of the Perl code distribution. 它使用模块,但该模块是Perl代码分发的一部分。 If that's not good enough, you may consider rolling your own.
如果这还不够好,你可以考虑自己动手。
I tried using this with the -i
flag ("edit-in-place") to have it edit the file. 我尝试使用
-i
标志(“编辑就地”)来编辑文件。 The documentation suggests it should work, but it doesn't. 文档表明它应该有效,但事实并非如此。 It still displays the shuffled file to stdout, but this time it deletes the original.
它仍然将混洗文件显示到stdout,但这次它会删除原始文件。 I suggest you don't use it.
我建议你不要用它。
Consider a shell script: 考虑一个shell脚本:
#!/bin/sh
if [[ $# -eq 0 ]]
then
echo "Usage: $0 [file ...]"
exit 1
fi
for i in "$@"
do
perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new
if [[ `wc -c $i` -eq `wc -c $i.new` ]]
then
mv $i.new $i
else
echo "Error for file $i!"
fi
done
Untested, but hopefully works. 未经测试,但希望有效。
cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7-
Read the file, prepend every line with a random number, sort the file on those random prefixes, cut the prefixes afterwards. 读取文件,在每行前面加一个随机数,在这些随机前缀上对文件进行排序,然后剪切前缀。 One-liner which should work in any semi-modern shell.
单衬里应该适用于任何半现代的外壳。
EDIT: incorporated Richard Hansen's remarks. 编辑:纳入理查德汉森的言论。
A one-liner for python: python的单线程:
python -c "import random, sys; lines = open(sys.argv[1]).readlines(); random.shuffle(lines); print ''.join(lines)," myFile
And for printing just a single random line: 并且只打印一条随机线:
python -c "import random, sys; print random.choice(open(sys.argv[1]).readlines())," myFile
But see this post for the drawbacks of python's random.shuffle()
. 但是请看这篇文章 ,了解python的
random.shuffle()
的缺点。 It won't work well with many (more than 2080) elements. 它不适用于许多(超过2080个)元素。
Related to Jim's answer: 与Jim的回答有关:
My ~/.bashrc
contains the following: 我的
~/.bashrc
包含以下内容:
unsort ()
{
LC_ALL=C sort -R "$@"
}
With GNU coreutils's sort, -R
= --random-sort
, which generates a random hash of each line and sorts by it. 使用GNU coreutils的排序,
-R
= --random-sort
,它生成每行的随机散列并按其排序。 The randomized hash wouldn't actually be used in some locales in some older (buggy) versions, causing it to return normal sorted output, which is why I set LC_ALL=C
. 随机散列实际上不会在某些较旧(错误)版本的某些语言环境中使用,导致它返回正常的排序输出,这就是我设置
LC_ALL=C
。
Related to Chris's answer: 与Chris的回答有关:
perl -MList::Util=shuffle -e'print shuffle<>'
is a slightly shorter one-liner. 是一个稍短的单线。 (
-Mmodule=a,b,c
is shorthand for -e 'use module qw(abc);'
.) (
-Mmodule=a,b,c
是-e 'use module qw(abc);'
缩写。)
The reason giving it a simple -i
doesn't work for shuffling in-place is because Perl expects that the print
happens in the same loop the file is being read, and print shuffle <>
doesn't output until after all input files have been read and closed. 给它一个简单的
-i
的原因不适用于就地改组是因为Perl期望print
发生在正在读取文件的同一循环中,并且print shuffle <>
直到所有输入文件都具有之后才输出已阅读并关闭。
As a shorter workaround, 作为一个较短的解决方法,
perl -MList::Util=shuffle -i -ne'BEGIN{undef$/}print shuffle split/^/m'
will shuffle files in-place. 将就地文件洗牌。 (
-n
means "wrap the code in a while (<>) {...}
loop; BEGIN{undef$/}
makes Perl operate on files-at-a-time instead of lines-at-a-time, and split/^/m
is needed because $_=<>
has been implicitly done with an entire file instead of lines.) (
-n
表示“将代码包装在while (<>) {...}
循环中; BEGIN{undef$/}
使Perl一次对文件进行操作,而不是一次一行地进行操作,并且split/^/m
,因为$_=<>
已隐式使用整个文件而不是行。)
When I install coreutils with homebrew 当我用自制软件安装coreutils时
brew install coreutils
shuf
becomes available as n
. shuf
成为n
。
Mac OS X with DarwinPorts: 带有DarwinPorts的Mac OS X:
sudo port install unsort
cat $file | unsort | ...
FreeBSD has its own random utility: FreeBSD有自己的随机工具:
cat $file | random | ...
It's in /usr/games/random, so if you have not installed games, you are out of luck. 它在/ usr / games / random中,所以如果你还没有安装游戏,那你就不走运了。
You could consider installing ports like textproc/rand or textproc/msort. 您可以考虑安装textproc / rand或textproc / msort等端口。 These might well be available on Linux and/or Mac OS X, if portability is a concern.
如果可移植性是一个问题,这些可能在Linux和/或Mac OS X上可用。
On OSX, grabbing latest from http://ftp.gnu.org/gnu/coreutils/ and something like 在OSX上,从http://ftp.gnu.org/gnu/coreutils/获取最新信息
./configure make sudo make install ./configure make sudo make install
...should give you /usr/local/bin/sort --random-sort ...应该给你/ usr / local / bin / sort --random-sort
without messing up /usr/bin/sort 没有弄乱/ usr / bin / sort
Or get it from MacPorts: 或者从MacPorts获取:
$ sudo port install coreutils
and/or 和/或
$ /opt/local//libexec/gnubin/sort --random-sort
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.