[英]Python Wand and ImageMagick on AWS Lambda
Trying to run Wand on AWS Lambda for some simple image manipulation.尝试在 AWS Lambda 上运行 Wand 以进行一些简单的图像处理。
So far I've:到目前为止,我已经:
installed imagemagick from source从源安装 imagemagick
wget ImageMagick cd ImageMagick-... ./configure --prefix=/home/ec2-user/im --without-x make && make install
set MAGICK_HOME to /home/ec2-user/im/
将 MAGICK_HOME 设置为
/home/ec2-user/im/
pip install wand
but when I try to import wand.image I get:但是当我尝试导入 wand.image 时,我得到:
import wand.image
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/cv/local/lib64/python2.7/site-packages/wand/image.py", line 20, in <module>
from .api import MagickPixelPacket, libc, libmagick, library
File "/home/ec2-user/cv/local/lib64/python2.7/site-packages/wand/api.py", line 206, in <module>
'Try to install:\n ' + msg)
ImportError: MagickWand shared library not found.
You probably had not installed ImageMagick library.
Try to install:
http://docs.wand-py.org/en/latest/guide/install.html
It works fine when I do a yum install ImageMagick
, but since I want to bundle it I thought I'd need to include the binary and point the wand installation to it.当我执行
yum install ImageMagick
时它工作正常,但由于我想捆绑它,我想我需要包含二进制文件并将魔杖安装指向它。
Any thoughts?有什么想法吗?
Since someone else is interested now, I solved this issue and currently manage my lambda projects similar to below.由于现在有人感兴趣,我解决了这个问题,目前管理我的 lambda 项目类似于下面。
folder structure文件夹结构
-project
-init_libs.py
-lambda_function.py
-libs
-libMagickWand-6.Q16.so
-libMagickCore-6.Q16.so
-libMagick++-6.Q16.so
-site-packages
-wand
-other_modules
init_libs.py
loads native libraries like Wand. init_libs.py
加载像 Wand 这样的原生库。
import sys
import os
from ctypes import cdll
from os.path import join
# manually load libraries and set paths so native libraries can be used.
exec_dir = os.getcwd()
sys.path.append(join(exec_dir, 'site-packages'))
# required for Wand
os.environ['MAGICK_HOME'] = exec_dir
lib_dir = join(exec_dir, 'lib')
# add lib to sys path for general loading
sys.path.append(lib_dir)
lambda_function.py
can now import wand lambda_function.py
现在可以导入魔杖
from wand.image import Image
print Image
This may not be helpful to the question asked, but if the desired goal is to do some simple file conversion (pdf to jpg/png etc.) one can bypass wand and invoke ImageMagick/GhostScript directly (these are packaged with the Python3.6 AWS environment).这可能对提出的问题没有帮助,但是如果所需的目标是进行一些简单的文件转换(pdf 到 jpg/png 等),则可以绕过 wand 并直接调用 ImageMagick/GhostScript(这些与 Python3.6 一起打包) AWS 环境)。
eg例如
# converts local pdf to jpg using ImageMagick
os.system("convert -density 200x200 /tmp/in_pdf.pdf /tmp/out_jpg.jpg")
# converts local pdf to jpg using GhostScript
os.system(
"gs -q -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT "
"-dMaxBitmap=500000000 -dAlignToPixels=0 -dGridFitTT=2 "
"-sDEVICE=jpeg -dTextAlphaBits=4 -dGraphicsAlphaBits=4 -r150 "
"-sOutputFile=/tmp/out_jpg-%d.jpg /tmp/in_pdf.pdf"
)
The reason I am using /tmp
as a file location is because lambda allows you to store files up to 500MB there during the running of the lambda function.我使用
/tmp
作为文件位置的原因是因为 lambda 允许您在运行 lambda 函数期间在那里存储高达 500MB 的文件。
I hope this helps someone!我希望这可以帮助别人!
Inspired from https://gist.github.com/bensie/56f51bc33d4a55e2fc9a#gistcomment-3133859 , I was able to successfully package ImageMagick and Ghostscript in a Python 3.8 compatible AWS Lambda layer and use wand
to convert the pages in a PDF into PNG by following these instructions:受https://gist.github.com/bensie/56f51bc33d4a55e2fc9a#gistcomment-3133859 的启发,我能够在 Python 3.8 兼容的 AWS Lambda 层中成功打包 ImageMagick 和 Ghostscript,并使用
wand
将 PDF 中的页面转换为 PNG遵循以下说明:
amzn2-ami-hvm-2.0.20210126.0-x86_64-gp2
.amzn2-ami-hvm-2.0.20210126.0-x86_64-gp2
。 wget https://download.imagemagick.org/ImageMagick/download/ImageMagick-6.9.11-60.tar.gz
tar zxvf ImageMagick-6.9.11-60.tar.gz
cd
into the extracted folder. cd
进入解压后的文件夹。 cd ImageMagick-6.9.11-60
policy.xml
file to allow PDF to PNG conversion.policy.xml
文件以允许将 PDF 转换为 PNG。 nano config/policy.xml
I copy-pasted the following content but you can modify it as needed. <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE policymap [ <!ELEMENT policymap (policy)+> <!ELEMENT policy (#PCDATA)> <!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED> <!ATTLIST policy name CDATA #IMPLIED> <!ATTLIST policy rights CDATA #IMPLIED> <!ATTLIST policy pattern CDATA #IMPLIED> <!ATTLIST policy value CDATA #IMPLIED> ]> <!-- Configure ImageMagick policies. Domains include system, delegate, coder, filter, path, or resource. Rights include none, read, write, and execute. Use | to combine them, for example: "read | write" to permit read from, or write to, a path. Use a glob expression as a pattern. Suppose we do not want users to process MPEG video images: <policy domain="delegate" rights="none" pattern="mpeg:decode" /> Here we do not want users reading images from HTTP: <policy domain="coder" rights="none" pattern="HTTP" /> Lets prevent users from executing any image filters: <policy domain="filter" rights="none" pattern="*" /> The /repository file system is restricted to read only. We use a glob expression to match all paths that start with /repository: <policy domain="path" rights="read" pattern="/repository/*" /> Let's prevent possible exploits by removing the right to use indirect reads. <policy domain="path" rights="none" pattern="@*" /> Any large image is cached to disk rather than memory: <policy domain="resource" name="area" value="1GB"/> Define arguments for the memory, map, area, width, height, and disk resources with SI prefixes (.eg 100MB). In addition, resource policies are maximums for each instance of ImageMagick (eg policy memory limit 1GB, -limit 2GB exceeds policy maximum so memory limit is 1GB). --> <policymap> <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> --> <policy domain="resource" name="memory" value="256MiB"/> <policy domain="resource" name="map" value="512MiB"/> <policy domain="resource" name="width" value="16KP"/> <policy domain="resource" name="height" value="16KP"/> <policy domain="resource" name="area" value="128MB"/> <policy domain="resource" name="disk" value="1GiB"/> <!-- <policy domain="resource" name="file" value="768"/> --> <!-- <policy domain="resource" name="thread" value="4"/> --> <!-- <policy domain="resource" name="throttle" value="0"/> --> <!-- <policy domain="resource" name="time" value="3600"/> --> <!-- <policy domain="system" name="precision" value="6"/> --> <!-- not needed due to the need to use explicitly by mvg: --> <!-- <policy domain="delegate" rights="none" pattern="MVG" /> --> <!-- use curl --> <policy domain="delegate" rights="none" pattern="URL" /> <policy domain="delegate" rights="none" pattern="HTTPS" /> <policy domain="delegate" rights="none" pattern="HTTP" /> <!-- in order to avoid to get image with password text --> <policy domain="path" rights="none" pattern="@*"/> <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/> <!-- disable ghostscript format types --> <policy domain="coder" rights="none" pattern="PS" /> <policy domain="coder" rights="none" pattern="EPI" /> <policy domain="coder" rights="read|write" pattern="PDF" /> <policy domain="coder" rights="none" pattern="XPS" /> <policy domain="coder" rights="read|write" pattern="LABEL" /> </policymap>
./configure --prefix=/var/task/imagemagick --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared=no --enable-static=yes --with-modules --with-perl=no --with-x=no --with-gslib=no --with-lcms --without-rsvg --with-xml --without-dps --disable-hdri --with-quantum-depth=8 --disable-openmp make sudo make install
.so
files..so
文件。 mkdir lib cd /usr/lib64/ cp -L libbz2.so.1 libexpat.so.1 libfontconfig.so.1 libfreetype.so.6 libgs.so.9 libjbig.so.2.0 libjpeg.so.62 liblcms2.so.2 liblzma.so.5 libpng15.so.15 libtiff.so.5 libxml2.so.2 libMagickCore-6.Q16.so.6 libMagickWand-6.Q16.so.6 libXext.so.6 libXt.so.6 libltdl.so.7 libSM.so.6 libICE.so.6 libX11.so.6 libgomp.so.1 libuuid.so.1 libxcb.so.1 libXau.so.6 libMagickCore-6.Q8.so.6 libMagickWand-6.Q8.so.6 libm.so.6 libz.so.1 libjasper.so.1 /home/ec2-user/lib/ cp -r ImageMagick-6.9.10/ ImageMagick-6.9.11/ /home/ec2-user/lib/ cd /home/ec2-user tar zcf lib.tar.gz lib/
Copy the lib.tar.gz
file from the server to your local machine.lib.tar.gz
文件从服务器复制到本地机器。 cd /var/task/imagemagick sudo tar zcf bin.tar.gz bin/ cp bin.tar.gz /home/ec2-user/bin.tar.gz
Copy the bin.tar.gz
file from the server to your local machine.bin.tar.gz
文件从服务器复制到本地机器。 cd /etc/ sudo tar zcf etc.tar.gz ImageMagick-6/ cp etc.tar.gz /home/ec2-user/etc.tar.gz
Copy the etc.tar.gz
file from the server to your local machine.etc.tar.gz
文件从服务器复制到本地机器。*.tar.gz
files.*.tar.gz
文件的内容。bin/
folder and rename it to gs
.bin/
文件夹并将其重命名为gs
。 Run chmod +x bin/gs
to make it executable.chmod +x bin/gs
使其可执行。lib
, bin
and etc
- folders into a ZIP file.lib
、 bin
和etc
- 文件夹压缩为 ZIP 文件。 The tree structure of the ZIP file would look like file.zip/ |-- bin | |-- convert | |-- ... | `-- gs |-- etc | `-- ImageMagick-6 | |-- coder.xml | |-- ... | `-- type.xml `-- lib |-- ImageMagick-6.9.10 | |-- config-Q16 | | `-- configure.xml | `-- modules-Q16 | |-- coders | | |-- aai.la | | |-- ... | | `-- yuv.so | `-- filters | |-- analyze.la | `-- analyze.so |-- ImageMagick-6.9.11 | |-- config-Q8 | | `-- configure.xml | `-- modules-Q8 | |-- coders | | |-- aai.la | | |-- ... | | `-- yuv.so | `-- filters | |-- analyze.la | `-- analyze.so |-- libICE.so.6 |-- ... `-- libz.so.1
I have used ...
wherever the folder contained more than 2 files to denote that there are more files present....
文件夹包含超过 2 个文件的任何地方表示存在更多文件。wand
can find ImageMagick.wand
可以找到 ImageMagick。 import os os.environ["PATH"] = f"/opt/bin:{os.environ['PATH']}" os.environ["LD_LIBRARY_PATH"] = f"/opt/lib:{os.environ['LD_LIBRARY_PATH']}" os.environ["MAGICK_HOME"] = "/opt/" os.environ["WAND_MAGICK_LIBRARY_SUFFIX"] = "-6.Q8" os.environ["MAGICK_CONFIGURE_PATH"] = "/opt/etc/ImageMagick-6/" os.environ["MAGICK_CODER_MODULE_PATH"] = "/opt/lib/ImageMagick-6.9.11/modules-Q8/coders/"
Note: If the size of the uncompressed ZIP file is too large and you reach AWS Lambda size limits, remove the binaries that you don't need from the bin/
folder.注意:如果未压缩的 ZIP 文件过大并且您达到 AWS Lambda 大小限制,请从
bin/
文件夹中删除您不需要的二进制文件。 In my case, I only kept Magick-config
, MagickCore-config
, MagickWand-config
, Wand-config
, convert
and gs
and removed others.就我而言,我只保留了
Magick-config
、 MagickCore-config
Magick-config
、 MagickCore-config
MagickWand-config
、 Wand-config
、 convert
和gs
并删除了其他人。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.