[英]How to apply pre-processing to images of a tf.data.Dataset?
If I understand correctly instead of loading aa full dataset into memory like this:如果我理解正确,而不是像这样将完整的数据集加载到 memory 中:
images = []
file_list = glob.glob('path/to/images/*.jpg')
for file in file_list:
images.append(img_to_array(load_img(file, target_size=input_shape)))
images = np.stack(images, axis=0)
images = preprocess(images)
# classify the image
print("[INFO] classifying image with '{}'...".format(used_model))
predictions = model.predict(images)
decoded_predictions = imagenet_utils.decode_predictions(predictions)
One should use to tensorflows data utilities for better memory management and performance:应该使用 tensorflows 数据实用程序以获得更好的 memory 管理和性能:
images = tf.keras.utils.image_dataset_from_directory(file_path, image_size=input_shape, labels=None)
AUTOTUNE = tf.data.AUTOTUNE
images = images.prefetch(buffer_size=AUTOTUNE)
# this line will now crash
images = preprocess(images)
# classify the image
print("[INFO] classifying image with '{}'...".format(used_model))
predictions = model.predict(images)
decoded_predictions = imagenet_utils.decode_predictions(predictions)
As written in the code above, I know have different data structures, which will not work with the same code.如上面的代码所写,我知道有不同的数据结构,它们不能用于相同的代码。 My question is: How can I apply pre-processing to my data?
我的问题是:如何对我的数据进行预处理? All the corresponding tutorials seem to deal with training, while I want to do simple inference.
对应的教程好像都在讲训练,而我想做简单的推理。
Additional Question : How would this be done, if the data is coming from an S3 bucket (with the script running in an Airflow-DAG)?附加问题:如果数据来自 S3 存储桶(脚本在 Airflow-DAG 中运行),将如何完成?
You can use tf.data.Dataset.map
to apply preprocessing to your images or batches of images.您可以使用
tf.data.Dataset.map
对图像或图像批次应用预处理。 Here is an example:这是一个例子:
import tensorflow as tf
import pathlib
dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True)
data_dir = pathlib.Path(data_dir)
batch_size = 32
train_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
seed=123,
image_size=(180, 180),
batch_size=batch_size)
scale_layer = tf.keras.layers.Rescaling(1./255)
def preprocess(images, labels):
images = tf.image.resize(scale_layer(images),[120, 120], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
return images, labels
train_ds = train_ds.map(preprocess)
In your case, you just have images so you can ignore the labels here.在你的例子中,你只有图像,所以你可以忽略这里的标签。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.