简体繁体中英

fp16 inference on cpu Pytorch

原文 2020-05-31 06:39:39 3 1 python/ pytorch/ eval/ cpu

I have a pretrained pytorch model I want to inference on fp16 instead of fp32 , I have already tried this while using the gpu but when I try it on cpu I get: "sum_cpu" not implemented for 'Half' torch . any fixes?

1 answers

As I know, a lot of CPU-based operations in Pytorch are not implemented to support FP16; instead, it's NVIDIA GPUs that have hardware support for FP16(eg tensor cores in Turing arch GPU) and PyTorch followed up since CUDA 7.0(ish). To accelerate inference on CPU by quantization to FP16, you may wanna try torch.bfloat16 dtype( https://github.com/pytorch/pytorch/issues/23509 ).

Trying to write OpenVINO inference engine but input image astype to FP16 get ValueError: could not convert string to float

Convert 16 bit hex value to FP16 in Python?

how to do convolution with fp16(Eigen::half) on tensorflow

OpenVINO failed to set Blob with FP16 precision not corresponding to user input precision

Run inference on CPU using pytorch and multiprocessing

What is the best way to use multiprocessing CPU inference for PyTorch models?

How forward pass and inference are different in pytorch?

Fluctuations in neural netwok inference time in PyTorch

Pytorch inference CUDA out of memory when multiprocessing

Seq2seq pytorch Inference slow

暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Trying to write OpenVINO inference engine but input image astype to FP16 get ValueError: could not convert string to float Convert 16 bit hex value to FP16 in Python? how to do convolution with fp16(Eigen::half) on tensorflow OpenVINO failed to set Blob with FP16 precision not corresponding to user input precision Run inference on CPU using pytorch and multiprocessing What is the best way to use multiprocessing CPU inference for PyTorch models? How forward pass and inference are different in pytorch? Fluctuations in neural netwok inference time in PyTorch Pytorch inference CUDA out of memory when multiprocessing Seq2seq pytorch Inference slow

Related Tags

fp16 inference on cpu Pytorch

Question

1 answers

solution1 0 2021-01-11 06:46:58

solution1
0 2021-01-11 06:46:58