The benefits of mixed-precision-training:
1、less memory, enabling the training and deployment of larger neural networks
2、require less memory bandwidth, speeds up data transfer operations
3、math operations run much faster
混合精度相关技术
为了想让深度学习训练可以使用FP16的好处，又要避免精度溢出和舍入误差。于是可以通过FP16和FP32的混合精度训练（Mixed-Precision），混合精度训练过程中可以引入权重备份（Weight Backup）、损失放大（Loss Scaling）、精度累加（Precision Accumulated）三种相关的技术。
3.1、权重备份（Weight Backup）
https://medium.com/@fanzongshaoxing/post-training-quantization-of-tensorflow-model-to-fp16-8d66b9dfa77f