Intel SIMD and AVX-512

Single Instruction, Multiple Data - computers with multiple processing elements perform the same operation on multiple data points simultaneously, which exploit data level parallelism, but not concurrency - simultaneous computations, only a single process

Intel and AMD comparation

AVX-512

Content

Intel® AVX-512 Foundation (F)

• 512-bit vector width.
• 32 512-bit long vector registers.
• Data expand and data compress instructions.
• Ternary logic instruction.
• 8 new 64-bit long mask registers.
• Two source cross-lane permute instructions.
• Scatter instructions. — Embedded broadcast/rounding.
• Transcendental support.

Intel® AVX-512 Conflict Detection Instructions (CD)

Intel® AVX-512 Exponential and Reciprocal Instructions (ER)

Intel® AVX-512 Prefetch Instructions (PF)

Intel® AVX-512 Byte and Word Instructions (BW)

Intel® AVX-512 Double Word and Quad Word Instructions (DQ) — New QWORD and Compute and Convert Instructions.

Intel® AVX-512 Vector Length Extensions (VL)

When any 512-bit instruction is used, there is a moderate reduction in speed, and if a core uses the heaviest of these instructions in a sustained way, the core may run much slower. Furthermore, the slowdown is usually worse when more cores use these new instructions. In the worst case, you might be running at half the advertised frequency and thus your whole application could run slower.

Downclocking

• Only happens when using 512-bits register - means we can benefit from AVX-512 instructions and features without downclocking.

• Never get downclocking when working on 128-bit registers

• Downclocking, when it happens, is per core and for a short time after you have used particular instructions

• Heavy instructions
  • floating point operations, integer multiplication, leading-zero bits AVX-512 instructions …
  • used in deep learning, numerical analysis, high performance computing, cryptograph
• Light instructions
  • integer operations, logical operations, data shuffling
  • text processing, fast compression routines, vectorized implementation of library routines
• Intel cores run in three model from license 0 to 2 - fastest to slowest
  • L2 will happen when sustained use of heavy 512-bit instructions - one such instruction every cycle
  • L1 will happen when sustained use of heavy 256-bit instructions
  • using taskset and numactl to control which cores run the heavy instructions

  Intel Xeon Gold 5120 processor…

  mode	1 active core	9 active cores
  Normal	3.2 GHz	2.7 GHz
  AVX2	3.1 GHz	2.3 GHz
  AVX-512	2.9 GHz	1.6 GHz

• Notes

  • Downclocking will happen in any case even if you are not using AVX-512 instructions
  • Downclocking is modest when using light AVX-512 even if it across all cores
  • Downclocking becomes significant when using heavy numerical work

Thoughts

1. We engineers should monitor the frequency of cores and identify massive downclocking using perf.
2. Massive downclocking is maybe not an important risk on machines with few cores.
3. Consider isolating heavy numerical instructions work to specific threads/processes/cores, which limit the downclocking to cores and take full advantages of AVX-512. We should ensure the benefits of AVX-512 are substantial compared with non-AVX-512.
4. Check that light AVX-512 gives you a greater gain compared with frequency downclocking. Even the cost is lower.
5. Library, especially performance sensitive libraries, should determine whether AVX-512 is worth it and document the approach with the likely speedups from the wider instructions.
6. From the compiler’s perspective, it may decide to use the AVX-512 instructions while copying structure, inline memcpy and vectorizing loops, the decision is difficult.
7. We may need a tool that monitor the workload and identify when and why downclocking occurs. Meanwhile, OS or application frameworks could assign threads to specific cores.