Code

Reuse KV cache of prefixes#5572

Code

Closed

tohtana wants to merge 18 commits intodeepspeedai:masterdeepspeedai/DeepSpeed:masterfrom tohtana:tohtana/cache_prefixtohtana/DeepSpeed:tohtana/cache_prefixCopy head branch name to clipboard

+282-39

Commits

Commits on Apr 23, 2024

• reuse prefix of kv cache

  tohtana

  committed
  696e3b8

  Copy full SHA for 696e3b8

Commits on Apr 26, 2024

• free block with no ref

  tohtana

  committed
  2e8ac1c

  Copy full SHA for 2e8ac1c

Commits on Apr 28, 2024

• reversed block id list when freeing

  tohtana

  committed
  619a363

  Copy full SHA for 619a363

Commits on Apr 29, 2024

• add option to enable prefix cache

  tohtana

  committed
  7307507

  Copy full SHA for 7307507

Commits on May 8, 2024

• fix use of prefix cache

  tohtana

  committed
  a28b706

  Copy full SHA for a28b706

Commits on May 25, 2024

• fix allocation bug

  tohtana

  committed
  d8e9d28

  Copy full SHA for d8e9d28

Commits on May 27, 2024

• use normal allocator if prefix sharing is disabled

  tohtana

  committed
  9cffae1

  Copy full SHA for 9cffae1

• match type hint

  tohtana

  committed
  f10f6f4

  Copy full SHA for f10f6f4

• refactor allocator

  tohtana

  committed
  0bedc39

  Copy full SHA for 0bedc39

• simplify loop

  tohtana

  committed
  bc92d2b

  Copy full SHA for bc92d2b

• refactor

  tohtana

  committed
  c2028ec

  Copy full SHA for c2028ec

• remove unnecessary reverse

  tohtana

  committed
  959204c

  Copy full SHA for 959204c

• fix attribute name

  tohtana

  committed
  9289ff7

  Copy full SHA for 9289ff7

• Merge branch 'master' into tohtana/cache_prefix

  tohtana

  authored
  30b9d0b

  Copy full SHA for 30b9d0b

Commits on May 30, 2024

• update prefix cache at every iteration

  tohtana

  committed
  0c8e0e6

  Copy full SHA for 0c8e0e6

Commits on May 31, 2024

• skip looking up cache

  tohtana

  committed
  ea50fb5

  Copy full SHA for ea50fb5

Commits on Jun 3, 2024

• fix prefix tokens to cache

  tohtana

  committed
  1493ab7

  Copy full SHA for 1493ab7

• add assertion

  tohtana

  committed
  07a4c44

  Copy full SHA for 07a4c44