Skip to content

Prof algo - #1

Open
SpiritedAwayCN wants to merge 10 commits into
xinhaoc:llama_2from
SpiritedAwayCN:prof_algo
Open

SpiritedAwayCN wants to merge 10 commits into
xinhaoc:llama_2from
SpiritedAwayCN:prof_algo

Conversation

@SpiritedAwayCN

Copy link
Copy Markdown

Description of changes:

  • Add profiling step to select best algorithm for cublas GEMM kernels

Note:

  • There's no command flag to disable the profiling yet. please search NO_PROF in linear_kernel.cu and inc_multihead_self_attention.cu, you can find a commented return at the beginning of the line. Removing all these comments to enable all the returns can disable this.
  • The main modifications were findBestAlgoID() funcions in both linear_kernel.cu and inc_multihead_self_attention.cu, where the profile code were implemented.
  • QKV kernels need to be grouped, and profile codes in inc_multihead_self_attention.cu requires slight changes. Please search To Xinhao in inc_multihead_self_attention.cu, the comment can help you to easily modify the profile code.
  • Some utilization functions to be moved to a better place and maybe some of the code violates the format/style rules.

Before merging:

  • Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

SpiritedAwayCN and others added 10 commits August 2, 2023 08:32
* n_kv_heads in inc_mha

* .

* .

* .

* .

* .

* fix

* fix

* tensor parallelism

* change weight layout

* tensor parallelism

* merge multiquery attention into inc_mha

* llama2 70B config

* spec infer change 1

* fix.

* spec infer.

* falcon spec infer.

* fix llama 70B

* fix

* fix & cleanup

* fix

* hip rocm

* issue 908

* clean debug code.

* format.

* remove multiquery. remove warning, fix python.

---------

Co-authored-by: goliaro <goliaro@cs.cmu.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants