Clarification on Code Structure and Usage

[DKmiyan]

Thanks for your excellent work!

I am a bit confused about the code structure and its usage:

GEARLM/GEARLM
    -- Simulated
    -- TrueCompression
        -- models
        -- old_models

What’s the difference between Simulated and TrueCompression?

Following the instructions in the README, I first ran:

python evaluation_gsm8k.py --model meta-llama/Meta-Llama-3-8B --prompt_file gsm8k_prompt_original.txt --batch_size 6 --max_new_tokens 256 --compress_method GEAR --attention_number 40 --quantize_bit 2 --group_size 64 --loop 3 --prefillrank 4 --prefillrankv 4 --rank 2 --rankv 2 --left 0.02 --streaming --streaming_gap 64

This calls GenerationBench/GenerationTest/evaluation_gsm8k.py, where line 380 compress_config calls GenerationBench/GenerationTest/GEARLM/Simulated/compress_config.py, and line 411 model = SimulatedGearLlamaForCausalLM.from_pretrained calls GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_new.py.

However, I haven't seen any modifications related to the GEAR method in modeling_llama_new.py, compress_config.py, or within GenerationTest/GEARLM/Simulated/. Could you clarify how the GEAR method is being invoked?

[HaoKang-Timmy]

Simulated is fake compression that has the same accuracy compared with real compression. True compression is real compression code without cuda kernel supported.
GEAR is invoked in modeling_llama_new.py.
Here

[DKmiyan]

Thank you for your prompt and detailed response.

Just to confirm my understanding: Simulated compression only simulates the compression process and achieves the same accuracy as true compression, but it does not simulate the changes in GPU memory usage. On the other hand, True Compression can simulate memory changes on the CPU, but it does not take advantage of CUDA acceleration.

I have one final question: To replicate the accuracy results from your paper, is it sufficient to use Simulated compression?