Test vision encoder only frames 32 out of memory on 16GB GPU

[lucasjinreal]

torch.Size([1, 8192, 1024])
torch.Size([1, 1024])

this is 8frame output last hiddent state, the tokens is extremly large.

[lucasjinreal]

for qwen3vl even a relatively small model, i can send 32 frames without problem.

[Luodian]

you’re likely not hitting the codec path yet.
[1, 8192, 1024] usually means dense frame tokens, not codec-packed input.

our llava_next codec flow is:

# 1) build env
cd OneVision-Encoder/llava_next
docker build -t ov_encoder_llava:26.01 .

# 2) stage1: extract codec saliency (MV/residual)
python Compressed_Video_Reader/tool/stage1.py \
  --dataset_path /path/to/raw_videos.json \
  --out_root /path/to/stage1_output \
  --sequence_length 64 \
  --keep_frames_equiv 8 \
  --square_size 576 \
  --patch_size 16 \
  --num_workers 8 \
  --keep_first_full_frame \
  --padding_policy zero

# 3) stage2: pack to model-ready assets
python Compressed_Video_Reader/tool/stage2.py \
  --mode pack \
  --input_dataset /path/to/raw_videos.json \
  --out_jsonl /path/to/training_codec.jsonl \
  --visidx_root /path/to/stage1_output \
  --out_image_root /path/to/stage2_images \
  --num_images 8 \
  --square_size 576 \
  --T 64 \
  --patch 16 \
  --write_positions \
  --num_workers 8 \
  --first_full

for eval, we load precomputed offline codec assets:

export LLAVA_CODEC_USE_OFFLINE=1
export LLAVA_CODEC_OFFLINE_ROOT=/path/to/<task>_offline/assets
export LLAVA_CODEC_VISIDX_MODE=pack_topk
export LLAVA_CODEC_SEQ_LEN_FRAMES=64
export LLAVA_CODEC_NUM_IMAGES=8
export LLAVA_CODEC_SQUARE_SIZE=576
export LLAVA_CODEC_PATCH_SIZE=16

please paste your exact preprocessing + forward snippet.
if it’s codec mode, we should see packed assets (mosaic_*.jpg + positions_thw.npy) in the pipeline, not dense frame-token behavior.