Feature/ideogram4 reference latent support

[silveroxides]

This PR implements reference latent support for Ideogram 4.

Ideogram_4_reflatent.json

[levzzz5154]

LGTM

[coderabbitai]

📝 Walkthrough

Walkthrough

The Ideogram4 transformer model gains optional reference latent support. _image_position_ids receives three new parameters (index, h_offset, w_offset) to produce spatially distinct coordinate grids per reference image. Both _run_conditional and _run_image_only can now append packed reference-image token sequences to the main token sequence, concatenate matching position IDs, and annotate transformer_options with per-reference token counts. _run_image_only additionally builds an all-image indicator tensor for the extended sequence and slices the backbone output back to the main-image length. _forward extracts ref_latents and ref_latents_method from kwargs and routes them into both methods. In model_base.py, Ideogram4 registers ref_latents in memory_usage_factor_conds, adds CONDList/CONDConstant entries in extra_conds(), and computes shape metadata in extra_conds_shapes().

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main feature addition: reference latent support for Ideogram 4.
Description check	✅ Passed	The description is related to the changeset, explaining reference latent support implementation with example outputs and configuration details.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

comfy/ldm/ideogram4/model.py (1)

247-286: 🏗️ Heavy lift

Extract the reference packing into a shared helper.

The conditional and image-only paths duplicate the same ref_latents_method parsing, tokenization, position-ID generation, and reference_image_num_tokens setup. Centralizing this keeps future reference-position fixes from landing in only one path.

Also applies to: 337-376

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@comfy/ldm/ideogram4/model.py` around lines 247 - 286, Extract the duplicate
reference latents processing logic (which includes ref_latents_method parsing,
offset calculation via index_ref_method and negative_ref_method conditionals,
tokenization via _img_to_tokens, position ID generation via _image_position_ids,
and list appending) into a shared helper method. This helper should accept the
reference latent, method type, current indices/offsets, device, and other
necessary parameters, then return the processed tokens, position IDs, and
updated state. Replace both the loop at lines 247-286 and the duplicate logic at
lines 337-376 with calls to this helper method to eliminate code duplication and
ensure fixes apply consistently across both paths.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@comfy/ldm/ideogram4/model.py`:
- Around line 247-286: Extract the duplicate reference latents processing logic
(which includes ref_latents_method parsing, offset calculation via
index_ref_method and negative_ref_method conditionals, tokenization via
_img_to_tokens, position ID generation via _image_position_ids, and list
appending) into a shared helper method. This helper should accept the reference
latent, method type, current indices/offsets, device, and other necessary
parameters, then return the processed tokens, position IDs, and updated state.
Replace both the loop at lines 247-286 and the duplicate logic at lines 337-376
with calls to this helper method to eliminate code duplication and ensure fixes
apply consistently across both paths.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5614747c-1fec-46e9-b241-7500c29b7ad1

📥 Commits

Reviewing files that changed from the base of the PR and between dc3f8f3 and 6e434c7.

📒 Files selected for processing (2)

• comfy/ldm/ideogram4/model.py
• comfy/model_base.py

[silveroxides]

@comfyanonymous I need verification about the CodeRabbit nitpick if that should be addressed or ignored.

[alexisrolland]

Hi @silveroxides and thanks for submitting this. I wanted to test it but the workflow contains a bunch of custom nodes and some relatively complex logic. Would it be please possible to have a minimal version?

[SvKora]

Hi @silveroxides and thanks for submitting this. I wanted to test it but the workflow contains a bunch of custom nodes and some relatively complex logic. Would it be please possible to have a minimal version?

i thing you need ComfyUI_SamplingUtils, ComfyUI-CES [WIP]. The second one is in the manager "dev"; the first is on the author’s page

[silveroxides]

I don't know what to do about the CES Conditioning formula. It is basically my method of not having to use the useless unconditional model by scheduling the negative conds with inverse positive conds for first step. Most other stuff in the workflow is nothing more than resolution selection or things that can be straight up deleted. ResolutionSelectionExtended is just an extended version of the core one. I have trouble reading the entire nodes 2.0 lauyout too since it obviously is not distinguishing missing nodes from missing models or missing images.

[silveroxides]

Thanks @SvKora . ComfyUI-UtilsCollection is also an option as it acts as drop in for SamplingUtils.

https://github.com/Clybius/ComfyUI-CES
https://github.com/silveroxides/ComfyUI-UtilsCollection

[silveroxides]

To put it blunt. The nodes present make up for what ComfyUI Core has been sorely lacking for a long time and it is very apparent in templates such as the Ideogram default one where math expressions and extract string from json complexity makes it look like a complete mess. But I guess I will have to make a simplified workflow because it feels like no one is able to take one apart and just figure stuff out anymore.

[silveroxides]

I could not test it though cause I do not have the text encoder on my PC anymore. I only use my own int8 rowwise quants which still has not gotten support in ComfyUI Core. The one I used to do the workflow was temporary and now deleted.

Ideogram_4_reflatent (1) (1).json

[alexisrolland]

@silveroxides

I could not test it though cause I do not have the text encoder on my PC anymore. I only use my own int8 rowwise quants which still has not gotten support in ComfyUI Core. The one I used to do the workflow was temporary and now deleted.

Ideogram_4_reflatent (1) (1).json

Thanks for providing the workflow. We receive many PRs so you will understand this helps. The passive aggressive tone does not. As far as int8 support is concerned, we are working on it.