drbaph

/

AuK-comfyui

Like 14

Greetings! First, thank you for releasing AuK and contributing the model and MIT-licensed weights to the community.

Our ComfyUI integration already includes an optional Prompt Enhance node. It uses the same Qwen2.5-Omni-3B model locally to classify the task, produce the canonical AuK instruction, estimate the target duration, and process the source audio where required.

We compared the FP32 and BF16 ComfyUI implementations against your Hugging Face demo using the same source audio, instructions, and prompt enhancement. Some tasks were still inconsistent, particularly those listed as unstable in our README.

During testing, we found one nuanced workaround that improved success rates for editing tasks: setting the requested output duration approximately 0.2–0.5 seconds shorter than the source audio. For example, with a 6-second input, manually requesting around 5.5 seconds often produced a more successful edit. This sometimes helped even when the replacement content would reasonably require a longer output.

This suggests that output-duration conditioning may affect editing adherence beyond simply providing enough time for the requested content.

We would appreciate any additional inference settings, prompt templates, preprocessing steps, duration rules, or task-specific parameters used by the Hugging Face demo that may differ from the published local inference code. We would be happy to update the integration and repeat our benchmarks.

Edit We also noticed some of the unstable tasks have more success when the ZH instruction is used instead of the EN from the cookbook

Thank you for the clarification. I think there may be a misunderstanding regarding the Prompt Enhancer implementation I was referring to.

Specifically, I was referring to the official AuK Prompt Enhancer. It is not simply a local Qwen2.5-Omni-3B-based classifier. Qwen2.5-Omni-3B serves as the multimodal encoder used by AuK itself, rather than the LLM used by the official Prompt Enhancer.

In the official implementation, after configuring an OpenAI-compatible LLM endpoint through LLM_API_KEY, LLM_BASE_URL, and LLM_MODEL_NAME—for example, a compatible DeepSeek or Tencent TokenHub endpoint—the Prompt Enhancer processes the user's free-form instruction together with the optional source audio. It produces:

• a canonical task-specific instruction for AuK;
• an estimated target duration; and
• when required, task-specific preprocessed source audio.

More specifically, the LLM is used for task/subtype classification, structured parameter extraction, and instruction rewriting for applicable tasks. The Prompt Enhancer then constructs the canonical instruction using predefined task templates and applies task-specific duration rules.

For tasks involving source audio, ASR is also used to understand and locate relevant content in the audio. Tencent Cloud ASR is used when configured, while the local CPU-based SenseVoiceSmall model serves as a fallback. The pipeline may additionally perform preprocessing such as VAD-based trimming and task-specific audio-level normalization.

Therefore, if the ComfyUI Prompt Enhance node uses Qwen2.5-Omni-3B locally to classify the task, generate the instruction, and estimate the target duration, that implementation would not be equivalent to the official AuK Prompt Enhancer pipeline.

Could you confirm whether your comparison used the official Prompt Enhancer implementation described above, including its OpenAI-compatible LLM and ASR pipeline, or the local Qwen2.5-Omni-based implementation?

That said, since you were also able to reproduce the instability in the hosted Hugging Face demo, I agree that differences in the Prompt Enhancer implementation may not fully explain the observed behavior. Your observations regarding duration conditioning and the difference between Chinese and English instructions are particularly useful and worth investigating further.

Thank you for the detailed clarification.

To clarify our testing setup, our local comparison used:

Qwen2.5-Omni-3B as the local prompt-enhancement helper
Whisper ASR to transcribe the complete source audio
The Whisper transcript as context for task classification, parameter extraction, and duration calculation
The task-specific instruction templates from the official AuK cookbook
The same source audio and equivalent task requests with FP32 and BF16
A text-output node connected to Prompt Enhance so we could inspect each generated internal instruction before inference
During our tests, the local helper consistently selected the intended task and produced the expected cookbook-style instruction. We manually inspected the task classification, extracted parameters, transcript context, and final instruction before every generation.

We then compared the outputs with the hosted Hugging Face demo using its Prompt Enhancer. We observed similar behavior in both environments on the tasks listed as unstable in our README.

Lyric editing was one particularly noticeable example. Replacements appeared more reliable when the original and replacement words were phonetically similar. For example, replacing “dirt” with “Bert” succeeded more often than replacing “dirt” with “pillow.” With larger phonetic differences, the original lyric was mostly retained.

We understand that our local Qwen-based helper does not reproduce every internal component of the official Prompt Enhancer. We only wanted to clarify that Whisper ASR context was provided, the generated instructions were inspected and correct, and similar output behavior was also observed in the hosted demo.

We appreciate the additional technical context and would be interested in any other preprocessing or inference details that might help us investigate this further.