### 不使用Pipeline直接调用ControlNet 在Diffusers库中，即使不通过预定义的`pipeline`也可以直接操作ControlNet模型来进行图像生成或处理。这涉及到更底层的操作，允许对过程有更大的控制。 首先加载必要的组件，包括但不限于VAE解码器、UNet模型以及具体的ControlNet实例。对于每一个ControlNet模块，初始化时需指定其配置文件路径和权重文件位置[^1]： ```python from diffusers import ControlNetModel, UNet2DConditionModel from transformers import CLIPTextModel import torch controlnet = ControlNetModel.from_pretrained("path_to_controlnet_config", "path_to_controlnet_weights") unet = UNet2DConditionModel.from_pretrained("path_to_unet_config", "path_to_unet_weights") text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14") ``` 准备输入数据，特别是提示词（prompt）及其对应的嵌入向量。这些信息会被传递给UNet用于指导图像合成方向。同时准备好条件图象作为ControlNet的额外输入： ```python from PIL import Image import numpy as np # 假设有一个PIL.Image对象image conditioning input condition_image = image.convert("RGB") processor = processor_class.from_pretrained('model_name') # 替换成实际使用的处理器名称 processed_condition = processor(condition_image) prompt_ids = tokenizer(prompt, padding="max_length", truncation=True, max_length=77, return_tensors="pt").input_ids encoder_hidden_states = text_encoder(prompt_ids)[0] ``` 构建推理循环，在此期间手动管理噪声预测与逐步去除的过程。这里的关键在于将ControlNet产生的特征映射融入到每一步的UNet计算之中： ```python scheduler = scheduler_class.from_pretrained('model_name') latents = torch.randn((batch_size, unet.in_channels, height // 8, width // 8)) for t in reversed(range(scheduler.num_train_timesteps)): latent_model_input = scheduler.scale_model_input(latents, timestep=t) down_block_res_samples, mid_block_res_sample = controlnet( processed_condition, t, encoder_hidden_states=encoder_hidden_states, return_dict=False, ) noise_pred = unet( latent_model_input, t, encoder_hidden_states=encoder_hidden_states, down_block_additional_residuals=down_block_res_samples, mid_block_additional_residual=mid_block_res_sample, ).sample latents = scheduler.step(noise_pred, t, latents).prev_sample ``` 最后利用VAE解码器将潜在空间表示转换回像素域内的图片形式完成整个流程： ```python vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse") # 或者其他合适的VAE模型 decoded_images = vae.decode(latents / vae.config.scaling_factor, return_dict=False)[0] final_images = (decoded_images / 2 + 0.5).clamp(0, 1) final_images = final_images.detach().cpu().permute(0, 2, 3, 1).numpy() pil_images = [Image.fromarray(np.uint8(img * 255)) for img in final_images] ```