多模态模型

Stable Diffusion 3.5来了！更真实，更可控，更开放！魔搭社区推理最佳实践

2024 年 10 月 28 日

SD 3.5来了！

Stability近期发布了最新模型Stable Diffusion 3.5系列（后简称 SD3.5），进行了全面的架构和培训更改，现在根据更新的、更宽松的社区license，增强了图像保真度、指令遵循和可控性。

ModelScope生成的测试图像

SD3.5 模型在以下领域表现出色

可定制性：轻松微调模型以满足特定创作需求，或根据定制的工作流构建应用程序，多个开发者测试下来，SD3.5不会迎合去生成非常“美”的图片，但是会生成很有特点和风格的图片。

高效性能：SD3.5可在消费级显卡运行，Stable Diffusion 3.5 Large fp8版本的comfyui在魔搭免费算力运行仅需要18G显存。

多样化输出：同样的提示词创建代表多样性的图像，而不仅仅是一种类型的人，具有不同的肤色和特征，无需大量提示，不会美的千篇一律。

风格多样：能够生成各种风格和美感，如 3D、摄影、绘画、线条艺术以及几乎任何可以想象的视觉风格。

同时，license支持输出所有权：保留所产生的图像的所有权，不受限制性许可的影响。

模型链接：

https://modelscope.cn/collections/Stable-Diffusion-35-8cd8a1c210b84a

体验链接（turbo快速生图）：

https://modelscope.cn/studios/AI-ModelScope/stable-diffusion-3.5-large-turbo

小程序体验：

体验链接（SD3.5 Large）：

https://modelscope.cn/studios/AI-ModelScope/stable-diffusion-3.5-large

小程序体验：

模型效果评估-diffusers推理

简单 Prompt

an apple	a car	a dog

复杂 Prompt

a bear standing on a car, sunset, winter

a boy and a girl, the boy stands at the left side, the boy wears a red t-shirt and blue pants, the girl wears a green t-shirt and pink pants.

the apple is in the box, the box is on the chair, the chair is on the desk, the desk is in the room

多实体生成能力很能打，颜色能做到分别控制，空间关系还做不到完美

多风格

Chinese ink painting, a girl, long hair, colorful hair, shining eyes	oil painting, a girl, long hair, colorful hair, shining eyes	anime, a girl, long hair, colorful hair, shining eyes

很喜欢这个油画的笔触，很真实

文本

a car, the number is NT352K	a book, the title is "how to marry with a rich woman"	text on the sky, "modelscope"

能处理简单的英文文本

多样性

sunset over the sea	sunset over the sea	sunset over the sea

多样性很好

模型推理

安装依赖

!pip install diffusers -U

推理代码

import torchfrom diffusers import StableDiffusion3Pipelinefrom modelscope import snapshot_download
model_dir = snapshot_download("AI-ModelScope/stable-diffusion-3.5-large")
pipe = StableDiffusion3Pipeline.from_pretrained(model_dir, torch_dtype=torch.bfloat16)pipe = pipe.to("cuda")
image = pipe("A capybara holding a sign that reads Hello World",num_inference_steps=28,guidance_scale=3.5,).images[0]image.save("capybara.png")

显存占用：

搭建 ComfyUI

从最新的ComfyUI的代码安装

# #@title Environment Setup
from pathlib import Path
OPTIONS = {}UPDATE_COMFY_UI = True#@param {type:"boolean"}INSTALL_COMFYUI_MANAGER = True#@param {type:"boolean"}INSTALL_ANIMATEDIFF = True#@param {type:"boolean"}INSTALL_CUSTOM_NODES_DEPENDENCIES = True#@param {type:"boolean"}OPTIONS['UPDATE_COMFY_UI'] = UPDATE_COMFY_UIOPTIONS['INSTALL_COMFYUI_MANAGER'] = INSTALL_COMFYUI_MANAGERINSTALL_CUSTOM_NODES_DEPENDENCIES
current_dir = !pwdWORKSPACE = f"{current_dir[0]}/ComfyUI"


%cd /mnt/workspace/
![ ! -d $WORKSPACE ] && echo -= Initial setup ComfyUI =- && git clone https://github.com/comfyanonymous/ComfyUI%cd $WORKSPACE
if OPTIONS['UPDATE_COMFY_UI']:!echo "-= Updating ComfyUI =-"!git pull

下载模型到对应的文件夹

#@markdown ###Download standard resources%cd /mnt/workspace/ComfyUI### SD3.5!modelscope download --model=AI-ModelScope/stable-diffusion-3.5-fp8 --local_dir ./models/checkpoints/ sd3.5_large_fp8_scaled.safetensors
### clip!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ clip_l.safetensors!modelscope download --model=AI-ModelScope/stable-diffusion-3.5-fp8 --local_dir ./models/clip/ text_encoders/clip_g.safetensors!modelscope download --model=AI-ModelScope/flux_text_encoders --local_dir ./models/clip/ t5xxl_fp8_e4m3fn.safetensors

通过cloudflared启动 ComfyUI ，

!wget "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/cloudflared-linux-amd64.deb"!dpkg -i cloudflared-linux-amd64.deb
%cd /mnt/workspace/ComfyUIimport subprocessimport threadingimport timeimport socketimport urllib.request
def iframe_thread(port):while True:time.sleep(0.5)sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)result = sock.connect_ex(('127.0.0.1', port))if result == 0:breaksock.close()print("\nComfyUI finished loading, trying to launch cloudflared (if it gets stuck here cloudflared is having issues)\n")
p = subprocess.Popen(["cloudflared", "tunnel", "--url", "http://127.0.0.1:{}".format(port)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)for line in p.stderr:l = line.decode()if "trycloudflare.com " in l:print("This is the URL to access ComfyUI:", l[l.find("http"):], end='')#print(l, end='')

threading.Thread(target=iframe_thread, daemon=True, args=(8188,)).start()
!python main.py --dont-print-server