How to apply LORAs like in SD WebUI to DreamShaper using python

Question

I have been using stable diffusion WebUI to try out different models and LORAs for my application. I'm trying to now do the equivalent of what I've been doing in WebUI but in Python. I have a safetensors file for DreamShaper v5 beta-2 with VAE baked in. I am using two other safetensors files for two LORAs I downloaded from civitai.com.

Here is the code that I tried:

import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

pipeline = StableDiffusionPipeline.from_ckpt("./DreamShaper_5_beta2_BakedVae.ckpt", torch_dtype=torch.float16)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
pipeline.unet.load_attn_procs("./flat illustration.safetensors", local_files_only=True)
pipeline.unet.load_attn_procs("./improve_backgrounds.safetensors", local_files_only=True)
pipeline.to("cuda")
pipeline.enable_xformers_memory_efficient_attention()

prompt = "Flat vector illustration of a scary and ominous grassy landscape with five or more trees, a large crack in the ground, and a gigantic monster sticking up high above the crack. The monster is based on an oak tree and made up of all kinds of litter and debris, including cans and bottles. The landscape is scattered with lots of litter and debris, especially tipped over garbage cans. There are hundreds of people running away from the monster, and the environment is dusty with no texture or shading. The color scheme of the grassy landscape is green and brown. <lora:flat illustration:1> <lora:improve_backgrounds:0.85>"
nprompt = "(deformed iris, deformed pupils, anime:1.4), text, close up, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, extremely focused on people"
image = pipeline(prompt, negative_prompt=nprompt, num_inference_steps=40, guidance_scale=7.5, cross_attention_kwargs={"scale": 1}).images[0]
image.save("blue_pokemon.png")

I tried using the ckpt file for dreamshaper instead of the safetensors file to evade the following error:

Traceback (most recent call last):
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\loratest\main.py", line 4, in <module>
    pipeline = StableDiffusionPipeline.from_ckpt("./DreamShaper_5_beta2_BakedVae.safetensors", torch_dtype=torch.float16)
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\venv\lib\site-packages\diffusers\loaders.py", line 1284, in from_ckpt
    pipe = download_from_original_stable_diffusion_ckpt(
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 1062, in download_from_original_stable_diffusion_ckpt
    raise ValueError(BACKENDS_MAPPING["safetensors"][1])
KeyError: 'safetensors'

But then I got this error:

global_step key not found in model
In this conversion only the non-EMA weights are extracted. If you want to instead extract the EMA weights (usually better for inference), please make sure to add the `--extract_ema` flag.
Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.20.mlp.fc1.weight', 'vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_model.encoder.layers.16.mlp.fc1.weight', 'vision_model.encoder.layers.10.layer_norm1.weight', 
...
'vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.layer_norm1.weight', 'vision_model.encoder.layers.14.mlp.fc1.bias', 'vision_model.encoder.layers.19.mlp.fc1.weight', 'vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.12.layer_norm1.bias', 'vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_model.encoder.layers.8.self_attn.out_proj.bias']
- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\loratest\main.py", line 6, in <module>
    pipeline.unet.load_attn_procs("./flat illustration.safetensors", local_files_only=True)
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\venv\lib\site-packages\diffusers\loaders.py", line 217, in load_attn_procs
    state_dict = torch.load(model_file, map_location="cpu")
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\venv\lib\site-packages\torch\serialization.py", line 815, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\***\PycharmProjects\KnowledgeGraph\venv\lib\site-packages\torch\serialization.py", line 1033, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
MemoryError

My source for this code was from the huggingface docs. I tried looking in the SD WebUI Github WIKI but didn't find anything.

Again, all I'm trying to do is apply the LORAs to DreamShaper using Python just as I would have in SD WebUI. If it helps, here is where I got the LORAs from:

https://civitai.com/models/19130/flat-illustration

https://civitai.com/models/42190/improve-backgrounds

Answer 1

It's my solution:

pip install safetensors