We made animations with AI (Stable Diffusion)

Hi gamers and gamedevs,

as you can see from the project page, we took part in a game jam: the All By AI game jam. We were very excited to see how we could use this to make better games. We did ask it for some game design ideas, but as I said in the jam community, I think AI is here to make tedious tasks easy to leave you more time to the fun stuff. Game design falls in that category for us so that is all us.

However, we did use it for all the art assets in the game. Among is them my favorite achievement during the jam and the post of this devlog, the sprite animations.

Before I get into the process here are the pros and cons.
Pros: You get a ready to go character spritesheet, where you only need to delete the background. It's amazing for making a very large number of different textured animations.
Cons: The result was determined by the original input. So whenever I wanted another look, I had to edit the spritesheet. At that point, it falls back on your art skills again and it doesn't save much time.

The process:

That was simply by using an input sprite sheet.  Like this one:

This is an example which I used in the beginning to figure out if it works. As you can see I only gave 2 sprites to each direction.

Then I put that image into three controlnets: canny, normal, and depth. There's no fixed settings I use, I played around a little but this makes it pretty constrained so you get similar results all the time. 

With this prompt: "medieval knight isometric game asset for tdg, borderlands, ((charturnerv2)), 3d render, trending on polygon, award winning digital art, no shadows," you get this:

If you look in the prompt, there's a segment ((charturnerv2)), which is a textual inversion prompt that gives the consistency to the character.
Get it here and check out the poster's examples, they're pretty cool: https://civitai.com/models/3036/charturner-character-turnaround-helper-for-15-an...

After that it's just a matter of deleting the background and inserting it in Godot. For this one it would be pretty hard to delete the background because it has similar colors, but playing around with the depth preprocessor in controlnet helps.

Anyway, pretty happy with how it turned out considering I had to go zero to hero in a few days. It'd be great if you could check out the game and rate it (there's only a few hours left at the time of this writing.)

Thanks for reading and hope you enjoyed this,
Zivan