Flamenco 3: A New Beginning
- Development
- 1 Mar 2022
- 4 min read
Almost 5 years after the release of Flamenco 2.0, development has started on a new version: Flamenco 3.
In this blog post Sybren Stüvel describes the current vision on Flamenco 3, with the intent to discuss this with anyone interested, get feedback, and improve the design.
Flamenco is the Open Source render farm software, developed by Blender Studio. It is aimed at performing tasks such as frame rendering, and video encoding. It supports any Blender version, hardware configuration and any other software with a command line interface.
Update 2022-03-02: while the original design proposed to use PostgreSQL as database, the current direction is to leverage an embedded, no-need-to-install database instead: SQLite. The blog post has been adjusted to reflect this.
Target Audience
Flamenco is meant for smaller animation studios and individuals at home. Think of roughly 1-10 artists using it, and 1-100 computers attached to the farm to execute tasks. Blender Studio uses various desktop machines for the render farm when they're not used by the artists, and some developer machines are powerful enough to run Flamenco and work on coding Blender at the same time.
Design Principles
The following principles guide the design of Flamenco 3:
Blender.org Project: Flamenco is a true blender.org project. This means that it's Free and Open Source, made by the community, lead by Blender HQ. Its development will fall under the umbrella of the Pipline, Assets & IO module.
Minimal Authentication & Organisation: Because Flamenco is aimed at small studios and individuals, it won't offer much in terms of user authentication, nor the organisation of users into groups. The first version will likely just have a "name" field when submitting jobs, and people are trusted to not mess with other people's jobs.
Minimize External Components: Running Flamenco should be extremely simple. This means that it should depend on as few external packages as possible. Apart from the Flamenco components themselves, all you need to install is Blender, FFmpeg. And we're even thinking of bundling the latter with Flamenco.
The downside of this is that development might take longer (because some things that an external service could solve need to be implemented).
No Errors, Guide Users To Success: Instead of stopping with a "no database configured" error, Flamenco should show a helpful interface in which you're guided towards a working system.
Customisable: Studio pipeline developers / TDs should be able to customise the behaviour of Flamenco. They should be able to create new job types, and adjust existing job types to their needs. For this, Flamenco 3 will most likely be using JavaScript to convert a job definition like "render this blend file, frames 1-100" into individual tasks for computers to execute.
These custom scripts should also be picked up by the Blender add-on that's used to submit files to Flamenco, so that there is just one place where customisations take place.
Work offline: Like Blender itself, Flamenco should be able to fully work offline. That is, work without internet connection. If any future feature should need such a connection, that feature should always be optional, and be disabled by default.
Data Storage: Data should be stored as plain files whenever possible. Where a higher level of coordination is required, an embedded database can be used; the idea is to use the no-install-required SQLite for this.
Infrastructure & Supported Platforms
Setting up a render farm is not as simple as pushing a button, but Flamenco aims to keep things as simple as possible. What you need to run Flamenco is:
- One or more computers to do the work, i.e. running Flamenco Worker. These machines also need Blender, of course, and FFmpeg for video processing.
- A computer to run the central software, Flamenco Manager. This could be one of the above computers, or a dedicated one.
- A local network with file sharing already set up, so that the above computers can all reach the same set of files.
Since Blender Studio fully runs on Open Source software, Linux is the main platform Flamenco is developed for. Windows and macOS will also be supported, but will need help from the community to get tested & developed well.
Development Plans
Sybren has been working on a proof of concept, which has now been published in the official Flamenco repository on developer.blender.org. Anna Sirota, Pablo Vazquez, and Francesco Siddi are also part of the team.
It will take several months of development to produce a minimal viable product.
My Questions To You
The goal of this post is not only to inform, but also to get feedback.
- What do you think about the design principles described above?
- If you considered using Flamenco 2 before, but didn't, what were the main hurdles?
- How do you feel about having to install Blender, FFmpeg, and some file sharing system (NAS, Samba/CIFS/NFS, that kind of thing)? And then tell Flamenco where to find those?
Leave a comment below, or join the discussions in #flamenco on Blender Chat.
19 comments
Sounds great, looking forward to using this! For reference here's our use case
We are a small game studio typically 5 people. We use Blender to render cut scenes, but since most of our effort is on developing game assets film production isn't our main focus. And typically we're 2x2x1 (2 devs, 2 artists, 1 'slash' director/artist/developer/composer/IT/HR/etc (yeah me) making the administration of this as simple as possible is a perfect goal. Presently we're just rendering on demand on our workstations but being able to farm out jobs - if it would simplify this task, would be great.
Our infrastructure is a business class NAS which runs all the tools - Perforce server, Perforce Swarm, Jenkins all in individual Ubuntu VM's. Adding another VM is for the manager VM is possible, with as simple configuration as possible is great. Target/support Ubuntu server 20.2 please - if you can support dpkg format (in a public apt server repo?) that would be best.
The render machines have to be Windows - no choice on this, again simple configuration would be best. For example presently Jenkins master is on the Linux VM as mentioned, and it farms out the build/test to Windows boxes with a high end GPU. Jenkins likes best to be on Linux, it takes some hoop-jumping to get it on Windows but it works fine otherwise.
If configuration isn't simple/obvious, then good install documentation will be needed and a fine solution. Installing Blender/FFmpeg etc is just fine - just make it clear what needs to be done exactly. Setting up Perforce and Swarm isn't simple, but those platforms have excellent installation documentation.
Accidentally hit return ... So basically think of your user as a 'slash' developer (IT is a part time job he really doesn't want to do), so installation and setup that doesn't require any thinking is best, if you know what I mean. Just push buttons and it works.
With Flamenco 2 we didn't look into it - the barrier was it just seemed too much trouble to be worth it. But with this being targeted to studios like ours I'm interesting in giving it a go. Hey if it simplifies cut scene development and speeds that up that's great.
On your new simplified architecture that looks great - it is similar to Jenkins. KISS - one of the things I dread as the IT guy is things getting out of date, or the system breaking down and I can't remember exactly the details of how it's set up.
On authentication groups it's not a big deal either way. Keeping user/group is fine - we have that already with Perforce and I rather like it because as we expand we're ready for that. So I'd suggest keeping some good auth with groups - it doesn't need to be overly complicated, but having a system ready for growth isn't a bad idea.
On the bundling/simplifying of dependencies, you know that's not a problem. Installing packages "apt install XYZ" is nothing. As long as Flamenco keeps current with latest it's not a problem. What I'd suggest is instead (since this will probably be a simpler development solution) to do the public dpkg depot as I suggested above. That way you can build in dependencies and installing will automatically install those too. I'm not sure how this works, but since you're open source I'm sure you can simply add Flamenco to one of the main Ubuntu update servers - that would be the best. Then just "apt install Flamenco" and it does everything needed, and keeps it up to date.
Final note, I happen to be baking an Ocean modifier at the moment which will take hours. If this supports these kinds of use cases (baking, physics, and even baking in the new texturing system) that would be lovely.
@Dan McLaughlin yea Imma second this one, I'm having trouble baking with this setup, and this would certainly ease up my workflow if I could cloud bake my materials as well
@Coker Yeah looks like this is all possible in Flamenco with a custom job and some python
OK more thoughts :) On this modifier point, our Hero assets utilize a high poly modifier stack that can take forever to calculate, and the problem being it obviously locks up Blender, and eventually might crash if you have a setting too high. I'm not clear on how submitting jobs will work exactly, but if it's of the form
- In the present Blend file, use the Flamenco addon to submit a job very simply and easily
- For example, take this object and it's modifier stack push it off to the render and leave this file alone, toggle the modifiers all visible, then export them out to a FBX or render an image.
- Just do it quickly, I should be able to just hit a key combo or button to send it off and forget about it. The idea is I don't know if I have my creases right or whatever but I need to test it, so if it later comes up with a render or model I can tell if it's good or not.
- This applies same to a cut scene render. Maybe I just want a temp created I can throw off to our audio guy so he can work.
- Notifications: If my original Blender instance could get a notification, and if it could receive the results (e.g. a single frame render of the model) that would be great. Ideally in a way where it didn't dump much stuff in the Blend file - I don't think we want to keep a reference to the sanity check render.
- Custom notification: Sending a notification to Teams would be ideal, as I can get that on my phone/watch. All you need is a simple web hook with a JSON payload, also need access to the header IIRC. So a generic form with
URL _________________ Body __________________ Form (checkbox) URL Encoded, JSON Encoded, XML Encoded Optional custom header And some kind of syntax to customize the message would be great, e.g. {Job} from {Who} completed {Status} on {Date} referencing {ObjectName}
Something not mentioned AFAIK - what is the GUI or access? A web view would be typical and the best solution, have it run an embedded web server or something so we can easily access it remotely, our other tools work this way.
@Dan McLaughlin The Manager will have a web UI, indeed. The Workers likely won't have any UI at all, and can be managed from the Manager's web interface. At some point the Workers might have a GUI for easily managing on someone's desktop machine; that's probably a good task for someone in the community to help with.
@Sybren A. Stüvel Sounds good. On the ability to create workflows on the modifier stack, not just rendering, do you see that possibility? Even something as simple as running a python script in the blend file should support this. Along with custom webhook notifications those are the two most important features to us (along with the rendering).
@Dan McLaughlin Not sure what you mean with "create workflows on the modifier stack". Maybe pop over to https://blender.chat/channel/flamenco, it's an easier place to discuss things.
Hello dr. Sybren, I've been using the current flamenco for about two years now. I tried some alternatives but ended up keeping flamenco since it's so well integrated with blender and supports 100% of the features. It's not the most userfriendly though. I had to set it up a couple of times since we moved the workplace, and then changed all the computers, and I think every time I ended up spamming a thread on devtalk as I encountered different problems (thanks for your help by the way, and sorry for not finding the chat sooner). It takes a bit of getting used to, but it's a real time saver in the end. From what I gather from your post and my experience so far, I think a fully offline experience should make things smoother and faster. We have a poor adsl connection at work making the flamenco server not very responsive, it's a bit of a hassle to find which task is currently rendering from the list, clicking next page, waiting 5 seconds to refresh etc etc. However, I did like the ability to connect to the server from the comfort of my home and see if eveything was still rendering correctly over the week end. So keeping this as an option would be nice for some scenarios. Also, one thing that was bugging me, was the inability to give a name to the rendered frames. Only the folder is named and all the frames within are just called ######.png. I figured it's probably for the better, since it allows to quickly swap single images and avoid the occasional typo in the name, forcing a manual rename of hundreds of frames (and I will probably keep this workflow from now on). But for a user that set up a name for his frames on his local blender file, he might expect flamenco to keep this name when rendering.
As for the last question, what I liked in the previous design is the use of "portable everything". I just set up one worker, with Blender and flamenco-worker in a folder, zipped that folder and extracted it on all my other workers. FFmpeg was put on the shared drive so that all computers could access it. Very fast to deploy on many computers. Having to actually install Blender, FFmpeg and a database on all machines could be intrusive for some cases and time consuming to maintain/update.
The last thing I could point out (I've only used windows, so it might be platform specific) is that the console is very hard to read. There are no colors, the lines are very long and often take more than one line for a single "sentence". Idealy, if the workers/manager could be minimized as an icon in the icon tray, and a simple interface to show errors and progress, it would make things cleaner and more userfriendly.
That is all I can think of for now, I am vey looking forward to seeing what you can conjure up :) Keep up the good work and many thanks to all of the team for this effort !
@Benoit Alexis Thanks for your feedback!
Having to actually install Blender, FFmpeg and a database on all machines could be intrusive for some cases and time consuming to maintain/update.
There is no need to install the database (it'll use SQLite, which is built into the executable). Blender won't all of a sudden become "must-be-installed" software; it's always been portable. And I'm pondering ways of embedding FFmpeg into the Worker executable as well, for further simplicity.
About the system tray: that's very system-specific, and on Linux it'll even be desktop-environment-specific. It would be great to have some GUI, but I want to do that in a way that doesn't all of a sudden require GUI support on something that could otherwise be a headless Linux server. Still, I fully agree a simple systray icon would be best :)
Simplicity sounds promising.Tried using a previous version of Flamenco, it worked really well for the project I set it up for (I was using it with
@stephen thomas Arg, apologies, I posted that before I was finished and can’t find a way to delete it. As I was saying, it worked really well for the project I set it up for (was using the blender cloud to manage the farm), but there was a problem when I tried to use it for the second project, didn’t have time to work out what was going wrong, so went back to rendering using the terminal. I’m quite a technical user, but if you’re aiming this at blender artists, asking them to install ffmpeg and a database might be a bit of a tall order. I’ve walked artists through doing pretty simple things in the command line before, it’s always difficult for them as it’s something they’re not used to. If it’s not double click to install, I’m not sure on the uptake for average users. I’m not sure if this is surprising, but even asking people to type out the location of where you’ve installed Blender & ffmpeg may be too much... Similar sort of thing to installation, users aren’t used to typing out commands or full file paths, one wrong slash somewhere and it doesn’t work will most likely mean they just give up.
@stephen thomas Thanks for your feedback!
I'm actually pondering a way to embed FFmpeg into the Worker executable, so that it doesn't need to get configured, or even be installed beforehand. Minimizing the number of variables you need to juggle to get things working is one of my primary aims.
I'm looking forward to the design without additional server. I had a lot of problems with tasks not beeing updated on the server. Another hurdle for me were the config files I needed to touch often. The example file for the manager had good explainations, however it didn't worked quiet away. It took me a while to figure out that the test render file was not available. Also ports made trouble. I learned something but I could have survived without learning this. Good to see that the answers it gives will be improved. Easier is of course better. Will it handle changing ips? 🐈 btw. No installing installing 3 things is not a problem. Thx a lot for improving it. 🙏🏻😁🙏🏻👍🏻
@Björn Eckhardt I haven't considered changing IPs before. What kind of setup would need that?
I'm crossing fingers and toes for this. With a simple 'how to' for installing the worker, and a check-list for the other computers in terms of setup, this should be something most of us (users) could manage I think ... :)
@Knut Einar Skjaer I did of course mean 'how to' for installing the manager, and a check list for the workers ... :)
Join to leave a comment.