It promises to leave a deep mark and represent a significant turning point in the history of cinema. For a more democratic cinema, more present among the people, uniting… What was the original cinema, and what it has failed to be for so long.

In this article I want to talk broadly, technically, about the basic idea. And share with you some possible areas of using artificial intelligence to get the most out of it on an, intentionally, small budget.

We can also consider it an update, at a much more advanced stage, of the last article written in February 2023 entitled. How to Make Low Budget Movies with Artificial Intelligence – First Steps. .

The revolutionary impact of Artificial Intelligence in cinema

At one time, special effects and narrative techniques were the magic wands of cinema, but today artificial intelligence is playing a revolutionary card. It is an incredible help, an accomplice that opens the door to unprecedented innovation. We are riding this wave, with AI by our side lending a hand in creating compelling stories, digital characters that feel real, and music that gets right to your heart. But let us not forget the human touch, the real beating heart of each of our creations.

“Artificial intelligence and cinema” is not a token motto, but the beginning of a new chapter in storytelling. We are ready to prove that the future of entertainment is no longer a distant dream: it is here, and it is animated by artificial intelligence.

Luna Dolph, Kyle and China: from virtual to real life

The story, scripted by Gérard Bernasconi , starts with the story of Luna Dolph, Kyle and China; not mere characters, but three-dimensional avatars breathing life into a fascinating and complex virtual world. These avatars are not only protagonists of a story, but symbols of our age. Vivid representations of our progressive immersion in the virtual universe, also known as the metaverse. This concept, once relegated to science fiction, is rapidly taking shape and influencing diverse industries, from video games to social networking platforms, transforming the way we interact, work and have fun.

Virtual Reality and Digital Identities

Our goal is not only to tell a compelling story, but to invite the audience to reflect on the nature of reality in a digital age. These avatars, with their complex interactions and intricate worlds, represent the ongoing fusion of our physical and digital lives. A convergence that is redefining the very meaning of identity, community and belonging.

At the core of our narrative we find Luna Dolph, Kyle and China. They are three three-dimensional avatars whose lives take place in a virtual world of extraordinary beauty. While existing in the digital ether, their story is a bridge to reality, a means of reminding our viewers of the irreplaceable value of human interaction and real life. In an age when digitization has transformed our ways of connecting, our narrative aims to use technology not as a refuge, but as a springboard to rediscover and reinvigorate authentic and tangible sociality.

Technology helps you live better

As Luna, Kyle and China navigate a fascinating metaverse, their experiences and challenges resonate with a clear message: technology, however advanced, is a tool that, when used wisely, can enrich but not replace the warmth and complexity of human connections. Our goal is to bring out the realization that despite the attractiveness of digital, real life takes place off the screen. In the shared laughter, handshakes, hugs and spontaneous moments that make up the fabric of our existence.

With this storytelling, we aspire to inspire viewers to lift their gaze from their devices and re-immerse themselves in the real world, enriching their lives with authentic experiences. Through the exploration of virtual worlds, we want to celebrate and promote the beauty and irreplaceable importance of real life and human sociality.

The film will represent the first outing in the real world for the three main characters.

How we use AI in film production

We are still in the pre-production stage, so from a practical point of view I will keep you updated in the coming months. For now, we have a rough idea: we have selected a compendium of the latest artificial intelligence (AI) technologies that are both affordable and available to all.

The film will be in fulldome, immersive 360×180-degree format. We will basically project it into domes and planetariums. This is a crucial innovation looking to the future as we push more and more toward total immersiveness. And we have a limited budget, let’s say between 10 and 20,000 euros at a glance. The film will last about 40 minutes, and will be about 30 percent in the virtual world of Luna and her friends (entirely recreated in Unreal Engine ), for the remaining 70% in the real world.

Using Luma AI for the cinema of the future

To begin talking about this technological arsenal I mention the following Luma AI, a pioneering solution that completely revolutionizes the generation of three-dimensional environments. Luma AI allows us to reproduce real environments within Unreal Engine, in photorealistic quality and even with an iPhone. Employing advanced technologies such as the Neural Radiance Fields (NeRF) and the brand new Gaussian Splatting (the latter published just three months ago by the University of the Côte d’Azur, France), we can capture the complexity and richness of the real world in digital format, put it into Unreal Engine (including the ability to move freely within the scene) and bring to life scenes previously relegated only to large budgets.

We lower the costs associated with creating detailed settings and complex objects, as well as streamline and speed up the production process. Luma AI not only democratizes access to world-class technologies, but also gives us the tools to experiment at a speed that was previously unimaginable. And it allows us to impart an unprecedented level of vividness and depth to our scenes.

Skybox AI for cinema – Simplified lighting on Unreal Engine

Another crucial tool in our repertoire is Skybox AI by Blockade Labs , for creating immersive skyboxes. Enriching virtual scenes with vital details about lighting and setting.

A skybox in Unreal Engine not only provides visually convincing surroundings (mountains, sky, distant houses, etc…), but also affects the overall lighting of 3D assets within the scene. This is what interests us most in filmmaking: it acts as an ambient light source, reflecting its colors and hues on objects, helping to create a consistent and realistic atmosphere. For example, a skybox depicting a sunset will infuse warm orange and red hues on the scene; while a night skybox will provide a cooler, dimmer light. This process helps integrate 3D assets into the surrounding environment, making the entire visual experience more immersive and coherent.

Aspect that proves essential in creating 360-degree fulldome environments. Every tiny detail is critical to sustaining the illusion of a fully immersive world.

Using Kaiber AI for our film

Kaiber AI is useful and quality, but I have a serious doubt: I don’t know if it works with fulldome video. I haven’t had time to experiment with it yet, but it will possibly help smooth out the scenes so that the assets can be merged even better. It works through AI directly on the final video files.

Artificial intelligence and cinema: Reverie AI

In anticipation of the launch of Reverie AI , we are stimulated by its promises to create virtual worlds for Unreal Engine almost by “copying” a pre-existing picture. Its ability to generate scenery that faithfully mimics reality, combined with its potential in color correction of virtual scenes, opens doors to unlimited possibilities in visual storytelling. Reverie AI promises to be an excellent complement to our workflow, improving visual consistency and ensuring smooth and convincing transitions between the virtual and real worlds.

Move.ai, cheap and working mocap

Another “smart” tool we plan to use for our film will be Move.ai, specifically the inexpensive service Move One (on launch offer at $15 a month, then expected to cost $30). With a simple iPhone app, we will be able to create simple, ready-made custom animations without too many fixes or clean-ups. This reduces time and cost, allowing you not to be limited by the animations already available in services such as Mixamo , ActorCore or the MoCap Market by Noitom.

Here a very quick video test:

Clearly, they also have the more expensive professional service that allows up to 8 rooms to be used. But we do not count on using it for this first production unless there is a very real need.

Speech-to-speech for film dubbing… Is it possible?

In our production, innovation does not stop with the creation of the digital world; in fact, we also want to take advantage of the ongoing technological revolution in traditional aspects of filmmaking, such as dubbing.

The voice of Luna and the other 3D avatars, must be consistent (not to say identical) in all media, both in the virtual and real worlds. We face a complex challenge: maintaining the same voice across multiple platforms, including social media and especially real-time streaming, without tying ourselves to a single voice actor. This allows us to have flexibility in storytelling and to adapt to various formats without depending on the availability of a specific actor.

The idea is to replicate the voices of some real actors and associate them with avatars (Luna, Kyle, and China for starters), then transforming any actor’s voice, in real time, into Luna’s, to use it for both dubbing movies and social content (as much in the original language as in translations), and for the live streaming superimposing these entries on the animated avatar in motion capture.

From the excellent but expensive Respeecher, to the ambiguous Voicemod

We explored options such as Respeecher , an advanced speech-to-speech conversion tool, but the costs for real-time are prohibitive. We’re talking about 1,000 or 2,000 a month for a few hours of use. Voicemod presents itself as a cheaper solution because of its AI voices, although there are conflicting rumors about its reliability (some even consider it to be malware or cryptojacking…). And there remains the problem of not having the rights to the voice, which they own, which will certainly prove to be a problem in the future. I do not yet know the cost for real-time conversion of Resemble.ai , which I have used in the past for the much cheaper text-to-speech, and of Veritone Voice .

Another tool that I have not been able to test is Voidol 3 , at a cost of about $300. I couldn’t find a demo version, but I admit I didn’t even try that hard to request one. It is one of several Oriental software created because of the typical Japanese passion for the anime world. But actually adaptable to our purpose, as we will see in a moment with another Japanese.

MetaVoice Live, Mangio RVC Fork and W-Okada Voice Changer, free and open-source

The final solutions, after much research, are. MetaVoice Live and the Voice Changer from W-Okada. Both open source, which reassures us that we can base the “future life” of the characters on these services, and both free. I cannot fail to mention the YouTube channel AI Tools Search which has been most useful to me. Among the most interesting videos in this area, certainly this one:

I particularly like MetaVoice: it is under heated development and has a cloud version for non-real-time conversions that provides greater quality and flexibility. This one is not free, but the cost between $10 and $25 per month can all in all be dealt with.

In contrast, W-Okada ‘s Voice Changer has many independent developers developing solutions compatible with it, such as GitHub user Mangio621 who created the Mangio RVC Fork , a software with Web interface that can transform the voice of a real actor into that of the chosen voice model. Many of these models, especially famous people, are already available on sites such as Voice-Models.com in “.pth” format. A classic format used in machine learning), but the best thing is that with the same Mangio RVC we can perform training, or training, of a custom voice. All locally thanks to a good video card. And therefore clearly always available, and for free.

I will do a specific article on this shortly, however. So you will follow me as I do some interesting tests.

Generating video with artificial intelligence

One use of AI in film production may be text-to-video, or video-to-video. But how useful can generating videos with artificial intelligence be? I mean right from scratch, describing to the AI in text (or with a very simplified video reference) what you want to achieve. Somewhat the future, to date tools like Runway Gen-1 , Genmo AI , o Moonvalley AI are little more than experiments. Useful at some junctures, but far from the quality and realism needed for a film production.

Evidently, we will have to work for a few more years to get our films

Canon EOS R5C and Dual Fisheye lens for 3D fulldome video

What about live action filming? In our journey, we are trying our hand at a very exciting combination of equipment. We always keep the goal in mind: little expense, much return. I had originally planned to shoot everything in full 360 degrees, also to make it already compatible in case of future virtual reality porting. But the cost became prohibitive, both for the camera (a professional camera like the Insta 360 Titan , which by the way has not been updated for years, costs more than 17,000 euros), and for the difficulties of starting from the very beginning with such a wide field.

So the idea is to produce the right video for the dome, in 360×180 degrees (basically half the sphere). And the lens Canon RF 5.2mm F2.8 L Dual Fisheye , paired with the Canon EOS R5C , should prove to be a winning choice. This setup not only provides immersive images, but also allows us to experiment with stereoscopic shots, which add an extra level of depth and realism.

The Canon EOS R5C camera, with its dynamic range between 11 and 12 stops and 8K sensor, offers excellent value for money. This is an important consideration for us, as we are trying to maintain a balanced budget without sacrificing quality. We will also consider whether to rent it … From Adcom , lens and camera are offered at about 200 euros daily.

To be honest, we can’t fully exploit the potential of stereoscopy yet because of projection, but it is definitely something we would like to explore in the future. It is one of those things that looks really cool and could add a special touch to our project. Have you ever projected stereoscopic fulldomes or do you have any suggestions on how we could integrate it into our work? I would be happy to hear your thoughts and ideas.

And if it will be lacking in anything compared to more emblazoned rooms…. Again, we count on leveraging AI for video quality improvement. But research, in the field, still needs to be done.

Conclusions

In the end, our choice of digital tools and equipment reflects a desire not to compromise on quality while keeping an eye on the budget.

In short, we are creating something beyond traditional cinema. Thanks to artificial intelligence and cutting-edge technology, the “cinema of the future” is no longer a dream. It is real, we want to see it happen. And with the genius of Gérard Bernasconi to the screenplay (who also gave us a great technical contribution), the precision of Michela Sette in the role of VFX Supervisor and the creativity of Michele Pelosio as director, we are forging a revolutionary cinematic experience. Get ready, because we are about to take you to a world where cinema and reality merge into a transcendental experience

L'articolo AI in Film Production: How to Produce an Innovative Low-Budget Film è apparso primo per Future of Cinema - by Dario Riccio.

I got the idea, albeit unintentionally, from Nicolas Perrier from the University of Lyon in France, with one of his posts on LinkedIn. Perrier is a skilled expert in innovation in augmented and virtual reality, and the post in question is about Plask; one of the many tools for creating 3D animations starting from a simple video. In practice, it’s Motion Capture without expensive and complex equipment. This technology may be of particular interest to us to produce animated videos at a fraction of the current cost, and even with actors remotely.

I’m also getting additional help, not just from Nicolas himself with his countless posts, but also from the website Futurepedia.io. It’s an “AI wiki”, featuring a selection of many tools currently available to the general public.

Let’s analyze some of these tools, specifically the ones that are useful in filmmaking. Both for writing and for technical production of videos, as well as for voices. We’ll evaluate the quality of the results, conduct experiments, and learn about their costs.

To better understand how to use them in our low-budget films, I decided to create a short film (with very low expectations, just for technical experimentation) using them as much as possible.

Making videos with artificial intelligence.

Let’s divide the services into three main categories: writing, video, and audio. Starting with the writing, having to have the idea first.

Film writing with artificial intelligence

We need a story. Created by an AI? Let’s see, writing tools are not lacking. And if you don’t agree, you can always argue in the comments.

How does GPT-3 work?

Most public AI writing services are currently based on GPT-3, which has 175 billion machine learning parameters. The alternatives are actually many: BigScience Bloom, a large-scale language that has recently been launched (with the advantage of being open source), or the German Aleph Alpha with its Luminous (with 200 billion parameters).

What are the parameters of an artificial intelligence?

Imagine having a task that requires predicting whether an image contains a cat or not. A machine learning model could be trained on many images labeled as “cat” or “not cat” to learn to recognize the distinctive features of cats.

To do this, the model uses a neural network, which is a set of interconnected nodes. Each node represents a computation performed on the input data. The parameters, or “weights,” are the values assigned to each node that affect the strength of the connection between nodes.

When the model sees a new image, these weights are used to perform a series of calculations that ultimately produce a prediction, such as “cat” or “not cat”. The number of weights used in the model is the number of parameters.

In summary, parameters are like “ajuste values” that influence how a machine learning model processes data to make a prediction, and are modified during training to improve the model’s performance.

A model with many parameters will have more opportunities to adjust its calculations to fit the training data, but it may also be more prone to overfitting, meaning an excessive adherence to the training data and a poor ability to generalize.

Other competitors of ChatGPT.

Other competitors are Nvidia with the Megatron NGL (huge, but not available to the public ), Google with their GLaM, LaMDA and PaLM (yes, they don’t leave anything out), OPT by Meta, AlexaTM by Amazon or various Chinese ones like Alibaba and its BaGuaLu.

To conclude, I mention GPT-J, another open source alternative to GPT-3.

Let’s quickly understand how GPT-3 works, using a rephrased response from the AI itself:

GPT-3 (Generative Pretrained Transformer 3) is a state-of-the-art language model developed by OpenAI. Utilizes a deep learning technique called “transformers“to generate human-like text based on a given input.

Here’s how GPT-3 works:

1. The model is trained on a large set of texts generated by humans, such as books, articles, and websites. This training process allows the model to learn patterns and structures of natural language.
2. When it receives an input, the model processes it using its layers of neural networks. When it receives an input, the model processes it using its layers of neural networks. Each layer performs a different task, such as analyzing the syntax, semantics, and context of the input.
3. The model then generates a response based on the input and its understanding of natural language. It does so by predicting the next words in the sentence, using the patterns and structures learned during training.
4. The model repeats this process several times, generating multiple responses to the input. These responses are then ranked and evaluated by the model, and the highest-scoring response is selected as the final output.

GPT-3 is known for its ability to generate text that is often difficult to distinguish from text written by a human being. It is used in various applications, such as language translation, text summarization, and text generation.

ChatGPT can tell stories for screenplay.

Let’s start directly with OpenAI’s product: ChatGPT. It is perhaps the tool of the moment; the first one to have made a leap towards the general public. It’s almost concerning how accurate and “real” it can be in conversations, but it has also been discussed too much and everywhere in the last few weeks.

I signed up for the OpenAI website and opened the chat to try talking to GPT-3 (I can’t imagine what GPT-4 will do next) by asking it to write me a story with some dialogues for a short digital graphic film. And well, it simply made it for me.

The story written by ChatGPT

This is the story as written by ChatGPT, with no modifications, translated into Italian (you can read the original in the English version of this article).

Let us consider this first story. Of course, he didn’t make it easy for me…. He could create a story in the middle of a random city; in contrast, the jungle will be more complex to recreate. But I have to be in it, that’s the game.

The same story, in a short version

Oh, I also asked for a shorter version. He gave me a summary. Then I specified that I wanted a short version but with dialogue, and he simply did it. I am reporting it to you for completeness:

ChatGPT automatically writes the script

I was considering collecting it into a screenplay, when I thought, “Why not ask him?” Said and done. He rewrote the story for me already ready for film:

Apart from Inside Jungle, it is otherwise a good starting point. But some form errors jumped out at Andrea Rocci, an “human” screenwriter, albeit without an in-depth analysis (we were chatting over a beer).

In fact, the script speaks in pictures, and phrases such as “discovering more and more treasures,” or “rich beyond expectation” are not at all clear. Which treasures? Statues, coins, anything else? And, what do you mean by rich? Are they filled with gold on them? Are they dressed flamboyantly? Everyone with the latest iPhone and the keys to a Ferrari?

Not to mention the lack of descriptions of environments. The jungle itself is left to the fullest imagination of the director, or set designers (or 3D artists, whatever).

However, we must make a virtue of necessity; we will leave any choice at the discretion of the director (if he existed, at least…). We will try to find a good one on character.ai, perhaps. In fact, try it out and talk to artificial “characters.” Even Albert Einstein is there!

Prices

Here it’s simple: it doesn’t cost anything basically. A $20/month version is starting to be marketed in some countries, which removes some limitations (mainly due to the computing power needed to handle the millions of requests coming into OpenAI each day).

Alternatives to ChatGPT

At present GPT-3 is hard to beat… While waiting for the most emblazoned candidates to come out (Google Bard soon), I asked ChatGPT itself about its competitors. He pointed me to ScriptBuddy, WriterDuet and AI Screenwriter to begin with. Asking for more, Plotbot, Amazon Storywriter, and InkTip Script Listing. Okay, I thought that was enough… Except that the answer is actually a partial lie. Here we see the current limitations of this artificial intelligence, in part (but not only) due to the fact that the data it has is up to 2021.

Meanwhile, ScriptBuddy, originally based on GPT-2, has not been maintained for more than 3 years; we exclude it. WriterDuet, on the other hand, is most useful in script arrangement. But he doesn’t write anything himself… AI Screenwriter I don’t think even exists, at least dear old Google doesn’t give me any results.

As for Plotbot, or rather Plot-bot, it actually generates movie plots by exploiting the GPT-2 engine. But, like ScriptBuddy, it has not been maintained for years, and to optimize time, I rule it out. The same is true for Amazon StoryWriter, while InkTip Script Listing is not for generating text.

Dramatron

Dramatron is a new alternative, branded Deepmind, that promises exactly what we are looking for: writing screenplays for film and theater. Unfortunately, it was only released a couple of days around December 9, and by the time I got to try it, it was too late.

It remains an interesting project to keep in mind when it is opened. For the time being, George Mike, author of this YouTube video, does not like it.

His comment? “…overall, it seemed insipid to me. And it didn’t give me much more than the dialogue that you find in all the movies. He clearly grasped the genre, but did not explore it in an interesting way. It did not advance the narrative“.

Rytr

Through Google and Futurepedia I then came to Rytr. A nice tool. Account and login, and I find myself in the work area. He basically asks for language, style and use case of the created text, and he acts. Since there was clearly no “film script,” I tried meanwhile with a tool called a “magic wand.”

Same request made to ChatGPT, however, the response was almost hilarious:

Let’s try something different and more specific… I asked “I need a script of about 2000 words, with dialogue, to make a CG short film. The title is: the lost city”, and set as a use case “story plot.” Prompt based on the previous ChatGPT output. But we are still a long way off, although we have a basic storyline:

Same request, but with the magic command as the use case:

Okay, a useful tool for short texts but still far from total autonomy. We come to prices, although in our case I maintain ChatGPT’s text.

Prices

Rytr has a completely free version, limited to 10000 characters per month. Otherwise it costs $9 per month for 100,000 characters, or $29 per month for unlimited use.

Shortly AI

Another tool is Shortly AI, primarily designed to combat “writer’s block”. Write your text or a sentence, and he continues.

Since I could not ask him to do the story for me, I used the first sentence of the story generated by ChatGPT: Quando il sole sorse all’orizzonte, un piccolo gruppo di avventurieri si mise in viaggio per trovare la città d’oro perduta. Although based on the same language model, the continuation was clearly different.

This is the final text:

Ultimately, a good story but with little meaning. The AI took bits of text from here and there, but without connecting them in an “intelligent” way.

Prices

Shortly AI is certainly useful for adding a sentence or two and overcoming writer’s block, but to let it do its own thing does not seem the best tool.

Nor is its cost negligible. We are talking about $79 per month, or $780 per year. I still used the free trial, so it is only required to register an account.

WriterX

Let’s look at WriterX. It has templates for creating articles, social posts, text conclusions and many others. I expect similar results to Shortly, not being designed for whole stories.

Unfortunately, however, the free trial version does not allow access to all features, and I didn’t feel like paying for a service (even a fairly expensive one) without first trying it out; even having a free alternative that I am partially satisfied with.

Although I am evaluating it for its useful marketing features… Like the bio of my social channels, or any future help in writing blog articles. I also used it to write the meta description of this same article.

Prices

WriterX costs $29 a month in the standard version (basically the trial I had), or $59 a month to have unlimited text and functions. It is available in 25 languages.

Jasper AI

I also wanted to try Jasper AI, which is ultimately a graphical user interface for GPT-3 itself. It is perhaps the most publicized, found everywhere. But I simply haven’t even begun to use it: it forces you to enter your credit card even for the free version, and to verify it takes not a few cents but the entire first month: $29. Unprofessional attitude, so I don’t want to deal with them and wanted my money back immediately.

GPT-J and Writey AI

To get out of the GPT-3 universe, I wanted to try the open source GPT-J via the 6b.eleuther.ai website; however, it always crashed with the message, “Unable to connect to the model. Please try again”. And Writey AI, also well-functioning but too specialized in writing blog articles. Which I recommend you check it out for, if only because of 5 articles a month totally free.

Ultimately, I am tired and any further research seems futile. After all, ChatGPT’s text is valid (if you can call it an “automatic” text), so I would say let’s move on to the technical realization of the short film.

Creating 3D characters

Can an artificial intelligence generate 3D characters?

So we need the characters for our story, but is it really possible to generate them with A.I.? Spoiler: today, not well. There are many promises and some solutions that come close to the result, but it is not yet possible. Let us look at them in brief, as they will be useful in the near future. But then we will go on to figure out how to have the characters in our story now with little money.

PIFuHD

To start, there is PIFuHD, which is already available to the public and promises to create a 3D character from a single photo. It works, but even from the presentation videos, a quality far from acceptable in the cinema is remarked upon.

Google DreamFusion

DreamFusion is one of the first A.I.s that can leverage 2D data from Stable Diffusion (the same data used to create photos using artificial intelligence, which is now widespread), to create 3D models.

Same problem as before: unsatisfactory quality even at first glance.

Microsoft Rodin Diffusion

A 2023 newcomer to Microsoft, Rodin Diffusion is not yet available to the public. It promises to create a realistic 3D avatar from a single photo. Well, judging from the photos released on their site the hair… Those are always the problem!

Nvidia Get3D

You certainly cannot miss the queen of graphics cards, Nvidia, among the 3D tools. And indeed its Get3D is superlative in creating 3D models from information learned from a dataset of 3D models.

The dataset is much more limited, and this is a disadvantage in the variety of 3D objects that can be realized. It is, however, open-source, a definite plus point, although the entire training must be done in one’s own system. This means high-end video cards galore… And they cost a little. Much.

Nvidia Magic3D

How did Nvidia solve the problem of the “limitation”, in terms of quantity, of objects that can be created by Get3D? As the article in The Decoder explains, simply copying Google… And trying to make its new Magic3D (researchers’ paper here) faster and more defined than DreamFusion. This video explains well how it works:

In practice, Get3D has a dataset based on other 3D models. Instead, Magic3D starts with images, themselves generated by A.I., paving the way for virtually infinite combinations.

Does it work well? What is certain is that it is not available to the public, but from the videos and examples on the Web it looks like a very promising technology. Although it still does not reach the necessary quality.

StyleGAN-NADA

For doing Pokemon or other fun things there is StyleGAN-NADA, trained following OpenAI’s CLIP (Contrastive Language-Image Pre-Training) model. It allows you to create images from just a textual description, without the need to see any references and without the need to collect additional training data.

It is also possible to modify existing images to make them similar to those in other domains, for example, using an image of a dog to generate a cat. The same approach can be applied to other generative architectures, opening up interesting possibilities for creating images quickly and accurately.

Pollinations

Among the “next steps,” Pollinations promises to do what we need it to do. From their website, “at the research level, our team is developing technology that allows people to generate 3D objects and avatars with the help of text alone”.

Here again, there is a wait. For now, it still allows you to do interesting things in the photo/video area. Maybe try it, however there is little of use for the purposes of this article.

Text2mesh

Small but interesting, Text2mesh is less of an exercise in style than its predecessors. Here you already have to have the model, but the AI promises to modify it independently; for example, increasing the number of polygons, changing its shape and color even creating the texture from scratch. All based on a text prompt, a written request.

Reminder to put in the diary in case we need it.

Luma AI

Luma AI is an interesting project to scan real objects by recreating them in 3D. The operation is interesting, and the quality of the scans is reasonably good. For props or figures in the background, I consider it more than acceptable, even in production.

It has also recently allowed you to create objects, and thus characters, in 3D from a text prompt. With the classic “imagine” command, already made famous by the MidJourney image generator. But here, again, the quality is not sublime. Good experiments, but definitely not usable for production purposes.

3D characters and objects without artificial intelligence

From all of this we understood only one thing: A.I. as of today, January 2023, still does not allow us to have good 3D models. Since we have to keep the budget low, however, let’s achieve them with the tools already available.

3D characters in our short film

I won’t go into a lot of research here, but I evaluate two 3D character creation tools that I already know: the simple Reallusion Character Creator, and Epic’s fantastic MetaHuman.

We will need 3 characters for our story: Maria, Jake, and Sarah.

MetaHuman Creator

Actually, in the case of MetaHuman there is a change from my past: I used the beta of MetaHuman Creator. It is phenomenal in that it has given me the ability to take advantage of the computing power of Epic’s servers by creating characters in a work break directly with my laptop (which only needs to receive a video stream).

So by requesting “Early Access” with one’s Epic Games account from metahuman.unrealengine.com, we end up with a choice of possible characters.

We imagine that all three are between the ages of 20 and 40, otherwise the script does not tell us much about them.

Jake

I choose to begin by selecting Aoi, as Jake. I don’t know, that beard gives me the idea of “adventurer.”

The software warns that some elements of the character has elements still under development (specifically hair) and therefore only LOD (level of detail) 0 (automatic) and 1 (highest quality) will be displayed. For us it is fine, the destination will be a pre-rendered video clearly at the highest quality and not a real-time video game.

A few changes to the character (shirtless, eye color, more “suitable” shoes and pants), and he is saved. Then we will export it with the Quixel Bridge plugin of Unreal Engine 5.

Maria

Let’s move on to Maria. I asked ChatGPT to come up with its characteristics, and the answer was that it could be a woman around 30 years old. Brown hair, shoulder-length and pulled back into a ponytail. Brown eyes, intense and deep, and of Latin ethnicity, with skin tanned from his outdoor adventures.

Let’s try to realize it. Let’s start from Roux. Let’s make a Blend with Lena, Kendra and Tori who seem suitable to modify her features and thus ethnicity a bit, give her a ponytail, brown eyes, modify the texture of her skin to give her a few more years, eliminate the make-up she would hardly have in the middle of the jungle, modify her clothing and that’s it.

Sarah

Finally Sarah. For ChatGPT he is about 25 years old, with short, wavy blond hair. Blue eyes, lively and inquisitive, around 1.70 meters, slender and muscular indicating an active and sporty person and of Northern European descent, with fair and delicate skin.

We rely on Vivian, various modifications until we make it something like the required. Clearly MetaHuman has many limitations, even more so in this online version. For example, on the body we have practically no possibility to operate, so “muscular” is a feature we will have to renounce, unless we model later. But, first I am not a 3D modeler; and this is a zero-budget project for educational purposes only. Also, for the same reason, it is not the case to waste more time on it than necessary.

Finally let’s leave them there; we’ll create the animations with mannequins and then we’ll retarget with MetaHuman characters directly in Unreal Engine 5.

Conclusions

Let’s limit ourselves here for today; in one of the next articles I will talk specifically about animation and Motion Capture with artificial intelligence (where it will be most useful to us), and then we will continue with environments, voices, music, and whatever else we need to finalize our little project.

I will give myself time to finish slowly, and possibly do other articles before continuing this one. For two reasons: they are elaborate operations, and artificial intelligence is in an explosive phase. An article written today may be old tomorrow. Maybe tonight.

Therefore, since we will need this information much more toward the end of this year for the actual production of a fulldome story, let us keep in mind all the news in the coming months.

As always, thank you for following me, and a hug.

L'articolo How to make low-budget films with Artificial Intelligence – Early stages è apparso primo per Future of Cinema - by Dario Riccio.