Notes for Creativity Training

Generative models of all kinds of modalities, including textual, image, and audio, are evolving very fast. They have started to replace humans in some labor-intensive scenarios, and they will keep doing so. Like many others, I believe this is a good trend in the long run. It’s very similar to the Industrial Revolution when much human labor was replaced by machines. Yes, it will cause some short-term turbulence. But look at the colorful clothes normal people on the street are wearing now, which were privileges of the higher classes a couple of centuries ago. How could we have achieved this without the Industrial Revolution? I believe we can expect an even more “colorful” future with AIGC technologies.

Generative models are like pens, brushes, and dictionaries, which serve to boost our productivity. Why is that? In the past, without pens or dictionaries, it was difficult for many people to read, write, or paint well; in the future, with the help of generative models, more and more people will be able to write and paint decently. And this to me, sounds like a liberation for humanity.

But tools are just tools. They can make you more productive but don’t necessarily make you more creative. I believe with the liberated productivity, we humans will and should devote more time to being creative.

As a practice for myself, I open this blog to keep track of some of my ideas. They are not brilliant ideas, and probably many of them sound stupid and funny, but I would like to see some of them come true because I believe they can help make the world a better place. Maybe some of them already exist in a corner I don’t know yet. If I come across them in the future, I’ll come back and add pointers to them.

I first started this exercise in my notes and thought that I was going to realize them at some point. Slowly, I came to feel that I wouldn’t be able to work on most of them, so why don’t I make them public? If they can be useful of any sort, that’s the best I want to see. But probably most of these ideas just don’t make much sense, and in that case, I hope you can still have some fun reading it. I also welcome everyone to join me for brainstorming new ideas, as an exercise. Just keep practicing, and who knows one day we won’t come up with a brilliant idea that can influence the world?

NOTE: Some of the following ideas, if not all, already have predecessors in the market. I still have them here because I believe they can be boosted by modern technologies, such as generative AI.

Using video game addiction for good causes

Video game addiction has been a concerning social problem for years. However, I’m afraid this is only going to get worse with the climbing unemployment rates – a lesson learned from the Hollywood growth during the Great Depression. The game addiction problem should receive more attention, especially because addicts are usually juveniles and young adults – the group of people who have the most potential.

Although new, responsible careers have been developed around the video game industry, and even the Olympic Games have adopted the Esports series, it’s usually difficult to see how people’s time spent playing games is directly benefiting themselves or society. Let’s make some comparisons. Bakers and chefs feed people, bus drivers transport people, and teachers educate people. What do gamers contribute, especially when they spend too much time? That is the main reason why our parents have strong stereotypes about video games, and they do have a point.

Is entertainment the primary nature of video games, or is it that we haven’t tried hard enough to endow them with more usefulness? I’m leaning towards the latter. It’s good to see that people from the serious games community have never stopped trying to bring values other than pure entertainment to (video) games. I can’t help but imagine what it be like if all game developers changed to work on serious games! It must be a whole new world where both education, relationships, work, and more are revolutionized. Metaverse is arguably a concept in the light of this possible future.

Just a side note, with the stunningly fast development of AI, I tend to become a believer in the Simulation Hypothesis. Imagine that one day we can simulate everything, including all human senses and interactions with the world, it might be appealing for us to submerge the virtual reality for our development as well as that of society. And this is becoming more and more realistic.

In the spirit of channeling game addiction to good causes, below are some ideas that can help. If we can’t overcome game addiction, let’s make good use of it.

Shoot to label

Among the top 5 most played games on Steam,¹ 3 are FPS games. What if we replace the players of human shapes with images that we collected from the real world, and ask the players to shoot the specified targets? For example, we collect some images of animals, in all lighting and surroundings, then we ask the players to shoot for a random type of animal in each round. In this way, we turn the FPS game into a labeling task! The images receive more bullets and are treated as annotated with higher quality. We can even mix in some “golden labels” to decide the accuracy of players’ shots, and also as a metric for deciding their ranks. Higher ranks mean higher skills in this task. Annotated data can be sold to AI companies to develop stronger models that will be put to real use.

Similarly, many other kinds of data can be collected and utilized. For instance, making use of the popularity of racing games and vehicle simulators, driving data can be used for training auto-piloting systems (maybe this is already happening). This also has the advantage of being privacy-proof because all the collected data is in a game. These ideas can make it possible that in the future there will be such a job that pays workers for playing games, for good reasons! In the future, maybe we can all sit back and play games as much as we like, and AI’s work to take care of everything else!

Games as textbooks

There is a type of game that’s called visual novels. It should be very simple to develop games for teaching history, literature social norms, etc. Science teaching is also possible, and potentially simple. Just design math exercises, physics, and chemistry experiments as engaging puzzles! I know it’s easier said than done, but we can keep adding weight to the right thing until it reaches the tipping point. Education has been always losing when it fights against entertainment like games. What’s different now is that we can seek technologies like video games for help. Especially with generative models getting popular, the development process might be hugely benefited.

The only concern is excessive screen time! But maybe this can be solved with the development of brain-computer interfaces.

Stock market simulator

Inspired by Generative Agents², we can also take stock market simulators to the next level. With generative AI, we can not only simulate stock managers, with the hope that they can help us with things like buy/sell decisions, but also simulate the world dynamics so that we can get trained for handling black-swan events, without the loss of a fortune.

Classroom simulation

Thinking along the line of simulations, another idea (that probably requires more developments in related fields) is to create a classroom simulator. When simulating the teaching from teachers to students, all powered by AI models, humans can supervise the process and give feedback. We can define what tools in the real world can be used/taught and how through which we make sure that AI’s understand what they’re doing. Besides, novel ways of teaching can be tested in this simulation. Human students can also learn in the virtual classroom.

Caring for others

With the progression of AI models, embodiment may also see a surge soon. When they are combined, many caring tasks can be done in an effective, quality, and cheap way. The dream of employing robots as tireless servants may come true soon. Even if they can’t undertake too many tasks, they can at least ring alarms and get human help in time.

Elderly companion

Hopefully, with liberated human labor, people can spend more time with family, but that’s in the far future. It should be possible already to build virtual companions for the elderly. A tool like this can not only keep the elderly company when their children can’t, but also help seniors make emergency calls when needed. Aging is a problem with increasing severity. Such a product can not only help address the caring but can also do it in a very accessible way. I hope such a product is already on the way.

Scientific research related

While generative models are receiving welcome from many many areas in the world, they are still very immature in the eyes of some scientists. For example, Yann LeCun has been known as a big protester against LLMs. Many things need to be done or well understood for LLMs, but IMHO this fact doesn’t rule them out from being useful. Below are some ideas with a focus on making generative models more useful, and with some values for research.

AIGC driven by brainwaves

Imagine that you don’t need to do prompt engineering when using AIGC anymore. Just focus on the idea in your mind, and the generative models will draw/write it for you! I feel this might have been studied because the concept is simple in theory. Generative models essentially use vector representations to work. Even though you input texts to ChatGPT or DALL·E, the texts are translated into vectors before the real work begins. Brainwaves can be easily represented as vectors with simple transforms, so it should be possible to finetune generative models with brainwave data. Indeed, this has been proven to work on image generation with the latest CVPR paper.³ Similar applications to text generation should be around the corner too.

UPDATE 10/29/2023: A new work from Meta⁴ has proven the possibility of decoding brain activity into images! In the following years, there will be more and more breakthroughs in using human-machine interface and AIGC to do work.

Use LLMs for ethical hacking

Add tools for a specialized and close-sourced LLM (because this is not safe to open-source) to generate code and API calls for hacking a target system, to find unknown loopholes.

Human-pet translator

When a multi-modal model is trained on textual, audio, and video data collected from both humans and pets, it might be possible for the shared model to capture the same patterns. If so, then a pet audio input can be translated into human speech or text and vice versa. This is a bold idea, but human brains share a lot in common with those of pets. If neural models can learn human languages well, it should be possible for them to learn that of pets’, as long as we find the right way to train such models.

Remarks

I know some of the above ideas may sound foolish, but I’ll keep working on them with good intentions in mind, in refining the ideas or implementing them. I believe I can be valuable as a practitioner or as a day-dreamer, as long as I put my time and energy to good use. I’m sure the compounding power will not let me down in the end, but anyway, I’m enjoying this process already so whatever. I’ll also keep updating this post as long as I can until someday I feel the energy needed for updating this post is better put on another more meaningful thing.