AI-powered text software Chat GPT is currently THE tech topic. The speech robot is supposed to be able to create texts with high precision and tell entire stories – all within a few seconds. But what’s really behind the hype? Can AI also work with highly technical issues for a software company? Will the texts be just as good, concise, and fluent as from human hand? We tried it out for ourselves. Here’s a review of our experience with Chat GPT-3.
Our learnings with Chat GPT-3 (based on German texts):
The very first time we googled Chat GPT in our team meeting chat and were linked to the company Open AI, we were told that the AI was currently at its maximum capacity limit. This will turn out to happen more often (even with the latest version Chat GPT-4, which we will look at in more detail in part 2 of our field report). Apparently, so many people are currently benefiting from the life-simplifying artificial intelligence that server capacities simply cannot keep up with the demand. This also means that the AI bot is still in its infancy in terms of what will eventually be possible – an unlimited, scalable, simultaneous access by people and companies around the world. A world that could benefit or become dependent on the intelligence and reliability of AI?
Although the hype around Chat GPT only picked up steam in early 2023, the chatbot and the idea behind it have been around for a lot longer. In addition to Chat GPT, Open AI also programmed Dall E, which can create images based on text – a capability that the new Chat GPT-4 version can also handle. Fun Fact: Open AI was co-founded by Elon Musk in 2015 (he left in 2018). The free version, Chat GPT-3, is still enjoying a boom, explained by the practical relevance that such a chatbot can represent in our everyday lives. Texting has become as deeply integrated into our daily tasks and routines as eating and sleeping – whether it’s writing emails, articles, applications, chores, or WhatsApp messages.
We quickly ask ourselves whether Chat GPT could possibly bring the long-awaited relief to the ever-grinding mills of B2B storytelling. Or will AI even challenge our editorial skills? We started a comparison experiment and challenged the AI bot on our software topics IoT, IIoT & Cloud Native Design. To do this, we wanted to test the following formats: a social media post, a mailing text, a blog post, an interview and the summary of a solution paper into a 1-pager. Can AI technically answer as correctly, concisely and with the necessary pinch of promotional enthusiasm as a human? Will it work without the perfectly reduced, structured specification of a “prompt”?
Prompt to Chat GPT: Put yourself in the role of a senior product manager. Share a tip about cloud migration in a LinkedIn Thought Leadership Post. Target group: Senior IT managers. The focus is on challenges that can arise when migrating Internet-of-Things systems from on-premises to the Azure cloud. Talk about the benefits of cloud deployments. Keep the post short and use concise sentences. The tone should be informative and helpful. End the post with an example.
We input the prompt in German as well as in English into Chat GPT-3 to see which language the chatbot is more familiar with and whether the language selection would make any difference in general. The result: The English text is significantly shorter, better structured and more informative than the German, due presumably to the more extensive English database. All in all, the AI text provided a very good framework for LinkedIn mailings and could speed up production – especially for non-texters from Sales, Product Management & Co. If the fluently pre-formulated building blocks were supplemented with suitable company-specific expertise and references, the reader would receive a solid, informative B2B e-mail.
Conclusion mailing: good framework for mails, fluently formulated, English better than German, to be supplemented with individual company insights.
Prompt to Chat GPT: Prompt to Chat GPT: Write a short announcement text for social media (LinkedIn) announcing the new podcast “We talk IoT” with Dr. Christian Liedtke and Andreas Frank from KUKA. Identify matching emojis and hashtags.
Short and concise texts should be Chat GPT’s masterpiece: The task here is to distill the most important facts. We wanted to find out if we could get a better result for a post in a matter of seconds with Chat GPT-3 than through our social media editors. We quickly realized however, that brevity has its pitfalls for the AI bot. Each answer turned out a little longer than desired. On the other hand, the bot – and this is where we have our first “aha” moment – can write entire essays on complex topics at the drop of a hat. However, the final text design of the post, at the appropriate length and with the necessary impact, still required comprehensive fine-tuning by us. After each submission, we had to tweak the text, get the bot to shorten further, point out the right core topic, and optimize emojis and hashtags.
We still got a good social media text with Chat GPT-3. Surprisingly similar to our own even although it took time to get there due to the many adjustments. After all, Chat GPT provides a variety of wording suggestions for promotional social media posts, from which you can choose according to your own taste.
Conclusion social media: Good, but not 100% to the point.
Our December 14, 2022, blog post “From 1G to 6G: How 5G is powering the Internet and what comes next” is perfect for a chatbot counter-comparison. The topic of 5G has come up frequently in recent years, providing a broad base of data before 2022 that can be picked up by Chat GPT. Expectation: A high quality and informative text with source references that can match the quality of our already written blog post.
Prompt to Chat GPT: Write a blog post about what the 1G-6G technologies mean in detail. Highlight their relevance to IoT. Explain a classic use case for real-time data such as Condition Monitoring: “Data processing and visualization are most useful when they occur in real-time or with minimal delay, only then can process optimization be achieved.” Include a concrete practical example from a user’s perspective. Connect the topic of IoT data to the Open Industry 4.0 Alliance. Add a title and three subheadings.
Experience with the AI bot showed that a prompt must be specified down to the smallest detail. If you don’t explicitly ask for headlines, you just get a long text. However, it was skillfully built up and, similar to our own text, followed the history of mobile communications from 1G to 6G. Chat GPT-3 needed just one second for this until it gave us its answer: We received a text including headlines that, at first glance – second “aha moment” – resembled our blog post. It would appear as if Chat GPT “thinks” the same way as a human. The only difference is that an AI software does not think, it merely imitates our behavior. The comparison of the original post (nochmal verlinken) with the new version shows just how good it is.
Once again, the AI copy provided us with a good text structure, but in terms of content, the post remained very superficial. Many phrases were too general and repetitive. Especially in B2B marketing, where it is essential to elegantly integrate a company’s expertise and competence into a text, thematic cross-connections and technological USPs are important to make a post effective. It would be tedious and time-consuming to brief the AI on such a level of detail, especially since the right ideas often only come to you when you are writing. Instead, the AI bot should learn over time to deliver the same quality that we receive from our in-house editors. The same applies to creative wordplay, such as headlines, which ideally should also be SEO-optimized. Of course, we could have included the keywords in the prompt, but for our test balloon, we wanted a quick result. And to be honest, the briefings we give our editors and freelancers are often much more vague, cryptic, and less developed than the prompt we used here. Humans still manage to turn a sketch into a painting – something that AI cannot do easily.
There is however one strong criticism regarding the sources: Links provided by Chat GPT-3 led exclusively to 404 error pages. Even when researched with titles and URLs, the online articles that appear to have been used remain untraceable. The AI bot “hallucinated”: it just invented and created sources. Fact-checking fell back on us. The expectation of generating a blog post within a few minutes and publishing it immediately is further than we thought. Nevertheless, our overall conclusion is positive.
Conclusion blog post: quick text framework as a template to enrich and embellish with further knowledge, side-facts and creative details, beware: Check facts.
We recently asked our CTO Thomas Stammeier about Smart Energy Solutions. The interview was published in our blog on January 30, 2023, and therefore does not fall within the data basis from which Chat GPT-3 draws its answers. However, Device Insight had previously published an eight-page Solution Paper on Smart Energy. We were very interested in comparing the responses from the AI bot to the same questions, on the condition that we fed Chat GPT-3 with our Solution Paper and instructed it to solely utilize this data basis. Who gave the better interview?
Prompt to Chat GPT: Answer my questions about Smart Energy Solutions based only on the following text, and do not refer to your training data.
This time Chat GPT-3 took a little longer to start writing. It was probably still “digesting” our PDF. The more often we asked questions about the text, however, the faster answers came. One thing that is immediately noticeable is that a human’s answers are more personal, while the AI bot chooses neutral formulations from the paper. AI also failed to make the transfer from presenting a solution to a value proposition. All interview questions, even specific follow-up and detailed questions were answered in a very uniform, almost repetitive, generalizing, and lengthy manner. Basically, the chatbot does what people in real life like to do when they don’t have an answer ready – a lot of talk about nothing! Concise and fact-based it is not! The question-and-answer principle of an interview did not seem to work with Chat GPT-3. Instead, we would have to precisely specify the expected answer for each question. So, there is no benefit for us.
After the interview, we asked the AI bot again to indicate which sources it relied on for its answers – and received a completely inappropriate statement that completely missed the mark on our topic and input. Chat GPT-3 was probably hallucinating again. Whether the chatbot also disregarded or “creatively” reinterpreted other information remains unclear.
Conclusion Interview: Chat GPT-3 is not suitable for an interview format.
We provided Chat GPT-3 with our aforementioned Smart Energy Solutions Paper and asked for a brief summary to fit on a 1-page document. Additionally, we fed in another 1-page PDF as a model for structure and design. This time, we took the time and effort to ask for each section we wanted to include in the 1-pager separately. In total, we used 6 prompts for this.
Prompts to Chat GPT: Create a German language 1-pager based on the following text. Explain in 500-600 words what Smart Energy Solutions from Device Insight is all about and how companies using the solution can benefit from it. Explain in 500-600 words how Smart Energy Solutions work and highlight the technological features. Extract a short, anonymized customer description from the following text passage “The practical example of a German automobile manufacturer”. Write a few sentences about Device Insight. (Note: Integrated presentation based on 6 prompts).
Within seconds, we received a meaningful and easily understandable text section each time – and this time we had nothing to complain about. For the metrics, the AI bot added further sources with information about the general improvement potential of Smart Energy. Although it had nothing to do with our offering, it was coherent and striking in itself. The design of the 1-pager PDF also hit the spot. Chat GPT-3 solved this task very well.
And now for the downer: In the last subsection “About us,” the AI bot claimed that Device Insight GmbH has been part of the Bosch Group since 2016. Wrong! In fact, we have been part of KUKA since 2019. When the question was entered again, the sentence disappeared from the answer. The fake fact still leaves a sour taste in the mouth though!
Conclusion 1-pager summary: based on 6 partial prompts very well done in text & design.
The overall balance of the 5 tasks we gave to Chat GPT-3, taken from our everyday editorial life as a B2B software company, was positive. First and foremost, the AI text suggestions helped against the fear of the “blank page.” Topics were pre-structured and populated with fluid formulation examples. English texts had an advantage due to a broader data base. This provided non-editors with a proven and efficient tool, which could actually reduce the communications department workload. If you enter the same request multiple times in Chat GPT, you get more variations to work with.
And now the big but: For true added value, any text – be it a social post, mailing or blog post – needs to be refined and enriched. The chatbot is particularly good at summarizing long content. The prerequisite for this – as for any high-quality AI text that can rival the human pen – is a targeted, precise prompt containing all required information about content, style, and impact. If you can master this art, you will benefit from Chat GPT.
However, fact-checking due to incorrect or missing sources remains the responsibility of humans. The obligation to exercise due diligence cannot be taken away by AI. Whether this works better with the latest version Chat GPT-4 and how the B2B marketing formats we have already tested and some more succeed, you will find out in part 2 of our field report.