Summary of Documents

My experiences with summarizing documents automatically

Over the last few months I’ve been working on automatically summarizing texts (novels/essays) for inclusion in Panofind. The results were mixed, sometimes the summaries were just a list of events without providing any real plot (or thread).
For the summaries, I used ChatGPT’s API, which is paid for by the number of tokens (about 1.5 tokens per word) and must be processed online (i.e., the text is sent to a server and the intermediate and final responses are returned). However, as long as it is not actively set in the account settings, none of the input or output should be used for a new version of ChatGPT (so it should not appear as an answer for someone else).
The question I have is whether it’s worth adding this feature to Panofind right now, or if the technology isn’t mature enough yet - or just too costly, too?

I had ChatGPT create the following summary from the original Romeo and Juliet, which is about 26,000 words. When converted, the summary could have been translated directly into another language, but to be comparable, the target language is English.
The following summary costs about 0.17 USD:

The novel begins with a street brawl between the servants of the Montagues and the Capulets, showcasing the tension between the two families. Benvolio, a nephew of Montague, tries to maintain peace, but Tybalt, a nephew of Lady Capulet, challenges him to a duel. The Prince of Verona arrives and warns the families of severe consequences if they continue to fight.

In Act II, Romeo, Mercutio, and Benvolio discuss Romeo’s unrequited love for Rosaline. They decide to attend the Capulet’s feast, where Romeo may find another love interest. At the feast, Romeo and Juliet meet and instantly fall in love. They express their desire to be together and make plans to get married.

In Act III, Tybalt confronts Mercutio and Benvolio, and Romeo tries to defuse the situation. However, Tybalt challenges Romeo to a duel. Romeo refuses to fight, as he has just married Juliet. Mercutio fights on Romeo’s behalf and is fatally wounded. Enraged, Romeo seeks revenge and kills Tybalt. The Prince banishes Romeo from Verona.

In Act IV, Juliet seeks Friar Lawrence’s help to avoid marrying Paris. The Friar devises a plan for Juliet to fake her own death using a potion. Juliet agrees and takes the potion.

In Act V, Romeo receives news of Juliet’s death and decides to return to Verona to be with her in death. However, Friar Lawrence’s letter to Romeo fails to reach him, and he is unaware of Juliet’s plan. Romeo arrives at the Capulet’s tomb and encounters Paris. They fight, and Romeo kills Paris. Romeo drinks poison and dies next to Juliet’s lifeless body. Juliet wakes up, finds Romeo dead, and stabs herself with his dagger. The tragic deaths of Romeo and Juliet bring an end to the feud between the Montagues and the Capulets.

Throughout the novel, Romeo transforms from a lovesick young man to a passionate individual willing to risk everything for love. Juliet starts as a sheltered daughter but gains the courage to defy her family for the sake of her love for Romeo. They both become strong and independent individuals, willing to die for their love.

The central themes of the novel include the power of love, the consequences of feuds and violence, and the role of fate. Romeo and Juliet’s love is portrayed as intense and all-consuming, leading them to make drastic decisions. The feud between the Montagues and the Capulets is shown to have tragic consequences, resulting in the deaths of innocent individuals. The characters’ actions are also influenced by fate, as their love and the events that unfold seem predetermined.

The novel explores the intricacies of the plot through the interconnection of various plotlines. The initial street brawl sets the stage for the ongoing conflict between the Montagues and the Capulets. Romeo and Juliet’s love story develops amidst this feud, leading to their secret marriage. The conflict escalates when Tybalt challenges Romeo to a duel, resulting in Mercutio’s death and Romeo’s banishment. Juliet’s desperate attempt to avoid marrying Paris leads to her faked death, which ultimately leads to the tragic deaths of Romeo and Juliet.

Overall, the novel Romeo and Juliet delves into the complexities of love, feuds, and fate. The characters undergo significant growth and development, and their actions have far-reaching consequences. The tragic ending serves as a cautionary tale about the destructive power of hatred and the importance of love and understanding.

Even though it is not as complete and good as a summary created by a human, (I think) it doesn’t contain such logical errors as the one provided as demo for summarizing books via ChatGPT:

Capulet invites many guests to his feast, including Romeo and his love, Juliet. […] Romeo sees Juliet and falls in love with her.

or

Romeo buys poison to kill Juliet at her grave.

In my experience so far, the quality of the result directly correlates to the length of the summary. The shorter the summary is or should be, the decision of what is really important gets more and more difficult (and can only performed by a human, if shortness is a must).
Additionally, summaries in English seems to be superior to the ones in other languages (e.g. German).