I tested NotebookLM and saw the promise of something great

AI has the potential to fully upend the way we work. More than any emerging technology before it, like AR/VR and the blockchain, generative language models like Google Bard have already become vital parts of many people’s work lives throughout dozens of industries. Google wants a piece of that cake, and it’s aiming at classrooms and researchers with its NotebookLM experiment.

NotebookLM is a tool that lets you upload documents and text to a dedicated space on Google’s servers and use a language model to ask questions, summarize, and take action on sources that may otherwise take hours to dig through. NotebookLM wants to be a virtual research assistant that’s supposed to help you get out first drafts faster than you otherwise would. Google claims that it seamlessly lets you transition between note-taking and working with sources, all while having the AI instantly ready to ask any questions about the text you may have.

I’m Google Editor here at Android Police, but I also work as a script reader on the side, producing evaluations of incoming materials for a TV channel. With a job so heavily focused on creating a knowledge base on individual scripts before I even start writing, I was wondering how much I could benefit from the help of AI to speed up certain processes.

Before I dive in, note that yet-to-be filmed scripts are a delicate matter that involve a lot of secrecy, so I didn’t feed NotebookLM with any of the actual scripts I’m evaluating. Google does say that “Your personal data is not used to train NotebookLM, so any private or sensitive information you have in your sources will stay private, unless you choose to share your notebooks with collaborators.”

In theory, this means it should be fine to feed NotebookLM any sensitive data. Given that the material I work with is the livelihood of other people, I didn’t want to take any chances, though. I figured I’d experiment with some existing films and opted to treat Christopher Nolan’s 2005 hit Batman Begins as a sample script instead.

What’s good about NotebookLM?

Getting started with NotebookLM is easy. Once you sign in with your Google account, you’re basically ready to go. You’re greeted by an option to add a new notebook as well as a selection of samples to give you an idea of what you can do with the vast empty canvas — a big problem with many chat-based large language models. If you want to dive right into your own projects, Google explains how you can add PDF files, Google Docs, or copied text, and that you can ask the built-in AI assistant questions about the added materials. The interface isn’t the most polished or intuitive, but it gets the job done for an experiment. It even works well enough on mobile browsers with a responsive interface, though it’s clear that it was designed with desktops in mind first.

To put the tool to the test, I uploaded a PDF of the Batman Begins script to NotebookLM, along with PDF files of Christopher Nolan and David S. Goyer (the author) for some peripheral information about the artists behind the project. This is usually the key information that I work with as a script reader, in addition to the guidelines and requirements for my TV channel’s programming (Fun fact: Batman Begins more than likely wouldn’t have made the cut due to it simply not meeting genre, crew, and budget requirements).

With this data imported, I quickly began to ask NotebookLM to do its magic. Rather than digging through the script and building an overview of the characters myself, I could just ask the AI for a quick bullet list of the most important characters. The AI could also get me a rough draft of the plot structure and summary, making it easier to start the writing process than it usually would be. I could also ask it for a quick summary of Christopher Nolan’s experience, with previous works being an important consideration for the evaluation.

While chatting, you can quickly toggle on or off individual sources with a checkbox on their previews. It’s not the most intuitive interface, but once you wrap your head around it, you can quickly guide the AI to only use certain bits of information, digging down into details much more quickly than otherwise possible.

Overall, NotebookLM offers a quick way to extract key information from the script without having to try to look everything up myself in the script. However, NotebookLM is still only good when you know the text you’re working with beforehand. It’s been a while since I’ve watched Batman Begins. The general vibe of the summary and the characters seems to be correct, but I still just can’t find myself fully trusting AI to get it right. NotebookLM could be a great tool to get certain tasks done more quickly, but I’d still have to read and work with the source myself. That leads us right to what I don’t like about NotebookLM, and boy, Google needs to work on the tool a lot to get it in a state where I could work with it.

What’s bad about NotebookLM?

Before I even dive into trust issues in AI, I first need to talk about the general workflow. Right off the bat, I noticed that imported PDF files are converted into text-only files. A PDF like a script that relies on indentation and other intricate formatting to differentiate between dialog, instructions, scenes, and multiple pages is not good to work with that way, to put it mildly.

Another first impression is that the AI doesn’t always work as expected, either. It readily provided automated summaries for the two Wikipedia articles about Christopher Nolan and David S. Goyer I uploaded, but just said “The system was unable to generate a guide for this source” for the script, where it would have been most useful.

This notion of a product that isn’t quite ready yet continues throughout the whole interface. The interface takes a lot of getting used to on smaller screen sizes, with a bottom-based UI that looks different from all other Google apps, letting you switch between the sources and the notes. Only when you view NotebookLM on a bigger width do you get a two-column view with the sources and the notes next to each other.

Then, you can’t reorder notes and sources, which is one of the most basic features a digital notebook should have. It’s an even bigger oversight when you consider that Google’s own Keep app has always been capable of doing that and that NotebookLM closely resembles a top-down view of a desk with sticky notes sprinkled on the right side of your sources.

You also can’t edit automatically generated notes, which are added by “pinning” a chat answer to a note (a weird workflow in itself) — to do that, you’d have to manually copy the contents to a written note. I suspect that Google did this in an effort to make it intuitively obvious which notes are your own thoughts and which are AI-generated, but for my ideal workflow, I would have to blend some of these together. For example, I would like to add further notes to the character overview, which I can’t do with the current restrictions.

For some of the questions, the answers were quite useless after a little bit of digging. For example, when I asked the AI to list the most important characters, I asked it to include the age and when a character was first mentioned. For all of the characters, the AI just said “first mentioned in the sources,” which is anything but helpful when you expect a scene or a page number. It also got some information wrong, like saying that Rachel Dawes doesn’t have an age specified when it was mentioned that she was 10 years old in one of the flashbacks. The same is true for Alfred, who is said to be 50 years old in another flashback.

Then there were also some questions that NotebookLM refused to answer unless I slightly reworded them. When I asked it to “Give me an overview of the key characters, the conflicts, and the quality of the dialog in two paragraphs,” it said that it couldn’t answer the question and I should try to rephrase or ask something else. A slight tweak to the question, essentially just removing the “quality” in the prompt, gave me a full-fledged answer (my rephrased question was: “Give me an overview of the key characters, the conflicts, and the the (sic!) dialog in two paragraphs.”)

Some advertised features are also still missing. In generated notes, you can hover over footnotes to reveal where the AI supposedly pulled the information from. In the NotebookLM preview, clicking this footnote highlights the spot in the source it’s from. This works when you look the source up right as you chat with the AI, but it doesn’t when you want to check back where some info came from in a saved note. The same is true for draft generation. When you highlight six or more notes, a button is supposed to appear that lets you create an outline or draft you can send straight to Google Docs for further work. This key feature is missing altogether for me.

Another issue with NotebookLM is that I don’t trust the AI to get all the facts right. I’ve noted above that it got some key details wrong right at first glance, like claiming that some characters didn’t have their age specified even though that was the case. If I have to go back into the text myself to check everything the AI tells me, I could just fully work with the text myself in the first place. This problem exists for all LLMs out there right now, but I’d hoped that this would be less of an issue when you limit the LLM’s sources to just one or two texts.

Is NotebookLM worth it today?

It’s certainly fun to use NotebookLM, exploring its capabilities and limitations. I wouldn’t say that the tool is ready for prime time just yet, though, even if the company opened it up to everyone in the US in December. It has a lot of potential, though, and might just upend how we work with sources and documents in the future. Right now, the interface limitations and the AI’s hallucinations just get in the way too much, but with the right tweaks, it could get there sooner than we might think. That’s particularly true once (or rather, if) we get a NotebookLM version powered by Gemini Ultra, Google’s most advanced AI.

At the same time, I wouldn’t expect Google to stick with NotebookLM in its current form. The tool is marketed as an experiment, and it’s clear that it’s more of a playground than anything. Given Google’s track record as a company that kills products on a whim, you definitely shouldn’t use it for any longer-term projects like your research papers, where a tool like this could arguably shine. With this in mind, I’m left both impressed and disillusioned — NotebookLM shows us where the future of AI could lead us, but right now, it’s just a promise of something great more than a viable tool that can help you with your work.