Similar issue here. Sometimes Leo just doesn’t seem to recognize that something is on the page, whereas a letter in the same hand, ink, weight, paper is recognized and transcribed. Asking to retranscribe is no solution in these cases
I put through a large number of documents today and came across quite a few failed transcriptions.
They were all documents from the Court of Exchequer, in English, from the late seventeenth and eighteenth centuries. The documents are all very large and often damaged (holes from mice, ink blots, rips and tears, smudging, water damage). I think photo quality and lighting may be an issue sometimes, but sometimes Leo is remarkable at deciphering something that I could not read due to photo quality. However, sometimes a very clear photo produces hallucinations and the ‘doom loop’ of repetitive text.
The writing on these documents is often squeezed in and very tight on the page, which I think could also affect the way Leo works.
Leo often works better with the bills and answers, but less well with depositional material. I think this is because depositions feature indented names and marginalia numbers, which Leo struggles to place on a page - in the transcript of the example photo Leo transcribed the main text, but often missed out the deponent’s information (name, age etc) which was inset on the page.
This may not be correct, but I also think the repetitive nature of court documents can confuse Leo. Words and phrases are frequently repeated in these documents and I think this causes Leo to skip lines and paragraphs sometimes, possibly confusing where lines start and end.
Obviously as it transcribes more of these documents it will improve, but I thought I would just raise the fact that it has been patchy for these kinds of documents.
1 Like
One thing I’ve noticed is that sometimes when there is a signature then Leo will start and stop with that - here is an example:
1 Like
I’ve noticed this too, often if there is a signature (which luckily they are mostly at the end of a page/letter) it will omit them and then stop transcribing. Perhaps there is a way if a signature is illegible to have LEO insert <signature_illegible> or something along those lines so the person receiving the transcription can understand the letter didn’t end with simply silence, and to go back and see who’s name it was if it truly was illegible. Oftentimes LEO will get signatures (which are very messily written) slightly wrong but still be able to transcribe them so I am unclear why sometimes it tries to decipher them and other times ignores them completley.
1 Like
I’m not sure why Leo is ignoring signatures. It’s possible it’s an issue in the training data that we need to go back and fix, so thanks for the heads up!
I see the logic on [illegible] signs though we decided not to use them as we didn’t want to kneecap the model. Our goal is that Leo will eventually provide not just useable or passable but superhuman transcriptions. What we do plan to add is some kind of confidence metric visible within the web-app, so users can see what’s likely to need correction.
Not sure this qualifies as a “transcription failure,” but many of the documents I have include pages where there’s some scribbling on it. In this case, it grabbed on one number (looks like $296.86 to my eyes), read it as “27” and then just added ten pages of “27” over and over again to the transcription.
Happily, here, this was the last page of the scan and looks like nothing important on the page.
Ah yes, this is a “hallucination”! See here:
Jon, I came across this same problem, with a series of strangely rotated images that resulted in similar repetitive loops. Unfortunately, that means that most of my 10 credits are wasted. Is there a way to get those credits back, so I can crop the images better and see if reducing the image to a single column of text works better? I’d love to play more with this before my official Beta Test starts up later this month. Thank you!
Yes, of course. I’ve replenished your account with some additional lifetime credits to test things out before the beta round officially begins. 
1 Like
A post was split to a new topic: PDF image extraction difficulties
Chiming in on transcription issues. In addition to struggling with images that aren’t oriented correctly, I’ve noticed occasional trouble with layered documents of the kind that appear often in bound volumes (handwriting next to printed text; pages of different sizes, so that a single image captures part of an additional document, etc). Here’s an example where transcription fell into the doom loop.
1 Like
I’ve had similar issues with transcriptions falling into a “doom loop”. This happens when some of the text on a page is upside down (for example, when someone reuses a printed page). Curiously, this also seems to be a problem for a set of pages that have a stamp in the top left-hand corner. It will transcribe the text in the stamp and then start repeating a random set of words. In one instance, I tried to re-upload a page and got a doom loop again but with different text.
1 Like