PDF Annotation to Image Location - Understanding Rotation

[TheCapybaraClub]

First, I am sorry for asking about this topic. Its been covered a lot, but I just don't get it.
#619
#536
...and more I'm sure

The task is to extract an region of interest from a PDF page that has been turned into an image, no matter the rotation.

PyMuPDF 1.19.3: Python bindings for the MuPDF 1.19.0 library.
Version date: 2021-12-12 06:51:56.
Built for Python 3.6 on win32 (64-bit).

rot.pdf
no_rot.pdf

This is straight forward and works as expected

import fitz
import cv2
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import imshow, show 

doc=fitz.open("no_rot.pdf")

# make a version of the PDF without annotations
doc_noannot = fitz.open()
doc_noannot.insert_pdf(doc)

### remove annotation from copied pdf
annots = []
for page in doc_noannot:
    annot = page.firstAnnot
    while annot:
        annots.append(annot)
        annot = page.delete_annot(annot)

pixmap = doc[0].get_pixmap(dpi=300)

pix_bytes = pixmap.tobytes("png")
nparr = np.frombuffer(pix_bytes, np.uint8)
cv2_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

plt.figure(figsize = (20,20))
plt.imshow(cv2_img, 'gray', interpolation='nearest')

mat = doc_noannot[0].rect.torect(pixmap.irect)
img_rect = annots[0].rect * mat
img_rect = img_rect.round()

x,y,w,h = [ img_rect.x0, img_rect.y0, img_rect.x1-img_rect.x0, img_rect.y1-img_rect.y0] #x,y,w,h


ROI = cv2_img[y:y+h, x:x+w]

plt.figure(figsize = (5,5))
plt.imshow(ROI, 'gray', interpolation='nearest')

Create Rotate -90 Version

doc[0].set_rotation(-90) # rotate page by 90 degrees counter-clockwise
doc.ez_save("rot.pdf")

Do it all Again, but try to respect the rotation matrixes

doc=fitz.open("rot.pdf")

# make a version of the PDF without annotations
doc_noannot = fitz.open()
doc_noannot.insert_pdf(doc)

# remove annotation from copied pdf
annots = []
for page in doc_noannot:
    annot = page.firstAnnot
    while annot:
        annots.append(annot)
        annot = page.delete_annot(annot)


pixmap = doc[0].get_pixmap(dpi=300)

pix_bytes = pixmap.tobytes("png")
nparr = np.frombuffer(pix_bytes, np.uint8)
cv2_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

plt.figure(figsize = (20,20))
plt.imshow(cv2_img, 'gray', interpolation='nearest')

Please help here

mat = doc_noannot[0].rect.torect(pixmap.irect)

# ??? Nope
#img_rect = annots[0].rect * mat

# ??? Nope
#img_rect = annots[0].rect * mat * doc[0].rotation_matrix

# ??? Nope
img_rect = annots[0].rect * mat * doc[0].derotationMatrix

img_rect = img_rect.round()

x,y,w,h = [ img_rect.x0, img_rect.y0, img_rect.x1-img_rect.x0, img_rect.y1-img_rect.y0] #x,y,w,h
x,y,w,h

ROI = cv2_img[y:y+h, x:x+w]

plt.figure(figsize = (5,5))
plt.imshow(ROI, 'gray', interpolation='nearest')

[TheCapybaraClub]

Still looking for help on this. Not sure if this comment will help bring this to anyone's attention.

[JorjMcKie]

1. Pixmaps are always made from the page rectangle as is, i.e. with any rotation.
2. You can choose to exclude annotations when making the pixmap. No need to make a separate, annotation-free PDF page for this purpose. Use annots=True/False.
3. Annotation rectangles however are computed for the unrotated page. So to compute their rotated coordinates, multiply them with the rotation matrix.
4. To compute pixmap coordinates of anything for a rotated page, you therefore must do this:
   • mat = page.rect.torect(pix.irect) - both rectangles pertain to rotated coordinates
   • so the annotation coordinates on the pixmap should be computed like this: annot.rect * page.transformation_matrix * mat.

[TheCapybaraClub]

Thank you so much for taking the time to write a response! I am unable to get expected results from following your directions strictly, but am able to get expected results by applying the concept in a different way. Example 1 I approach directly as you have instructed, in Example 2 I set the page rotation to 0 to match the expected annotation rotation. Why doesn't Example 1 work?

Imports

import fitz
import cv2
import numpy as np

import matplotlib.pyplot as plt
from matplotlib.pyplot import imshow, show 

Example 1

Doesn't Work: img_rect = annot.rect * page.transformation_matrix * mat

doc=fitz.open("rot.pdf")

page = doc[0]
annot = page.firstAnnot
print(annot.rect) #this rect is for the unrotated page

pixmap = page.get_pixmap(dpi=300, annots=False) #from #2 above, do not duplicate the pdf, annots=False
mat = page.rect.torect(pixmap.irect)

img_rect = annot.rect * page.transformation_matrix * mat #from #4 above
img_rect = img_rect.round()

pix_bytes = pixmap.tobytes("png")
nparr = np.frombuffer(pix_bytes, np.uint8)
cv2_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

x,y,w,h = [ img_rect.x0, img_rect.y0, img_rect.x1-img_rect.x0, img_rect.y1-img_rect.y0] #x,y,w,h
print(x,y,w,h)
ROI = cv2_img[y:y+h, x:x+w]

plt.figure(figsize = (5,5))
plt.imshow(ROI, 'gray', interpolation='nearest')

Example 2

Works: page.set_rotation(0) & img_rect = annot.rect * mat

doc=fitz.open("rot.pdf")

page = doc[0]
annot = page.firstAnnot
print(annot.rect) #this rect is for the unrotated page

page.set_rotation(0) # not from #4 above, page rotated to 0
pixmap = page.get_pixmap(dpi=300, annots=False) #from #2 above, do not duplicate the pdf, annots=False
mat = page.rect.torect(pixmap.irect)

img_rect = annot.rect * mat # not from #4 above, page already rotated to 0
img_rect = img_rect.round()

pix_bytes = pixmap.tobytes("png")
nparr = np.frombuffer(pix_bytes, np.uint8)
cv2_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

x,y,w,h = [ img_rect.x0, img_rect.y0, img_rect.x1-img_rect.x0, img_rect.y1-img_rect.y0] #x,y,w,h
print(x,y,w,h)
ROI = cv2_img[y:y+h, x:x+w]

plt.figure(figsize = (5,5))
plt.imshow(ROI, 'gray', interpolation='nearest')

[JorjMcKie]

sorry, my bad:
Instead of annot.rect * page.transformation_matrix * mat I meant annot.rect * page.rotation_matrix * mat.

[TheCapybaraClub]

Not trying to tag along on this topic, but for a more full understanding, is it valid that this formula can return negative values for the rectangle? I am sorry I cannot share the actual PDF for this example.

my rect is:

Rect(1978.8399658203125, 2198.2529296875, 1992.9300537109375, 2212.866943359375)

my rotation_matrix is:

Matrix(0.0, 1.0, -1.0, 0.0, 3168.0, 0.0)

my matrix is:

Matrix(4.166666507720947, 0.0, 0.0, 4.166666507720947, 0.0, 0.0)

the formula and result

= annot.rect * page.rotation_matrix * mat
Rect(-6052.2783203125, 8245.166015625, -5991.38671875, 8303.875)

how should these negative values be understood to be used to crop a pixel map?

[JorjMcKie]

but of course, all is also fine if you (temporarily) nullify page rotation ...

[JorjMcKie]

is it valid that this formula can return negative values for the rectangle?

Yes it is. The rectangle page.rect equals the page's /CropBox - potentially width and height exchanged when rotated, and top-left is always (0,0).
Other than that items (text, images, drawings, ...) might have been inserted not being fully inserted inside that area. Or of course, the cropbox might have been shrinked later. Completely legal.
Then you will get negative coordinates or such beyond the width / height limits.