Custom searches using timestamps in LOGBOOK in org-mode

Question

I'd like to create a custom agenda search which will find TODO items based on time entries in the LOGBOOK. Specifically, I'd like to find items tagged WAITING based on the timestamp which marked the entry to the waiting state. These entries look like this:

:LOGBOOK:
- State "WAITING"     from "TODO"    [2011-11-02 Wed 15:10] \\
  Emailed so-and-so about such-and-such.
:END:

Can I do this with the information in the logbook? I'm using version 7.5 but can upgrade if necessary.

Thanks!

Edit: One use case might be to find WAITING todo's which have been the waiting state for more than a week. (Which usually means I need to bug somebody again.)

Answer 1

The following should do what you need. You'll simply have to adjust the Custom Agenda commands to fit your use-case. (When testing and configuring it I used my TODO keywords). It is possible that one portion of this code duplicates the work of a built-in org function, particularly since it resembles the Scheduled and Deadline approach/overdue behaviour, yet I could not see any specific function that would be reusable.

The actual function to use in the custom command follows.

(defun zin/since-state (since todo-state &optional done all)
  "List Agenda items that are older than SINCE.

TODO-STATE is a regexp for matching to TODO states.  It is provided to
`zin/find-state' to match inactive timestamps.
SINCE is compared to the result of `zin/org-date-diff'.  If
`zin/org-date-diff' is greater than SINCE, the entry is shown in the
Agenda. 
Optional argument DONE allows for done and not-done headlines to be
evaluated.  If DONE is non-nil, match completed tasks.
Optional argument ALL is passed to `zin/find-state' to specify whether
to search for any possible match of STATE, or only in the most recent
log entry."
  (let ((next-headline (save-excursion (or (outline-next-heading) (point-max)))))
    ;; If DONE is non-nil, look for done keywords, if nil look for not-done
    (if (member (org-get-todo-state)
                (if done
                    org-done-keywords
                  org-not-done-keywords))
        (let* ((subtree-end (save-excursion (org-end-of-subtree t)))
               (subtree-valid (save-excursion
                               (forward-line 1)
                               (if (and (< (point) subtree-end)
                                        ;; Find the timestamp to test
                                        (zin/find-state todo-state subtree-end all))
                                   (let ((startpoint (point)))
                                     (forward-word 3)
                                     ;; Convert timestamp into days difference from today
                                     (zin/org-date-diff startpoint (point)))))))
          (if (or (not subtree-valid)
                  (<= subtree-valid since))
              next-headline
            nil))
      (or next-headline (point-max)))))

The following function finds logbook entries through re-search-forward.

(defun zin/find-state (state &optional end all)
  "Used to search through the logbook of subtrees.

Tests to see if the first line of the logbook is a change of todo
status to status STATE
- Status \"STATE\" from ...
The search brings the point to the start of YYYY-MM-DD in inactive timestamps.

Optional argument END defines the point at which to stop searching.
Optional argument ALL when non-nil specifies to look for any occurence
of STATE in the subtree, not just in the most recent entry."
  (let ((drawer (if all "" ":.*:\\W")))
    (re-search-forward (concat drawer ".*State \\\"" state "\\\"\\W+from.*\\[") end t)))

The last function determines the number of days difference between today and the timestamp found by the above function.

(defun zin/org-date-diff (start end &optional compare)
  "Calculate difference between  selected timestamp to current date.

The difference between the dates is calculated in days.
START and END define the region within which the timestamp is found.
Optional argument COMPARE allows for comparison to a specific date rather than to current date."
  (let* ((start-date (if compare compare (calendar-current-date))))
    (- (calendar-absolute-from-gregorian start-date) (org-time-string-to-absolute (buffer-substring-no-properties start end)))
    ))

Two sample custom agenda commands using the above functions. The first matches up to your use-case, you'll simply have to change "PEND" to "WAITING" for it to match the right keyword. The second looks for DONE keywords that were completed more than 30 days ago (As opposed to looking for timestamps that have a month matching this/last month as done in the example I'd linked in my first comment).

(setq org-agenda-custom-commands
      (quote (("T" "Tasks that have been pending more than 7 days." tags "-REFILE/"
               ((org-agenda-overriding-header "Pending tasks")
                (org-agenda-skip-function '(zin/since-state 7 "PEND"))))
              ("A" "Tasks that were completed more than 30 days ago." tags "-REFILE/"
               ((org-agenda-overriding-header "Archivable tasks")
                (org-agenda-skip-function '(zin/since-state 30 "\\\(DONE\\\|CANC\\\)" t))))
              )))

Answer 2

In addition to Jonathan Leech-Pepin answer, if you want to look at CLOSED: drawer added by (setq org-log-done 'time) configuration, you can improve the zin/find-state function like this:

(defun zin/find-state (state &optional end all)
  (let ((drawer (if all "" ":.*:\\W" "CLOSED:")))
    (or (re-search-forward (concat drawer ".*State \\\"" state "\\\"\\W+from.*\\[") end t)
        (re-search-forward (concat drawer ".*\\[") end t))))

PS: This is just a improvement for the answer, the correct answer is the Jonathan's answer.