[Calendarserver-maintainers] Bug#489188: Bug#489188: Events more than 356 days from creation are ignored

[Peter Mogensen]

Guido Günther wrote:
>> Searches seems to not use the index when there's a time-range element in
>> the filter, but the exception detecting this is only thrown from the
>> filter expression generator for prop-filters, ... which again means that
>> if you don't have a prop-filter in your query, you'll get a search using
>> the index, which will ignore these 356+ days events.
> Could you work on a fix? It seems you've got most of the details
> together already. I attached your analysis to Trac ticket #287.

I don't have any "perfect" fix. That would be a more complex patch and 
would have to take into account upcoming changes for 1.3.

Here's some more analysis and suggestions for work-arounds, though:

According to RFC4791 a simple calendar-query REPORT would look like this:
================
<calendar-query xmlns:D="DAV:" xmlns="urn:ietf:params:xml:ns:caldav">
   <D:prop>
     <calendar-data/>
   </D:prop>
   <filter>
     <comp-filter name="VCALENDAR">
       <comp-filter name="VEVENT">
         <time-range start="20070420T120039Z" end="20070620T120039Z"/>
       </comp-filter>
     </comp-filter>
   </filter>
</calendar-query>
=================

The "time-range" element is almost in all use cases part of a 
comp-filter. (It could also be used in a prop-filter.)

When an event is inserted into the database its instances are calculated 
  up to 356 days into the future. Any such instances are put into the 
TIMESPAN table of the index. If this is not a complete expansion 
RECURRANCE_MAX is set in the RESOURCE table to indicate how far we have 
indexed this event.
As wsanchez says in Apples Trac ticket 207:

"The index is supposed to indicate how far out expansion has happened 
for each event, and if a query against the index goes beyond that time, 
the expansion should be brought out farther, up to a defined limit.

If we have expanded to the limit, then any queries that goes beyond the 
limit would have to read the icalendar data to see if that event matches 
(far more expensive, but the idea is that such queries should be rare)."

However... RECURRANCE_MAX is never used in the DCS 1.2 code as a quick 
grep of the source will show!!

So no where is the expansion ever brought out farther for events beyond 
356 days from creation!
This means that searches using the index will never see such events - 
even though they are stored correctly in the database.

Searches are using the index if index_query_ok becomes true in
twistedcaldav/method/report_calquery.py which it does when 
index.searchValid is true which in turn is the case when 
query.calendarquery.sqlcalendarquery does not encounter a ValueError.

Now, what to do about it?

In the DCS v/1.2 source in twistedcaldav/query/calendarquery.py line 159 
a ValueError exception is raised since we don't handle time-range within 
prop-filter. Obviously (as described above) we don't correctly handle 
time-range within comp-filter, but the equivalent exception is not 
raised in line 102-105.
On fix would be to completely disable use of the index by just raising a 
ValueError in calendarquery.py by replacing line 104, 105 with
"raise ValueError".
This disables the whole TIMESPAN index.

Another not as brutal, but equally simple fix would be to take 
RECURRANCE_MAX into account, but modifying the SQL qualifier which 
decides which events to include in the search result which is defined in 
twistedcaldav/query/sqlgenerator.py line 49:

Changing:
TIMESPANTEST  = "((TIMESPAN.FLOAT == 'N' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s) OR (TIMESPAN.FLOAT == 'Y' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s)) AND TIMESPAN.NAME == RESOURCE.NAME"

To:
TIMESPANTEST  = "((((TIMESPAN.FLOAT == 'N' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s) OR (TIMESPAN.FLOAT == 'Y' AND TIMESPAN.START < %s AND 
TIMESPAN.END > %s)) AND TIMESPAN.NAME == RESOURCE.NAME) OR 
(RESOURCE.RECURRANCE_MAX < %s))"

This will use the index for actually indexed events but also include 
every event beyond the index expansion date.
This will return "wrong" results and thus violate the CalDAV RFC section 
7.8 which says:

       "The response body for a successful CALDAV:calendar-query REPORT
       request MUST contain a DAV:response element for each iCalendar
       object that matched the search filter."

... but that would be ok IFF the only clients accessing the server do 
their own recurrence expansion and filtering.

>> The solution seems to be in the roadmap for DCS 1.3, however I would
>> regard this as a major flaw for 1.2 users.
> It certainly is.

Unfortunately, Apple have bumped the milestone for this bug to 2.x.
I don't understand that. This is a serious blocker. The calendar can not 
handle events like "same time next year" at all - which I would assume 
is often used.

/Peter