<div dir="ltr"><div><div><div>Hi Kent.<br><br></div>I'll respond to a few points. The rest are outside my knowledge / scope (EX: mwxml2sql; Debian).<br><br>Hope this helps, and good luck on wp-mirror.<br><br>----<br></div>
<div><br>> interlanguage links have been removed to the wikidata project, the rendering of which requires mediawiki-1.21+;<br></div>You will need a local instance of <a href="http://www.wikidata.org">http://www.wikidata.org</a> . You could probably build it using the files from here: <a href="http://dumps.wikimedia.org/wikidatawiki/">http://dumps.wikimedia.org/wikidatawiki/</a><br>
</div><br>Each wiki would also need the wikibase extension: <a href="https://www.mediawiki.org/wiki/Extension:Wikibase">https://www.mediawiki.org/wiki/Extension:Wikibase</a>. <br><br>Note that wikidata is also being used in infoboxes. For example, <a href="https://simple.wikipedia.org/w/index.php?title=Google&action=edit">https://simple.wikipedia.org/w/index.php?title=Google&action=edit</a> now has multiple {{#property}} statements.<br>
<br><div><div>> infoboxes now require the scribunto extension which requires mediawiki-1.20+<br></div><div>I believe most of the wikis have moved the infobox generation from Templates to Modules. They've moved a lot of other functionality as well (for example: references and message boxes).<br>
<br>The Scribunto extension is at <a href="https://www.mediawiki.org/wiki/Extension:Scribunto">https://www.mediawiki.org/wiki/Extension:Scribunto</a> . It transforms a {{#invoke:Module_name|function_name|arguments}} into the appropriate text.<br>
<br></div><div>> category - dump files now have 5 fields, whereas the database schema has 6 fields;<br><br></div><div>I believe they removed the cat_hidden field which was effectively deprecated. A category's hidden status is saved in page_props<br>
<br></div><div>> The large image dump tarballs are now a year old. <br><br></div><div>I raised this issue back in July. See here for Kevin Day's response (from <a href="http://your.org">your.org</a>): <a href="http://lists.wikimedia.org/pipermail/xmldatadumps-l/2013-July/000861.html">http://lists.wikimedia.org/pipermail/xmldatadumps-l/2013-July/000861.html</a><br>
</div><div>I think there's still some infrastructure work that needs to be done on the Wikimedia side.<br><br>> We are beginning to see thumb dumps from the xowa project.<br><br>I've been uploading thumbs for the major wikis to <a href="http://archive.org">archive.org</a>. See here for a summary: <a href="https://archive.org/search.php?query=xowa">https://archive.org/search.php?query=xowa</a><br>
<br></div><div>I'm planning to upload all thumbs for all the major languages that are listed as > 200,000 on <a href="https://en.wikipedia.org/wiki/Main_Page">https://en.wikipedia.org/wiki/Main_Page</a> . There are roughly 27 languages listed there. My progress has been about 1 wiki per week (I'm also uploading sister wikis for a given language). I've done 4 so far. I'm hoping to be done with the other 23 sometime by mid-year.<br>
<br>At about that time, I'm hoping to have a more automated way of generating updates. Currently, I'm only releasing monthly updates for <a href="http://en.wikipedia.org">en.wikipedia.org</a> and quarterly updates for the other main wikis.<br>
</div><div><br></div><div>Note that the thumbs are uploaded as sqlite databases. I chose sqlite b/c tarballs are slower to extract / update / query. The database schema is fairly basic, and you should be able to retrieve any file with a sqlite library.<br>
</div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 10, 2014 at 2:43 AM, wp mirror <span dir="ltr"><<a href="mailto:wpmirrordev@gmail.com" target="_blank">wpmirrordev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear Ariel,<div><br></div><div>Happy New Year. I am gearing up for wp-mirror-0.7. To that end, I would like to list some issues that I see; and I would like to offer my help in solving them.</div>
<div><br>
</div><div>0) Problem Statements</div><div><br></div><div>0.1) Page Rendering. Wp-mirror-0.6 works well in the sense that it builds a faithful mirror of any of your wikis. However, during 2013 the rendering of pages eroded materially. For example,</div>
<div><br></div><div> o interlanguage links have vanished both from rendered pages and from dump files;</div><div> o infoboxes are no longer rendered;</div><div> o most transclusions now render as redlinks even though the templates are easily found in the underlying database; etc.<br>
</div><div><br></div><div>I understand that this erosion occurred because wp-mirror-0.6 still uses mediawiki-1.19, whereas WMF has moved on to mediawiki-1.23. For example, I understand that:</div><div><br></div><div> o interlanguage links have been removed to the wikidata project, the rendering of which requires mediawiki-1.21+;</div>
<div> o infoboxes now require the scribunto extension which requires mediawiki-1.20+</div><div><br></div><div>0.2) Database Schema. Some differences in database schema have appeared.<br></div><div><br></div><div> o category - dump files now have 5 fields, whereas the database schema has 6 fields;</div>
<div> o exterallinks - dump files now have 4 fields, whereas the database schema has 3 fields.</div><div><br></div><div>Loading these two tables generate the error message: ``Column count doesn't match value at row 1.''</div>
<div><div><br></div><div>0.3) Version Lifecycle. According to <<a href="http://www.mediawiki.org/wiki/Version_lifecycle" target="_blank">http://www.mediawiki.org/wiki/Version_lifecycle</a>> mediawiki 1.23 LTS is slated for May 2014. However, the Debian packaging team is silent as to their plans for a transition from mediawiki-1.19 LTS to mediawiki-1.23 LTS.</div>
<div><br></div></div><div>0.4) Image Dumps. The large image dump tarballs are now a year old. This means that, while wp-mirror still downloads the bulk of its images from these tarballs, there are a growing number that must be downloaded individually from WMF.</div>
<div><br></div><div>0.5) Thumbs. One person has asked me if dump files of thumbs could be made available. We are beginning to see thumb dumps from the xowa project.</div><div><br></div><div>0.6) IPv6. I am glad to see that <<a href="http://gerrit.wikimedia.org" target="_blank">gerrit.wikimedia.org</a>> has an IPv6 address. However, <<a href="http://bastion.wmflabs.org" target="_blank">bastion.wmflabs.org</a>> still does not. My internal network is IPv6 only.</div>
<div><br></div><div>1) mwxml2sql</div><div><br></div><div>This utility from Ariel Glenn has proved invaluable to the wp-mirror project. This utility, together with MySQL 5.5 fast index creation, allows wp-mirror to build mirrors much faster than before (80% less time). </div>
<div><br></div><div>1.1) Need for update. According to its version information, mwxml2sql may only be valid through mediawiki-1.21.</div><div><br></div><div><div>(shell)$ mwxml2sql --version</div><div>mwxml2sql 0.0.2</div>
<div>Supported input schema versions: 0.4 through 0.8.</div><div>Supported output MediaWiki versions: 1.5 through 1.21.</div></div><div><br></div><div>Whereas, I am looking forward to mediawiki-1.23 LTS (see below), I would like to know if mwxml2sql should be updated.<br>
</div><div><br></div><div>1.2) Help Offer. If mwxml2sql does need updating, I would be happy to help with this; and to package it for Debian as I have done before. Perhaps we could call it mwxml2sql-0.0.3.</div><div><br>
</div><div>2) mediawiki-1.23 LTS. </div><div><br></div><div>2.1) Vision. I would like wp-mirror-0.7 to be able to build a mirror that serves pages that look no different than those served by WMF.</div><div><br></div><div>
2.2) DEB package. To that end, I am thinking of packaging mediawiki-1.23 together with the extensions needed for rendering WMF wikis with wikidata content, infoboxes, math, transclusions, etc. Given WMF's ``continuous integration'' development model, I would like to be able to automatically generate a tarball and DEB package each time WMF pushes an update to its servers.</div>
<div><br></div><div>2.3) Debian package repository. Such a DEB package would be distributed with wp-mirror. In preparation for this, I have set up a Debian package repository at <<a href="http://download.savannah.gnu.org/releases/wp-mirror/" target="_blank">http://download.savannah.gnu.org/releases/wp-mirror/</a>>. It is currently used to distribute wp-mirror-0.6 and an unstable version of wp-mirror-0.7. Home page <<a href="http://www.nongnu.org/wp-mirror/" target="_blank">http://www.nongnu.org/wp-mirror/</a>>.<br>
</div><div><br></div><div>2.4) Help Offer. I am happy to do most of this work myself. However, I will need some guidance on interacting with the appropriate GIT repositories. I hope that you can put me in touch with someone involved in the ``continuous integration'' process.</div>
<div><br></div><div>3) Media dumps</div><div><br></div><div>I am thinking that updating the image dumps annually would be adequate. Including thumbs in those dumps would materially assist the off-line community. I could easily update wp-mirror-0.7 to give the user a choice (no media files, thumbs only, full size media files).</div>
<div><br></div><div>Sincerely Yours,</div><div>Kent</div><div><br></div></div>
<br>_______________________________________________<br>
Xmldatadumps-l mailing list<br>
<a href="mailto:Xmldatadumps-l@lists.wikimedia.org">Xmldatadumps-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l</a><br>
<br></blockquote></div><br></div>