[DRE-commits] [coderay] 01/03: Imported Upstream version 1.1.0
Youhei SASAKI
uwabami-guest at alioth.debian.org
Wed Sep 4 09:55:06 UTC 2013
This is an automated email from the git hooks/post-receive script.
uwabami-guest pushed a commit to branch master
in repository coderay.
commit 88155c0cc90875d194a47531d253f642807aac39
Author: Youhei SASAKI <uwabami at gfd-dennou.org>
Date: Wed Sep 4 18:47:20 2013 +0900
Imported Upstream version 1.1.0
---
LICENSE | 504 -----------------------------
Rakefile | 4 +-
bin/coderay | 8 +-
checksums.yaml.gz | Bin 0 -> 270 bytes
lib/coderay.rb | 15 +-
lib/coderay/encoders/debug.rb | 22 +-
lib/coderay/encoders/debug_lint.rb | 63 ++++
lib/coderay/encoders/html.rb | 185 +++++------
lib/coderay/encoders/html/css.rb | 14 +-
lib/coderay/encoders/html/numbering.rb | 45 +--
lib/coderay/encoders/html/output.rb | 2 +-
lib/coderay/encoders/lint.rb | 59 ++++
lib/coderay/encoders/statistic.rb | 1 -
lib/coderay/encoders/terminal.rb | 226 +++++++------
lib/coderay/helpers/file_type.rb | 106 +++---
lib/coderay/helpers/gzip.rb | 41 ---
lib/coderay/helpers/plugin.rb | 40 +--
lib/coderay/helpers/word_list.rb | 5 -
lib/coderay/scanner.rb | 84 +++--
lib/coderay/scanners/c.rb | 2 +-
lib/coderay/scanners/cpp.rb | 2 +-
lib/coderay/scanners/css.rb | 47 ++-
lib/coderay/scanners/debug.rb | 26 +-
lib/coderay/scanners/diff.rb | 84 +++--
lib/coderay/scanners/{c.rb => go.rb} | 155 +++++----
lib/coderay/scanners/groovy.rb | 21 +-
lib/coderay/scanners/html.rb | 66 ++--
lib/coderay/scanners/java.rb | 2 +-
lib/coderay/scanners/java_script.rb | 36 ++-
lib/coderay/scanners/json.rb | 27 +-
lib/coderay/scanners/lua.rb | 280 ++++++++++++++++
lib/coderay/scanners/php.rb | 31 +-
lib/coderay/scanners/python.rb | 6 +-
lib/coderay/scanners/raydebug.rb | 45 +--
lib/coderay/scanners/ruby.rb | 35 +-
lib/coderay/scanners/ruby/patterns.rb | 41 +--
lib/coderay/scanners/ruby/string_state.rb | 14 +-
lib/coderay/scanners/sass.rb | 232 +++++++++++++
lib/coderay/scanners/sql.rb | 11 +-
lib/coderay/scanners/taskpaper.rb | 36 +++
lib/coderay/scanners/yaml.rb | 4 +-
lib/coderay/styles/alpha.rb | 56 ++--
lib/coderay/token_kinds.rb | 145 ++++-----
lib/coderay/tokens.rb | 100 ++----
lib/coderay/version.rb | 2 +-
metadata.yml | 30 +-
test/functional/basic.rb | 4 +-
test/functional/examples.rb | 6 +-
test/functional/for_redcloth.rb | 14 +-
49 files changed, 1622 insertions(+), 1362 deletions(-)
diff --git a/LICENSE b/LICENSE
deleted file mode 100644
index c00103d..0000000
--- a/LICENSE
+++ /dev/null
@@ -1,504 +0,0 @@
- GNU LESSER GENERAL PUBLIC LICENSE
- Version 2.1, February 1999
-
- Copyright (C) 1991, 1999 Free Software Foundation, Inc.
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
-[This is the first released version of the Lesser GPL. It also counts
- as the successor of the GNU Library Public License, version 2, hence
- the version number 2.1.]
-
- Preamble
-
- The licenses for most software are designed to take away your
-freedom to share and change it. By contrast, the GNU General Public
-Licenses are intended to guarantee your freedom to share and change
-free software--to make sure the software is free for all its users.
-
- This license, the Lesser General Public License, applies to some
-specially designated software packages--typically libraries--of the
-Free Software Foundation and other authors who decide to use it. You
-can use it too, but we suggest you first think carefully about whether
-this license or the ordinary General Public License is the better
-strategy to use in any particular case, based on the explanations below.
-
- When we speak of free software, we are referring to freedom of use,
-not price. Our General Public Licenses are designed to make sure that
-you have the freedom to distribute copies of free software (and charge
-for this service if you wish); that you receive source code or can get
-it if you want it; that you can change the software and use pieces of
-it in new free programs; and that you are informed that you can do
-these things.
-
- To protect your rights, we need to make restrictions that forbid
-distributors to deny you these rights or to ask you to surrender these
-rights. These restrictions translate to certain responsibilities for
-you if you distribute copies of the library or if you modify it.
-
- For example, if you distribute copies of the library, whether gratis
-or for a fee, you must give the recipients all the rights that we gave
-you. You must make sure that they, too, receive or can get the source
-code. If you link other code with the library, you must provide
-complete object files to the recipients, so that they can relink them
-with the library after making changes to the library and recompiling
-it. And you must show them these terms so they know their rights.
-
- We protect your rights with a two-step method: (1) we copyright the
-library, and (2) we offer you this license, which gives you legal
-permission to copy, distribute and/or modify the library.
-
- To protect each distributor, we want to make it very clear that
-there is no warranty for the free library. Also, if the library is
-modified by someone else and passed on, the recipients should know
-that what they have is not the original version, so that the original
-author's reputation will not be affected by problems that might be
-introduced by others.
-
- Finally, software patents pose a constant threat to the existence of
-any free program. We wish to make sure that a company cannot
-effectively restrict the users of a free program by obtaining a
-restrictive license from a patent holder. Therefore, we insist that
-any patent license obtained for a version of the library must be
-consistent with the full freedom of use specified in this license.
-
- Most GNU software, including some libraries, is covered by the
-ordinary GNU General Public License. This license, the GNU Lesser
-General Public License, applies to certain designated libraries, and
-is quite different from the ordinary General Public License. We use
-this license for certain libraries in order to permit linking those
-libraries into non-free programs.
-
- When a program is linked with a library, whether statically or using
-a shared library, the combination of the two is legally speaking a
-combined work, a derivative of the original library. The ordinary
-General Public License therefore permits such linking only if the
-entire combination fits its criteria of freedom. The Lesser General
-Public License permits more lax criteria for linking other code with
-the library.
-
- We call this license the "Lesser" General Public License because it
-does Less to protect the user's freedom than the ordinary General
-Public License. It also provides other free software developers Less
-of an advantage over competing non-free programs. These disadvantages
-are the reason we use the ordinary General Public License for many
-libraries. However, the Lesser license provides advantages in certain
-special circumstances.
-
- For example, on rare occasions, there may be a special need to
-encourage the widest possible use of a certain library, so that it becomes
-a de-facto standard. To achieve this, non-free programs must be
-allowed to use the library. A more frequent case is that a free
-library does the same job as widely used non-free libraries. In this
-case, there is little to gain by limiting the free library to free
-software only, so we use the Lesser General Public License.
-
- In other cases, permission to use a particular library in non-free
-programs enables a greater number of people to use a large body of
-free software. For example, permission to use the GNU C Library in
-non-free programs enables many more people to use the whole GNU
-operating system, as well as its variant, the GNU/Linux operating
-system.
-
- Although the Lesser General Public License is Less protective of the
-users' freedom, it does ensure that the user of a program that is
-linked with the Library has the freedom and the wherewithal to run
-that program using a modified version of the Library.
-
- The precise terms and conditions for copying, distribution and
-modification follow. Pay close attention to the difference between a
-"work based on the library" and a "work that uses the library". The
-former contains code derived from the library, whereas the latter must
-be combined with the library in order to run.
-
- GNU LESSER GENERAL PUBLIC LICENSE
- TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
-
- 0. This License Agreement applies to any software library or other
-program which contains a notice placed by the copyright holder or
-other authorized party saying it may be distributed under the terms of
-this Lesser General Public License (also called "this License").
-Each licensee is addressed as "you".
-
- A "library" means a collection of software functions and/or data
-prepared so as to be conveniently linked with application programs
-(which use some of those functions and data) to form executables.
-
- The "Library", below, refers to any such software library or work
-which has been distributed under these terms. A "work based on the
-Library" means either the Library or any derivative work under
-copyright law: that is to say, a work containing the Library or a
-portion of it, either verbatim or with modifications and/or translated
-straightforwardly into another language. (Hereinafter, translation is
-included without limitation in the term "modification".)
-
- "Source code" for a work means the preferred form of the work for
-making modifications to it. For a library, complete source code means
-all the source code for all modules it contains, plus any associated
-interface definition files, plus the scripts used to control compilation
-and installation of the library.
-
- Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope. The act of
-running a program using the Library is not restricted, and output from
-such a program is covered only if its contents constitute a work based
-on the Library (independent of the use of the Library in a tool for
-writing it). Whether that is true depends on what the Library does
-and what the program that uses the Library does.
-
- 1. You may copy and distribute verbatim copies of the Library's
-complete source code as you receive it, in any medium, provided that
-you conspicuously and appropriately publish on each copy an
-appropriate copyright notice and disclaimer of warranty; keep intact
-all the notices that refer to this License and to the absence of any
-warranty; and distribute a copy of this License along with the
-Library.
-
- You may charge a fee for the physical act of transferring a copy,
-and you may at your option offer warranty protection in exchange for a
-fee.
-
- 2. You may modify your copy or copies of the Library or any portion
-of it, thus forming a work based on the Library, and copy and
-distribute such modifications or work under the terms of Section 1
-above, provided that you also meet all of these conditions:
-
- a) The modified work must itself be a software library.
-
- b) You must cause the files modified to carry prominent notices
- stating that you changed the files and the date of any change.
-
- c) You must cause the whole of the work to be licensed at no
- charge to all third parties under the terms of this License.
-
- d) If a facility in the modified Library refers to a function or a
- table of data to be supplied by an application program that uses
- the facility, other than as an argument passed when the facility
- is invoked, then you must make a good faith effort to ensure that,
- in the event an application does not supply such function or
- table, the facility still operates, and performs whatever part of
- its purpose remains meaningful.
-
- (For example, a function in a library to compute square roots has
- a purpose that is entirely well-defined independent of the
- application. Therefore, Subsection 2d requires that any
- application-supplied function or table used by this function must
- be optional: if the application does not supply it, the square
- root function must still compute square roots.)
-
-These requirements apply to the modified work as a whole. If
-identifiable sections of that work are not derived from the Library,
-and can be reasonably considered independent and separate works in
-themselves, then this License, and its terms, do not apply to those
-sections when you distribute them as separate works. But when you
-distribute the same sections as part of a whole which is a work based
-on the Library, the distribution of the whole must be on the terms of
-this License, whose permissions for other licensees extend to the
-entire whole, and thus to each and every part regardless of who wrote
-it.
-
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is to
-exercise the right to control the distribution of derivative or
-collective works based on the Library.
-
-In addition, mere aggregation of another work not based on the Library
-with the Library (or with a work based on the Library) on a volume of
-a storage or distribution medium does not bring the other work under
-the scope of this License.
-
- 3. You may opt to apply the terms of the ordinary GNU General Public
-License instead of this License to a given copy of the Library. To do
-this, you must alter all the notices that refer to this License, so
-that they refer to the ordinary GNU General Public License, version 2,
-instead of to this License. (If a newer version than version 2 of the
-ordinary GNU General Public License has appeared, then you can specify
-that version instead if you wish.) Do not make any other change in
-these notices.
-
- Once this change is made in a given copy, it is irreversible for
-that copy, so the ordinary GNU General Public License applies to all
-subsequent copies and derivative works made from that copy.
-
- This option is useful when you wish to copy part of the code of
-the Library into a program that is not a library.
-
- 4. You may copy and distribute the Library (or a portion or
-derivative of it, under Section 2) in object code or executable form
-under the terms of Sections 1 and 2 above provided that you accompany
-it with the complete corresponding machine-readable source code, which
-must be distributed under the terms of Sections 1 and 2 above on a
-medium customarily used for software interchange.
-
- If distribution of object code is made by offering access to copy
-from a designated place, then offering equivalent access to copy the
-source code from the same place satisfies the requirement to
-distribute the source code, even though third parties are not
-compelled to copy the source along with the object code.
-
- 5. A program that contains no derivative of any portion of the
-Library, but is designed to work with the Library by being compiled or
-linked with it, is called a "work that uses the Library". Such a
-work, in isolation, is not a derivative work of the Library, and
-therefore falls outside the scope of this License.
-
- However, linking a "work that uses the Library" with the Library
-creates an executable that is a derivative of the Library (because it
-contains portions of the Library), rather than a "work that uses the
-library". The executable is therefore covered by this License.
-Section 6 states terms for distribution of such executables.
-
- When a "work that uses the Library" uses material from a header file
-that is part of the Library, the object code for the work may be a
-derivative work of the Library even though the source code is not.
-Whether this is true is especially significant if the work can be
-linked without the Library, or if the work is itself a library. The
-threshold for this to be true is not precisely defined by law.
-
- If such an object file uses only numerical parameters, data
-structure layouts and accessors, and small macros and small inline
-functions (ten lines or less in length), then the use of the object
-file is unrestricted, regardless of whether it is legally a derivative
-work. (Executables containing this object code plus portions of the
-Library will still fall under Section 6.)
-
- Otherwise, if the work is a derivative of the Library, you may
-distribute the object code for the work under the terms of Section 6.
-Any executables containing that work also fall under Section 6,
-whether or not they are linked directly with the Library itself.
-
- 6. As an exception to the Sections above, you may also combine or
-link a "work that uses the Library" with the Library to produce a
-work containing portions of the Library, and distribute that work
-under terms of your choice, provided that the terms permit
-modification of the work for the customer's own use and reverse
-engineering for debugging such modifications.
-
- You must give prominent notice with each copy of the work that the
-Library is used in it and that the Library and its use are covered by
-this License. You must supply a copy of this License. If the work
-during execution displays copyright notices, you must include the
-copyright notice for the Library among them, as well as a reference
-directing the user to the copy of this License. Also, you must do one
-of these things:
-
- a) Accompany the work with the complete corresponding
- machine-readable source code for the Library including whatever
- changes were used in the work (which must be distributed under
- Sections 1 and 2 above); and, if the work is an executable linked
- with the Library, with the complete machine-readable "work that
- uses the Library", as object code and/or source code, so that the
- user can modify the Library and then relink to produce a modified
- executable containing the modified Library. (It is understood
- that the user who changes the contents of definitions files in the
- Library will not necessarily be able to recompile the application
- to use the modified definitions.)
-
- b) Use a suitable shared library mechanism for linking with the
- Library. A suitable mechanism is one that (1) uses at run time a
- copy of the library already present on the user's computer system,
- rather than copying library functions into the executable, and (2)
- will operate properly with a modified version of the library, if
- the user installs one, as long as the modified version is
- interface-compatible with the version that the work was made with.
-
- c) Accompany the work with a written offer, valid for at
- least three years, to give the same user the materials
- specified in Subsection 6a, above, for a charge no more
- than the cost of performing this distribution.
-
- d) If distribution of the work is made by offering access to copy
- from a designated place, offer equivalent access to copy the above
- specified materials from the same place.
-
- e) Verify that the user has already received a copy of these
- materials or that you have already sent this user a copy.
-
- For an executable, the required form of the "work that uses the
-Library" must include any data and utility programs needed for
-reproducing the executable from it. However, as a special exception,
-the materials to be distributed need not include anything that is
-normally distributed (in either source or binary form) with the major
-components (compiler, kernel, and so on) of the operating system on
-which the executable runs, unless that component itself accompanies
-the executable.
-
- It may happen that this requirement contradicts the license
-restrictions of other proprietary libraries that do not normally
-accompany the operating system. Such a contradiction means you cannot
-use both them and the Library together in an executable that you
-distribute.
-
- 7. You may place library facilities that are a work based on the
-Library side-by-side in a single library together with other library
-facilities not covered by this License, and distribute such a combined
-library, provided that the separate distribution of the work based on
-the Library and of the other library facilities is otherwise
-permitted, and provided that you do these two things:
-
- a) Accompany the combined library with a copy of the same work
- based on the Library, uncombined with any other library
- facilities. This must be distributed under the terms of the
- Sections above.
-
- b) Give prominent notice with the combined library of the fact
- that part of it is a work based on the Library, and explaining
- where to find the accompanying uncombined form of the same work.
-
- 8. You may not copy, modify, sublicense, link with, or distribute
-the Library except as expressly provided under this License. Any
-attempt otherwise to copy, modify, sublicense, link with, or
-distribute the Library is void, and will automatically terminate your
-rights under this License. However, parties who have received copies,
-or rights, from you under this License will not have their licenses
-terminated so long as such parties remain in full compliance.
-
- 9. You are not required to accept this License, since you have not
-signed it. However, nothing else grants you permission to modify or
-distribute the Library or its derivative works. These actions are
-prohibited by law if you do not accept this License. Therefore, by
-modifying or distributing the Library (or any work based on the
-Library), you indicate your acceptance of this License to do so, and
-all its terms and conditions for copying, distributing or modifying
-the Library or works based on it.
-
- 10. Each time you redistribute the Library (or any work based on the
-Library), the recipient automatically receives a license from the
-original licensor to copy, distribute, link with or modify the Library
-subject to these terms and conditions. You may not impose any further
-restrictions on the recipients' exercise of the rights granted herein.
-You are not responsible for enforcing compliance by third parties with
-this License.
-
- 11. If, as a consequence of a court judgment or allegation of patent
-infringement or for any other reason (not limited to patent issues),
-conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License. If you cannot
-distribute so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you
-may not distribute the Library at all. For example, if a patent
-license would not permit royalty-free redistribution of the Library by
-all those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Library.
-
-If any portion of this section is held invalid or unenforceable under any
-particular circumstance, the balance of the section is intended to apply,
-and the section as a whole is intended to apply in other circumstances.
-
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the
-integrity of the free software distribution system which is
-implemented by public license practices. Many people have made
-generous contributions to the wide range of software distributed
-through that system in reliance on consistent application of that
-system; it is up to the author/donor to decide if he or she is willing
-to distribute software through any other system and a licensee cannot
-impose that choice.
-
-This section is intended to make thoroughly clear what is believed to
-be a consequence of the rest of this License.
-
- 12. If the distribution and/or use of the Library is restricted in
-certain countries either by patents or by copyrighted interfaces, the
-original copyright holder who places the Library under this License may add
-an explicit geographical distribution limitation excluding those countries,
-so that distribution is permitted only in or among countries not thus
-excluded. In such case, this License incorporates the limitation as if
-written in the body of this License.
-
- 13. The Free Software Foundation may publish revised and/or new
-versions of the Lesser General Public License from time to time.
-Such new versions will be similar in spirit to the present version,
-but may differ in detail to address new problems or concerns.
-
-Each version is given a distinguishing version number. If the Library
-specifies a version number of this License which applies to it and
-"any later version", you have the option of following the terms and
-conditions either of that version or of any later version published by
-the Free Software Foundation. If the Library does not specify a
-license version number, you may choose any version ever published by
-the Free Software Foundation.
-
- 14. If you wish to incorporate parts of the Library into other free
-programs whose distribution conditions are incompatible with these,
-write to the author to ask for permission. For software which is
-copyrighted by the Free Software Foundation, write to the Free
-Software Foundation; we sometimes make exceptions for this. Our
-decision will be guided by the two goals of preserving the free status
-of all derivatives of our free software and of promoting the sharing
-and reuse of software generally.
-
- NO WARRANTY
-
- 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
-WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
-EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
-OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
-KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
-LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
-THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
-
- 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
-WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
-AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
-FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
-CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
-LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
-RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
-FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
-SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
-DAMAGES.
-
- END OF TERMS AND CONDITIONS
-
- How to Apply These Terms to Your New Libraries
-
- If you develop a new library, and you want it to be of the greatest
-possible use to the public, we recommend making it free software that
-everyone can redistribute and change. You can do so by permitting
-redistribution under these terms (or, alternatively, under the terms of the
-ordinary General Public License).
-
- To apply these terms, attach the following notices to the library. It is
-safest to attach them to the start of each source file to most effectively
-convey the exclusion of warranty; and each file should have at least the
-"copyright" line and a pointer to where the full notice is found.
-
- <one line to give the library's name and a brief idea of what it does.>
- Copyright (C) <year> <name of author>
-
- This library is free software; you can redistribute it and/or
- modify it under the terms of the GNU Lesser General Public
- License as published by the Free Software Foundation; either
- version 2.1 of the License, or (at your option) any later version.
-
- This library is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- Lesser General Public License for more details.
-
- You should have received a copy of the GNU Lesser General Public
- License along with this library; if not, write to the Free Software
- Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
-
-Also add information on how to contact you by electronic and paper mail.
-
-You should also get your employer (if you work as a programmer) or your
-school, if any, to sign a "copyright disclaimer" for the library, if
-necessary. Here is a sample; alter the names:
-
- Yoyodyne, Inc., hereby disclaims all copyright interest in the
- library `Frob' (a library for tweaking knobs) written by James Random Hacker.
-
- <signature of Ty Coon>, 1 April 1990
- Ty Coon, President of Vice
-
-That's all there is to it!
-
-
diff --git a/Rakefile b/Rakefile
index ba6c34e..c9b1e8a 100644
--- a/Rakefile
+++ b/Rakefile
@@ -1,3 +1,5 @@
+require 'bundler/gem_tasks'
+
$:.unshift File.dirname(__FILE__) unless $:.include? '.'
ROOT = '.'
@@ -32,4 +34,4 @@ else
rd.rdoc_dir = 'doc'
end
-end
\ No newline at end of file
+end
diff --git a/bin/coderay b/bin/coderay
index d78cd57..889ae72 100755
--- a/bin/coderay
+++ b/bin/coderay
@@ -125,7 +125,7 @@ when 'highlight', nil
end
if output_file
- output_format ||= CodeRay::FileType[output_file]
+ output_format ||= CodeRay::FileType[output_file] || :plain
else
output_format ||= :terminal
end
@@ -143,7 +143,6 @@ when 'highlight', nil
if output_file
File.open output_file, 'w'
else
- $stdout.sync = true
$stdout
end
CodeRay.encode(input, input_lang, output_format, :out => file)
@@ -156,8 +155,9 @@ when 'highlight', nil
puts boom.message
end
# puts "I don't know this plugin: #{boom.message[/Could not load plugin (.*?): /, 1]}."
- rescue CodeRay::Scanners::Scanner::ScanError # FIXME: rescue Errno::EPIPE
- # this is sometimes raised by pagers; ignore [TODO: wtf?]
+ rescue CodeRay::Scanners::Scanner::ScanError
+ # this is sometimes raised by pagers; ignore
+ # FIXME: rescue Errno::EPIPE
ensure
file.close if output_file
end
diff --git a/checksums.yaml.gz b/checksums.yaml.gz
new file mode 100644
index 0000000..a6391ab
Binary files /dev/null and b/checksums.yaml.gz differ
diff --git a/lib/coderay.rb b/lib/coderay.rb
index 876d770..f759ed6 100644
--- a/lib/coderay.rb
+++ b/lib/coderay.rb
@@ -78,12 +78,12 @@
# Read this to get a general view what CodeRay provides.
#
# == Scanning
-#
-# Scanning means analysing an input string, splitting it up into Tokens.
-# Each Token knows about what type it is: string, comment, class name, etc.
+#
+# Scanning means analysing an input string, splitting it up into Tokens.
+# Each Token knows about what type it is: string, comment, class name, etc.
#
-# Each +lang+ (language) has its own Scanner; for example, <tt>:ruby</tt> code is
-# handled by CodeRay::Scanners::Ruby.
+# Each +lang+ (language) has its own Scanner; for example, <tt>:ruby</tt> code is
+# handled by CodeRay::Scanners::Ruby.
#
# CodeRay.scan:: Scan a string in a given language into Tokens.
# This is the most common method to use.
@@ -127,14 +127,14 @@ module CodeRay
$CODERAY_DEBUG ||= false
- CODERAY_PATH = File.join File.dirname(__FILE__), 'coderay'
+ CODERAY_PATH = File.expand_path('../coderay', __FILE__)
# Assuming the path is a subpath of lib/coderay/
def self.coderay_path *path
File.join CODERAY_PATH, *path
end
- require coderay_path('version')
+ require 'coderay/version'
# helpers
autoload :FileType, coderay_path('helpers', 'file_type')
@@ -166,7 +166,6 @@ module CodeRay
#
# See also demo/demo_simple.
def scan code, lang, options = {}, &block
- # FIXME: return a proxy for direct-stream encoding
TokensProxy.new code, lang, options, block
end
diff --git a/lib/coderay/encoders/debug.rb b/lib/coderay/encoders/debug.rb
index 95d6138..f4db330 100644
--- a/lib/coderay/encoders/debug.rb
+++ b/lib/coderay/encoders/debug.rb
@@ -9,7 +9,6 @@ module Encoders
#
# You cannot fully restore the tokens information from the
# output, because consecutive :space tokens are merged.
- # Use Tokens#dump for caching purposes.
#
# See also: Scanners::Debug
class Debug < Encoder
@@ -18,37 +17,26 @@ module Encoders
FILE_EXTENSION = 'raydebug'
- def initialize options = {}
- super
- @opened = []
- end
-
def text_token text, kind
if kind == :space
@out << text
else
- # TODO: Escape (
- text = text.gsub(/[)\\]/, '\\\\\0') # escape ) and \
- @out << kind.to_s << '(' << text << ')'
+ text = text.gsub('\\', '\\\\\\\\') if text.index('\\')
+ text = text.gsub(')', '\\\\)') if text.index(')')
+ @out << "#{kind}(#{text})"
end
end
def begin_group kind
- @opened << kind
- @out << kind.to_s << '<'
+ @out << "#{kind}<"
end
def end_group kind
- if @opened.last != kind
- puts @out
- raise "we are inside #{@opened.inspect}, not #{kind}"
- end
- @opened.pop
@out << '>'
end
def begin_line kind
- @out << kind.to_s << '['
+ @out << "#{kind}["
end
def end_line kind
diff --git a/lib/coderay/encoders/debug_lint.rb b/lib/coderay/encoders/debug_lint.rb
new file mode 100644
index 0000000..a4eba2c
--- /dev/null
+++ b/lib/coderay/encoders/debug_lint.rb
@@ -0,0 +1,63 @@
+module CodeRay
+module Encoders
+
+ load :lint
+
+ # = Debug Lint Encoder
+ #
+ # Debug encoder with additional checks for:
+ #
+ # - empty tokens
+ # - incorrect nesting
+ #
+ # It will raise an InvalidTokenStream exception when any of the above occurs.
+ #
+ # See also: Encoders::Debug
+ class DebugLint < Debug
+
+ register_for :debug_lint
+
+ def text_token text, kind
+ raise Lint::EmptyToken, 'empty token for %p' % [kind] if text.empty?
+ raise Lint::UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
+ super
+ end
+
+ def begin_group kind
+ @opened << kind
+ super
+ end
+
+ def end_group kind
+ raise Lint::IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_group)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+ @opened.pop
+ super
+ end
+
+ def begin_line kind
+ @opened << kind
+ super
+ end
+
+ def end_line kind
+ raise Lint::IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_line)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+ @opened.pop
+ super
+ end
+
+ protected
+
+ def setup options
+ super
+ @opened = []
+ end
+
+ def finish options
+ raise 'Some tokens still open at end of token stream: %p' % [@opened] unless @opened.empty?
+ super
+ end
+
+ end
+
+end
+end
diff --git a/lib/coderay/encoders/html.rb b/lib/coderay/encoders/html.rb
index 635a4d8..d2ebb5a 100644
--- a/lib/coderay/encoders/html.rb
+++ b/lib/coderay/encoders/html.rb
@@ -126,22 +126,21 @@ module Encoders
protected
- HTML_ESCAPE = { #:nodoc:
- '&' => '&',
- '"' => '"',
- '>' => '>',
- '<' => '<',
- }
+ def self.make_html_escape_hash
+ {
+ '&' => '&',
+ '"' => '"',
+ '>' => '>',
+ '<' => '<',
+ # "\t" => will be set to ' ' * options[:tab_width] during setup
+ }.tap do |hash|
+ # Escape ASCII control codes except \x9 == \t and \xA == \n.
+ (Array(0x00..0x8) + Array(0xB..0x1F)).each { |invalid| hash[invalid.chr] = ' ' }
+ end
+ end
- # This was to prevent illegal HTML.
- # Strange chars should still be avoided in codes.
- evil_chars = Array(0x00...0x20) - [?\n, ?\t, ?\s]
- evil_chars.each { |i| HTML_ESCAPE[i.chr] = ' ' }
- #ansi_chars = Array(0x7f..0xff)
- #ansi_chars.each { |i| HTML_ESCAPE[i.chr] = '&#%d;' % i }
- # \x9 (\t) and \xA (\n) not included
- #HTML_ESCAPE_PATTERN = /[\t&"><\0-\x8\xB-\x1f\x7f-\xff]/
- HTML_ESCAPE_PATTERN = /[\t"&><\0-\x8\xB-\x1f]/
+ HTML_ESCAPE = make_html_escape_hash
+ HTML_ESCAPE_PATTERN = /[\t"&><\0-\x8\xB-\x1F]/
TOKEN_KIND_TO_INFO = Hash.new do |h, kind|
h[kind] = kind.to_s.gsub(/_/, ' ').gsub(/\b\w/) { $&.capitalize }
@@ -172,77 +171,41 @@ module Encoders
def setup options
super
+ check_options! options
+
if options[:wrap] || options[:line_numbers]
@real_out = @out
@out = ''
end
- options[:break_lines] = true if options[:line_numbers] == :inline
-
@break_lines = (options[:break_lines] == true)
- @HTML_ESCAPE = HTML_ESCAPE.dup
- @HTML_ESCAPE["\t"] = ' ' * options[:tab_width]
+ @HTML_ESCAPE = HTML_ESCAPE.merge("\t" => ' ' * options[:tab_width])
@opened = []
@last_opened = nil
@css = CSS.new options[:style]
- hint = options[:hint]
- if hint && ![:debug, :info, :info_long].include?(hint)
- raise ArgumentError, "Unknown value %p for :hint; \
- expected :info, :info_long, :debug, false, or nil." % hint
- end
-
- css_classes = TokenKinds
- case options[:css]
- when :class
- @span_for_kind = Hash.new do |h, k|
- if k.is_a? ::Symbol
- kind = k_dup = k
- else
- kind = k.first
- k_dup = k.dup
- end
- if kind != :space && (hint || css_class = css_classes[kind])
- title = HTML.token_path_to_hint hint, k if hint
- css_class ||= css_classes[kind]
- h[k_dup] = "<span#{title}#{" class=\"#{css_class}\"" if css_class}>"
- else
- h[k_dup] = nil
- end
- end
- when :style
- @span_for_kind = Hash.new do |h, k|
- kind = k.is_a?(Symbol) ? k : k.first
- h[k.is_a?(Symbol) ? k : k.dup] =
- if kind != :space && (hint || css_classes[kind])
- title = HTML.token_path_to_hint hint, k if hint
- style = @css.get_style Array(k).map { |c| css_classes[c] }
- "<span#{title}#{" style=\"#{style}\"" if style}>"
- end
- end
- else
- raise ArgumentError, "Unknown value %p for :css." % options[:css]
- end
+ @span_for_kinds = make_span_for_kinds(options[:css], options[:hint])
@set_last_opened = options[:hint] || options[:css] == :style
end
def finish options
unless @opened.empty?
- warn '%d tokens still open: %p' % [@opened.size, @opened] if $CODERAY_DEBUG
@out << '</span>' while @opened.pop
@last_opened = nil
end
- @out.extend Output
- @out.css = @css
- if options[:line_numbers]
- Numbering.number! @out, options[:line_numbers], options
+ if @out.respond_to? :to_str
+ @out.extend Output
+ @out.css = @css
+ if options[:line_numbers]
+ Numbering.number! @out, options[:line_numbers], options
+ end
+ @out.wrap! options[:wrap]
+ @out.apply_title! options[:title]
end
- @out.wrap! options[:wrap]
- @out.apply_title! options[:title]
if defined?(@real_out) && @real_out
@real_out << @out
@@ -255,20 +218,10 @@ module Encoders
public
def text_token text, kind
- if text =~ /#{HTML_ESCAPE_PATTERN}/o
- text = text.gsub(/#{HTML_ESCAPE_PATTERN}/o) { |m| @HTML_ESCAPE[m] }
- end
+ style = @span_for_kinds[@last_opened ? [kind, *@opened] : kind]
- style = @span_for_kind[@last_opened ? [kind, *@opened] : kind]
-
- if @break_lines && (i = text.index("\n")) && (c = @opened.size + (style ? 1 : 0)) > 0
- close = '</span>' * c
- reopen = ''
- @opened.each_with_index do |k, index|
- reopen << (@span_for_kind[index > 0 ? [k, *@opened[0 ... index ]] : k] || '<span>')
- end
- text[i .. -1] = text[i .. -1].gsub("\n", "#{close}\n#{reopen}#{style}")
- end
+ text = text.gsub(/#{HTML_ESCAPE_PATTERN}/o) { |m| @HTML_ESCAPE[m] } if text =~ /#{HTML_ESCAPE_PATTERN}/o
+ text = break_lines(text, style) if @break_lines && (style || @opened.size > 0) && text.index("\n")
if style
@out << style << text << '</span>'
@@ -279,25 +232,19 @@ module Encoders
# token groups, eg. strings
def begin_group kind
- @out << (@span_for_kind[@last_opened ? [kind, *@opened] : kind] || '<span>')
+ @out << (@span_for_kinds[@last_opened ? [kind, *@opened] : kind] || '<span>')
@opened << kind
@last_opened = kind if @set_last_opened
end
def end_group kind
- if $CODERAY_DEBUG && (@opened.empty? || @opened.last != kind)
- warn 'Malformed token stream: Trying to close a token (%p) ' \
- 'that is not open. Open are: %p.' % [kind, @opened[1..-1]]
- end
- if @opened.pop
- @out << '</span>'
- @last_opened = @opened.last if @last_opened
- end
+ check_group_nesting 'token group', kind if $CODERAY_DEBUG
+ close_span
end
# whole lines to be highlighted, eg. a deleted line in a diff
def begin_line kind
- if style = @span_for_kind[@last_opened ? [kind, *@opened] : kind]
+ if style = @span_for_kinds[@last_opened ? [kind, *@opened] : kind]
if style['class="']
@out << style.sub('class="', 'class="line ')
else
@@ -311,16 +258,74 @@ module Encoders
end
def end_line kind
- if $CODERAY_DEBUG && (@opened.empty? || @opened.last != kind)
- warn 'Malformed token stream: Trying to close a line (%p) ' \
- 'that is not open. Open are: %p.' % [kind, @opened[1..-1]]
+ check_group_nesting 'line', kind if $CODERAY_DEBUG
+ close_span
+ end
+
+ protected
+
+ def check_options! options
+ unless [false, nil, :debug, :info, :info_long].include? options[:hint]
+ raise ArgumentError, "Unknown value %p for :hint; expected :info, :info_long, :debug, false, or nil." % [options[:hint]]
end
+
+ unless [:class, :style].include? options[:css]
+ raise ArgumentError, 'Unknown value %p for :css.' % [options[:css]]
+ end
+
+ options[:break_lines] = true if options[:line_numbers] == :inline
+ end
+
+ def css_class_for_kinds kinds
+ TokenKinds[kinds.is_a?(Symbol) ? kinds : kinds.first]
+ end
+
+ def style_for_kinds kinds
+ css_classes = kinds.is_a?(Array) ? kinds.map { |c| TokenKinds[c] } : [TokenKinds[kinds]]
+ @css.get_style_for_css_classes css_classes
+ end
+
+ def make_span_for_kinds method, hint
+ Hash.new do |h, kinds|
+ begin
+ css_class = css_class_for_kinds(kinds)
+ title = HTML.token_path_to_hint hint, kinds if hint
+
+ if css_class || title
+ if method == :style
+ style = style_for_kinds(kinds)
+ "<span#{title}#{" style=\"#{style}\"" if style}>"
+ else
+ "<span#{title}#{" class=\"#{css_class}\"" if css_class}>"
+ end
+ end
+ end.tap do |span|
+ h.clear if h.size >= 100
+ h[kinds] = span
+ end
+ end
+ end
+
+ def check_group_nesting name, kind
+ if @opened.empty? || @opened.last != kind
+ warn "Malformed token stream: Trying to close a #{name} (%p) that is not open. Open are: %p." % [kind, @opened[1..-1]]
+ end
+ end
+
+ def break_lines text, style
+ reopen = ''
+ @opened.each_with_index do |kind, index|
+ reopen << (@span_for_kinds[index > 0 ? [kind, *@opened[0...index]] : kind] || '<span>')
+ end
+ text.gsub("\n", "#{'</span>' * @opened.size}#{'</span>' if style}\n#{reopen}#{style}")
+ end
+
+ def close_span
if @opened.pop
@out << '</span>'
@last_opened = @opened.last if @last_opened
end
end
-
end
end
diff --git a/lib/coderay/encoders/html/css.rb b/lib/coderay/encoders/html/css.rb
index 6de4b46..164d7f8 100644
--- a/lib/coderay/encoders/html/css.rb
+++ b/lib/coderay/encoders/html/css.rb
@@ -11,7 +11,7 @@ module Encoders
end
def initialize style = :default
- @classes = Hash.new
+ @styles = Hash.new
style = CSS.load_stylesheet style
@stylesheet = [
style::CSS_MAIN_STYLES,
@@ -20,12 +20,12 @@ module Encoders
parse style::TOKEN_COLORS
end
- def get_style styles
- cl = @classes[styles.first]
+ def get_style_for_css_classes css_classes
+ cl = @styles[css_classes.first]
return '' unless cl
style = ''
- 1.upto styles.size do |offset|
- break if style = cl[styles[offset .. -1]]
+ 1.upto css_classes.size do |offset|
+ break if style = cl[css_classes[offset .. -1]]
end
# warn 'Style not found: %p' % [styles] if style.empty?
return style
@@ -52,8 +52,8 @@ module Encoders
for selector in selectors.split(',')
classes = selector.scan(/[-\w]+/)
cl = classes.pop
- @classes[cl] ||= Hash.new
- @classes[cl][classes] = style.to_s.strip.delete(' ').chomp(';')
+ @styles[cl] ||= Hash.new
+ @styles[cl][classes] = style.to_s.strip.delete(' ').chomp(';')
end
end
end
diff --git a/lib/coderay/encoders/html/numbering.rb b/lib/coderay/encoders/html/numbering.rb
index 8bc6259..a1b9c04 100644
--- a/lib/coderay/encoders/html/numbering.rb
+++ b/lib/coderay/encoders/html/numbering.rb
@@ -1,15 +1,15 @@
module CodeRay
module Encoders
-
+
class HTML
-
+
module Numbering # :nodoc:
-
+
def self.number! output, mode = :table, options = {}
return self unless mode
-
+
options = DEFAULT_OPTIONS.merge options
-
+
start = options[:line_number_start]
unless start.is_a? Integer
raise ArgumentError, "Invalid value %p for :line_number_start; Integer expected." % start
@@ -17,7 +17,7 @@ module Encoders
anchor_prefix = options[:line_number_anchors]
anchor_prefix = 'line' if anchor_prefix == true
- anchor_prefix = anchor_prefix.to_s[/\w+/] if anchor_prefix
+ anchor_prefix = anchor_prefix.to_s[/[\w-]+/] if anchor_prefix
anchoring =
if anchor_prefix
proc do |line|
@@ -26,7 +26,7 @@ module Encoders
"<a href=\"##{anchor}\" name=\"#{anchor}\">#{line}</a>"
end
else
- proc { |line| line.to_s } # :to_s.to_proc in Ruby 1.8.7+
+ :to_s.to_proc
end
bold_every = options[:bold_every]
@@ -56,12 +56,17 @@ module Encoders
raise ArgumentError, 'Invalid value %p for :bolding; false or Integer expected.' % bold_every
end
- line_count = output.count("\n")
- position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
- if position_of_last_newline
+ if position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
after_last_newline = output[position_of_last_newline + 1 .. -1]
ends_with_newline = after_last_newline[/\A(?:<\/span>)*\z/]
- line_count += 1 if not ends_with_newline
+
+ if ends_with_newline
+ line_count = output.count("\n")
+ else
+ line_count = output.count("\n") + 1
+ end
+ else
+ line_count = 1
end
case mode
@@ -70,34 +75,34 @@ module Encoders
line_number = start
output.gsub!(/^.*$\n?/) do |line|
line_number_text = bolding.call line_number
- indent = ' ' * (max_width - line_number.to_s.size) # TODO: Optimize (10^x)
+ indent = ' ' * (max_width - line_number.to_s.size)
line_number += 1
"<span class=\"line-numbers\">#{indent}#{line_number_text}</span>#{line}"
end
-
+
when :table
line_numbers = (start ... start + line_count).map(&bolding).join("\n")
line_numbers << "\n"
line_numbers_table_template = Output::TABLE.apply('LINE_NUMBERS', line_numbers)
-
+
output.gsub!(/<\/div>\n/, '</div>')
output.wrap_in! line_numbers_table_template
output.wrapped_in = :div
-
+
when :list
raise NotImplementedError, 'The :list option is no longer available. Use :table.'
-
+
else
raise ArgumentError, 'Unknown value %p for mode: expected one of %p' %
[mode, [:table, :inline]]
end
-
+
output
end
-
+
end
-
+
end
-
+
end
end
diff --git a/lib/coderay/encoders/html/output.rb b/lib/coderay/encoders/html/output.rb
index 9132d94..de6f6ea 100644
--- a/lib/coderay/encoders/html/output.rb
+++ b/lib/coderay/encoders/html/output.rb
@@ -124,7 +124,7 @@ module Encoders
TABLE = Template.new <<-TABLE
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><%LINE_NUMBERS%></pre></td>
+ <td class="line-numbers"><pre><%LINE_NUMBERS%></pre></td>
<td class="code"><pre><%CONTENT%></pre></td>
</tr></table>
TABLE
diff --git a/lib/coderay/encoders/lint.rb b/lib/coderay/encoders/lint.rb
new file mode 100644
index 0000000..88c8bd1
--- /dev/null
+++ b/lib/coderay/encoders/lint.rb
@@ -0,0 +1,59 @@
+module CodeRay
+module Encoders
+
+ # = Lint Encoder
+ #
+ # Checks for:
+ #
+ # - empty tokens
+ # - incorrect nesting
+ #
+ # It will raise an InvalidTokenStream exception when any of the above occurs.
+ #
+ # See also: Encoders::DebugLint
+ class Lint < Debug
+
+ register_for :lint
+
+ InvalidTokenStream = Class.new StandardError
+ EmptyToken = Class.new InvalidTokenStream
+ UnknownTokenKind = Class.new InvalidTokenStream
+ IncorrectTokenGroupNesting = Class.new InvalidTokenStream
+
+ def text_token text, kind
+ raise EmptyToken, 'empty token for %p' % [kind] if text.empty?
+ raise UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
+ end
+
+ def begin_group kind
+ @opened << kind
+ end
+
+ def end_group kind
+ raise IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_group)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+ @opened.pop
+ end
+
+ def begin_line kind
+ @opened << kind
+ end
+
+ def end_line kind
+ raise IncorrectTokenGroupNesting, 'We are inside %s, not %p (end_line)' % [@opened.reverse.map(&:inspect).join(' < '), kind] if @opened.last != kind
+ @opened.pop
+ end
+
+ protected
+
+ def setup options
+ @opened = []
+ end
+
+ def finish options
+ raise 'Some tokens still open at end of token stream: %p' % [@opened] unless @opened.empty?
+ end
+
+ end
+
+end
+end
diff --git a/lib/coderay/encoders/statistic.rb b/lib/coderay/encoders/statistic.rb
index 2315d9e..b2f8b83 100644
--- a/lib/coderay/encoders/statistic.rb
+++ b/lib/coderay/encoders/statistic.rb
@@ -67,7 +67,6 @@ Token Types (%d):
@type_stats['TOTAL'].count += 1
end
- # TODO Hierarchy handling
def begin_group kind
block_token ':begin_group', kind
end
diff --git a/lib/coderay/encoders/terminal.rb b/lib/coderay/encoders/terminal.rb
index 005032d..c7ae014 100644
--- a/lib/coderay/encoders/terminal.rb
+++ b/lib/coderay/encoders/terminal.rb
@@ -19,105 +19,135 @@ module CodeRay
register_for :terminal
TOKEN_COLORS = {
- :annotation => '35',
- :attribute_name => '33',
- :attribute_value => '31',
- :binary => '1;35',
+ :debug => "\e[1;37;44m",
+
+ :annotation => "\e[34m",
+ :attribute_name => "\e[35m",
+ :attribute_value => "\e[31m",
+ :binary => {
+ :self => "\e[31m",
+ :char => "\e[1;31m",
+ :delimiter => "\e[1;31m",
+ },
:char => {
- :self => '36', :delimiter => '1;34'
+ :self => "\e[35m",
+ :delimiter => "\e[1;35m"
+ },
+ :class => "\e[1;35;4m",
+ :class_variable => "\e[36m",
+ :color => "\e[32m",
+ :comment => {
+ :self => "\e[1;30m",
+ :char => "\e[37m",
+ :delimiter => "\e[37m",
},
- :class => '1;35',
- :class_variable => '36',
- :color => '32',
- :comment => '37',
- :complex => '1;34',
- :constant => ['1;34', '4'],
- :decoration => '35',
- :definition => '1;32',
- :directive => ['32', '4'],
- :doc => '46',
- :doctype => '1;30',
- :doc_string => ['31', '4'],
- :entity => '33',
- :error => ['1;33', '41'],
- :exception => '1;31',
- :float => '1;35',
- :function => '1;34',
- :global_variable => '42',
- :hex => '1;36',
- :include => '33',
- :integer => '1;34',
- :key => '35',
- :label => '1;15',
- :local_variable => '33',
- :octal => '1;35',
- :operator_name => '1;29',
- :predefined_constant => '1;36',
- :predefined_type => '1;30',
- :predefined => ['4', '1;34'],
- :preprocessor => '36',
- :pseudo_class => '1;34',
+ :constant => "\e[1;34;4m",
+ :decorator => "\e[35m",
+ :definition => "\e[1;33m",
+ :directive => "\e[33m",
+ :docstring => "\e[31m",
+ :doctype => "\e[1;34m",
+ :done => "\e[1;30;2m",
+ :entity => "\e[31m",
+ :error => "\e[1;37;41m",
+ :exception => "\e[1;31m",
+ :float => "\e[1;35m",
+ :function => "\e[1;34m",
+ :global_variable => "\e[1;32m",
+ :hex => "\e[1;36m",
+ :id => "\e[1;34m",
+ :include => "\e[31m",
+ :integer => "\e[1;34m",
+ :imaginary => "\e[1;34m",
+ :important => "\e[1;31m",
+ :key => {
+ :self => "\e[35m",
+ :char => "\e[1;35m",
+ :delimiter => "\e[1;35m",
+ },
+ :keyword => "\e[32m",
+ :label => "\e[1;33m",
+ :local_variable => "\e[33m",
+ :namespace => "\e[1;35m",
+ :octal => "\e[1;34m",
+ :predefined => "\e[36m",
+ :predefined_constant => "\e[1;36m",
+ :predefined_type => "\e[1;32m",
+ :preprocessor => "\e[1;36m",
+ :pseudo_class => "\e[1;34m",
:regexp => {
- :self => '31',
- :content => '31',
- :delimiter => '1;29',
- :modifier => '35',
- :function => '1;29'
+ :self => "\e[35m",
+ :delimiter => "\e[1;35m",
+ :modifier => "\e[35m",
+ :char => "\e[1;35m",
},
- :reserved => '1;31',
+ :reserved => "\e[32m",
:shell => {
- :self => '42',
- :content => '1;29',
- :delimiter => '37',
+ :self => "\e[33m",
+ :char => "\e[1;33m",
+ :delimiter => "\e[1;33m",
+ :escape => "\e[1;33m",
},
:string => {
- :self => '32',
- :modifier => '1;32',
- :escape => '1;36',
- :delimiter => '1;32',
+ :self => "\e[31m",
+ :modifier => "\e[1;31m",
+ :char => "\e[1;35m",
+ :delimiter => "\e[1;31m",
+ :escape => "\e[1;31m",
+ },
+ :symbol => {
+ :self => "\e[33m",
+ :delimiter => "\e[1;33m",
},
- :symbol => '1;32',
- :tag => '1;34',
- :type => '1;34',
- :value => '36',
- :variable => '1;34',
+ :tag => "\e[32m",
+ :type => "\e[1;34m",
+ :value => "\e[36m",
+ :variable => "\e[34m",
- :insert => '42',
- :delete => '41',
- :change => '44',
- :head => '45'
+ :insert => {
+ :self => "\e[42m",
+ :insert => "\e[1;32;42m",
+ :eyecatcher => "\e[102m",
+ },
+ :delete => {
+ :self => "\e[41m",
+ :delete => "\e[1;31;41m",
+ :eyecatcher => "\e[101m",
+ },
+ :change => {
+ :self => "\e[44m",
+ :change => "\e[37;44m",
+ },
+ :head => {
+ :self => "\e[45m",
+ :filename => "\e[37;45m"
+ },
}
+
TOKEN_COLORS[:keyword] = TOKEN_COLORS[:reserved]
TOKEN_COLORS[:method] = TOKEN_COLORS[:function]
- TOKEN_COLORS[:imaginary] = TOKEN_COLORS[:complex]
- TOKEN_COLORS[:begin_group] = TOKEN_COLORS[:end_group] =
- TOKEN_COLORS[:escape] = TOKEN_COLORS[:delimiter]
+ TOKEN_COLORS[:escape] = TOKEN_COLORS[:delimiter]
protected
def setup(options)
super
@opened = []
- @subcolors = nil
+ @color_scopes = [TOKEN_COLORS]
end
public
def text_token text, kind
- if color = (@subcolors || TOKEN_COLORS)[kind]
- if Hash === color
- if color[:self]
- color = color[:self]
- else
- @out << text
- return
- end
- end
+ if color = @color_scopes.last[kind]
+ color = color[:self] if color.is_a? Hash
- @out << ansi_colorize(color)
- @out << text.gsub("\n", ansi_clear + "\n" + ansi_colorize(color))
- @out << ansi_clear
- @out << ansi_colorize(@subcolors[:self]) if @subcolors && @subcolors[:self]
+ @out << color
+ @out << (text.index("\n") ? text.gsub("\n", "\e[0m\n" + color) : text)
+ @out << "\e[0m"
+ if outer_color = @color_scopes.last[:self]
+ @out << outer_color
+ end
else
@out << text
end
@@ -130,50 +160,36 @@ module CodeRay
alias begin_line begin_group
def end_group kind
- if @opened.empty?
- # nothing to close
- else
- @opened.pop
- @out << ansi_clear
- @out << open_token(@opened.last)
+ if @opened.pop
+ @color_scopes.pop
+ @out << "\e[0m"
+ if outer_color = @color_scopes.last[:self]
+ @out << outer_color
+ end
end
end
def end_line kind
- if @opened.empty?
- # nothing to close
- else
- @opened.pop
- # whole lines to be highlighted,
- # eg. added/modified/deleted lines in a diff
- @out << "\t" * 100 + ansi_clear
- @out << open_token(@opened.last)
- end
+ @out << (@line_filler ||= "\t" * 100)
+ end_group kind
end
private
def open_token kind
- if color = TOKEN_COLORS[kind]
- if Hash === color
- @subcolors = color
- ansi_colorize(color[:self]) if color[:self]
+ if color = @color_scopes.last[kind]
+ if color.is_a? Hash
+ @color_scopes << color
+ color[:self]
else
- @subcolors = {}
- ansi_colorize(color)
+ @color_scopes << @color_scopes.last
+ color
end
else
- @subcolors = nil
+ @color_scopes << @color_scopes.last
''
end
end
-
- def ansi_colorize(color)
- Array(color).map { |c| "\e[#{c}m" }.join
- end
- def ansi_clear
- ansi_colorize(0)
- end
end
end
-end
\ No newline at end of file
+end
diff --git a/lib/coderay/helpers/file_type.rb b/lib/coderay/helpers/file_type.rb
index 637001b..7de34d5 100644
--- a/lib/coderay/helpers/file_type.rb
+++ b/lib/coderay/helpers/file_type.rb
@@ -38,7 +38,7 @@ module CodeRay
(TypeFromExt[ext2.downcase] if ext2) ||
TypeFromName[name] ||
TypeFromName[name.downcase]
- type ||= shebang(filename) if read_shebang
+ type ||= type_from_shebang(filename) if read_shebang
type
end
@@ -63,7 +63,7 @@ module CodeRay
protected
- def shebang filename
+ def type_from_shebang filename
return unless File.exist? filename
File.open filename, 'r' do |f|
if first_line = f.gets
@@ -77,53 +77,58 @@ module CodeRay
end
TypeFromExt = {
- 'c' => :c,
- 'cfc' => :xml,
- 'cfm' => :xml,
- 'clj' => :clojure,
- 'css' => :css,
- 'diff' => :diff,
- 'dpr' => :delphi,
- 'erb' => :erb,
- 'gemspec' => :ruby,
- 'groovy' => :groovy,
- 'gvy' => :groovy,
- 'h' => :c,
- 'haml' => :haml,
- 'htm' => :html,
- 'html' => :html,
- 'html.erb' => :erb,
- 'java' => :java,
- 'js' => :java_script,
- 'json' => :json,
- 'mab' => :ruby,
- 'pas' => :delphi,
- 'patch' => :diff,
- 'php' => :php,
- 'php3' => :php,
- 'php4' => :php,
- 'php5' => :php,
- 'prawn' => :ruby,
- 'py' => :python,
- 'py3' => :python,
- 'pyw' => :python,
- 'rake' => :ruby,
- 'raydebug' => :raydebug,
- 'rb' => :ruby,
- 'rbw' => :ruby,
- 'rhtml' => :erb,
- 'rjs' => :ruby,
- 'rpdf' => :ruby,
- 'ru' => :ruby,
- 'rxml' => :ruby,
- # 'sch' => :scheme,
- 'sql' => :sql,
- # 'ss' => :scheme,
- 'tmproj' => :xml,
- 'xhtml' => :html,
- 'xml' => :xml,
- 'yaml' => :yaml,
- 'yml' => :yaml,
+ 'c' => :c,
+ 'cfc' => :xml,
+ 'cfm' => :xml,
+ 'clj' => :clojure,
+ 'css' => :css,
+ 'diff' => :diff,
+ 'dpr' => :delphi,
+ 'erb' => :erb,
+ 'gemspec' => :ruby,
+ 'go' => :go,
+ 'groovy' => :groovy,
+ 'gvy' => :groovy,
+ 'h' => :c,
+ 'haml' => :haml,
+ 'htm' => :html,
+ 'html' => :html,
+ 'html.erb' => :erb,
+ 'java' => :java,
+ 'js' => :java_script,
+ 'json' => :json,
+ 'lua' => :lua,
+ 'mab' => :ruby,
+ 'pas' => :delphi,
+ 'patch' => :diff,
+ 'phtml' => :php,
+ 'php' => :php,
+ 'php3' => :php,
+ 'php4' => :php,
+ 'php5' => :php,
+ 'prawn' => :ruby,
+ 'py' => :python,
+ 'py3' => :python,
+ 'pyw' => :python,
+ 'rake' => :ruby,
+ 'raydebug' => :raydebug,
+ 'rb' => :ruby,
+ 'rbw' => :ruby,
+ 'rhtml' => :erb,
+ 'rjs' => :ruby,
+ 'rpdf' => :ruby,
+ 'ru' => :ruby, # config.ru
+ 'rxml' => :ruby,
+ 'sass' => :sass,
+ 'sql' => :sql,
+ 'taskpaper' => :taskpaper,
+ 'template' => :json, # AWS CloudFormation template
+ 'tmproj' => :xml,
+ 'xaml' => :xml,
+ 'xhtml' => :html,
+ 'xml' => :xml,
+ 'yaml' => :yaml,
+ 'yml' => :yaml,
}
for cpp_alias in %w[cc cpp cp cxx c++ C hh hpp h++ cu]
TypeFromExt[cpp_alias] = :cpp
@@ -136,6 +141,9 @@ module CodeRay
'Rakefile' => :ruby,
'Rantfile' => :ruby,
'Gemfile' => :ruby,
+ 'Guardfile' => :ruby,
+ 'Vagrantfile' => :ruby,
+ 'Appraisals' => :ruby
}
end
diff --git a/lib/coderay/helpers/gzip.rb b/lib/coderay/helpers/gzip.rb
deleted file mode 100644
index 245014a..0000000
--- a/lib/coderay/helpers/gzip.rb
+++ /dev/null
@@ -1,41 +0,0 @@
-module CodeRay
-
- # A simplified interface to the gzip library +zlib+ (from the Ruby Standard Library.)
- module GZip
-
- require 'zlib'
-
- # The default zipping level. 7 zips good and fast.
- DEFAULT_GZIP_LEVEL = 7
-
- # Unzips the given string +s+.
- #
- # Example:
- # require 'gzip_simple'
- # print GZip.gunzip(File.read('adresses.gz'))
- def GZip.gunzip s
- Zlib::Inflate.inflate s
- end
-
- # Zips the given string +s+.
- #
- # Example:
- # require 'gzip_simple'
- # File.open('adresses.gz', 'w') do |file
- # file.write GZip.gzip('Mum: 0123 456 789', 9)
- # end
- #
- # If you provide a +level+, you can control how strong
- # the string is compressed:
- # - 0: no compression, only convert to gzip format
- # - 1: compress fast
- # - 7: compress more, but still fast (default)
- # - 8: compress more, slower
- # - 9: compress best, very slow
- def GZip.gzip s, level = DEFAULT_GZIP_LEVEL
- Zlib::Deflate.new(level).deflate s, Zlib::FINISH
- end
-
- end
-
-end
diff --git a/lib/coderay/helpers/plugin.rb b/lib/coderay/helpers/plugin.rb
index 06c1233..9a724ff 100644
--- a/lib/coderay/helpers/plugin.rb
+++ b/lib/coderay/helpers/plugin.rb
@@ -30,7 +30,7 @@ module CodeRay
# * a file could not be found
# * the requested Plugin is not registered
PluginNotFound = Class.new LoadError
- HostNotFound = Class.new LoadError
+ HostNotFound = Class.new LoadError
PLUGIN_HOSTS = []
PLUGIN_HOSTS_BY_ID = {} # dummy hash
@@ -49,8 +49,8 @@ module CodeRay
def [] id, *args, &blk
plugin = validate_id(id)
begin
- plugin = plugin_hash.[] plugin, *args, &blk
- end while plugin.is_a? Symbol
+ plugin = plugin_hash.[](plugin, *args, &blk)
+ end while plugin.is_a? String
plugin
end
@@ -95,7 +95,7 @@ module CodeRay
def map hash
for from, to in hash
from = validate_id from
- to = validate_id to
+ to = validate_id to
plugin_hash[from] = to unless plugin_hash.has_key? from
end
end
@@ -131,7 +131,7 @@ module CodeRay
# A Hash of plugion_id => Plugin pairs.
def plugin_hash
- @plugin_hash ||= make_plugin_hash
+ @plugin_hash ||= (@plugin_hash = make_plugin_hash).tap { load_plugin_map }
end
# Returns an array of all .rb files in the plugin path.
@@ -158,7 +158,6 @@ module CodeRay
# This is done automatically when plugin_path is called.
def load_plugin_map
mapfile = path_to '_map'
- @plugin_map_loaded = true
if File.exist? mapfile
require mapfile
true
@@ -171,24 +170,16 @@ module CodeRay
# Return a plugin hash that automatically loads plugins.
def make_plugin_hash
- @plugin_map_loaded ||= false
Hash.new do |h, plugin_id|
id = validate_id(plugin_id)
path = path_to id
begin
- raise LoadError, "#{path} not found" unless File.exist? path
require path
rescue LoadError => boom
- if @plugin_map_loaded
- if h.has_key?(:default)
- warn '%p could not load plugin %p; falling back to %p' % [self, id, h[:default]]
- h[:default]
- else
- raise PluginNotFound, '%p could not load plugin %p: %s' % [self, id, boom]
- end
+ if h.has_key?(:default)
+ h[:default]
else
- load_plugin_map
- h[plugin_id]
+ raise PluginNotFound, '%p could not load plugin %p: %s' % [self, id, boom]
end
else
# Plugin should have registered by now
@@ -206,22 +197,22 @@ module CodeRay
File.join plugin_path, "#{plugin_id}.rb"
end
- # Converts +id+ to a Symbol if it is a String,
- # or returns +id+ if it already is a Symbol.
+ # Converts +id+ to a valid plugin ID String, or returns +nil+.
#
# Raises +ArgumentError+ for all other objects, or if the
# given String includes non-alphanumeric characters (\W).
def validate_id id
- if id.is_a? Symbol or id.nil?
- id
- elsif id.is_a? String
+ case id
+ when Symbol
+ id.to_s
+ when String
if id[/\w+/] == id
- id.downcase.to_sym
+ id.downcase
else
raise ArgumentError, "Invalid id given: #{id}"
end
else
- raise ArgumentError, "String or Symbol expected, but #{id.class} given."
+ raise ArgumentError, "Symbol or String expected, but #{id.class} given."
end
end
@@ -272,7 +263,6 @@ module CodeRay
end
def aliases
- plugin_host.load_plugin_map
plugin_host.plugin_hash.inject [] do |aliases, (key, _)|
aliases << key if plugin_host[key] == self
aliases
diff --git a/lib/coderay/helpers/word_list.rb b/lib/coderay/helpers/word_list.rb
index ea969c3..4a42c4a 100644
--- a/lib/coderay/helpers/word_list.rb
+++ b/lib/coderay/helpers/word_list.rb
@@ -4,11 +4,6 @@ module CodeRay
#
# <b>A Hash subclass designed for mapping word lists to token types.</b>
#
- # Copyright (c) 2006-2011 by murphy (Kornelius Kalnbach) <murphy rubychan de>
- #
- # License:: LGPL / ask the author
- # Version:: 2.0 (2011-05-08)
- #
# A WordList is a Hash with some additional features.
# It is intended to be used for keyword recognition.
#
diff --git a/lib/coderay/scanner.rb b/lib/coderay/scanner.rb
index 907cf00..b3f7e17 100644
--- a/lib/coderay/scanner.rb
+++ b/lib/coderay/scanner.rb
@@ -182,16 +182,9 @@ module CodeRay
# Scan the code and returns all tokens in a Tokens object.
def tokenize source = nil, options = {}
options = @options.merge(options)
- @tokens = options[:tokens] || @tokens || Tokens.new
- @tokens.scanner = self if @tokens.respond_to? :scanner=
- case source
- when Array
- self.string = self.class.normalize(source.join)
- when nil
- reset
- else
- self.string = self.class.normalize(source)
- end
+
+ set_tokens_from_options options
+ set_string_from_source source
begin
scan_tokens @tokens, options
@@ -261,6 +254,22 @@ module CodeRay
def setup # :doc:
end
+ def set_string_from_source source
+ case source
+ when Array
+ self.string = self.class.normalize(source.join)
+ when nil
+ reset
+ else
+ self.string = self.class.normalize(source)
+ end
+ end
+
+ def set_tokens_from_options options
+ @tokens = options[:tokens] || @tokens || Tokens.new
+ @tokens.scanner = self if @tokens.respond_to? :scanner=
+ end
+
# This is the central method, and commonly the only one a
# subclass implements.
#
@@ -277,19 +286,15 @@ module CodeRay
@binary_string = nil if defined? @binary_string
end
- # Scanner error with additional status information
- def raise_inspect msg, tokens, state = self.state || 'No state given!', ambit = 30, backtrace = caller
- raise ScanError, <<-EOE % [
+ SCAN_ERROR_MESSAGE = <<-MESSAGE
-***ERROR in %s: %s (after %d tokens)
+***ERROR in %s: %s (after %s tokens)
tokens:
%s
-current line: %d column: %d pos: %d
-matched: %p state: %p
-bol? = %p, eos? = %p
+%s
surrounding code:
%p ~~ %p
@@ -297,16 +302,43 @@ surrounding code:
***ERROR***
- EOE
- File.basename(caller[0]),
- msg,
- tokens.respond_to?(:size) ? tokens.size : 0,
- tokens.respond_to?(:last) ? tokens.last(10).map { |t| t.inspect }.join("\n") : '',
- line, column, pos,
- matched, state, bol?, eos?,
+ MESSAGE
+
+ def raise_inspect_arguments message, tokens, state, ambit
+ return File.basename(caller[0]),
+ message,
+ tokens_size(tokens),
+ tokens_last(tokens, 10).map(&:inspect).join("\n"),
+ scanner_state_info(state),
binary_string[pos - ambit, ambit],
- binary_string[pos, ambit],
- ], backtrace
+ binary_string[pos, ambit]
+ end
+
+ SCANNER_STATE_INFO = <<-INFO
+current line: %d column: %d pos: %d
+matched: %p state: %p
+bol?: %p, eos?: %p
+ INFO
+
+ def scanner_state_info state
+ SCANNER_STATE_INFO % [
+ line, column, pos,
+ matched, state || 'No state given!',
+ bol?, eos?,
+ ]
+ end
+
+ # Scanner error with additional status information
+ def raise_inspect message, tokens, state = self.state, ambit = 30, backtrace = caller
+ raise ScanError, SCAN_ERROR_MESSAGE % raise_inspect_arguments(message, tokens, state, ambit), backtrace
+ end
+
+ def tokens_size tokens
+ tokens.size if tokens.respond_to?(:size)
+ end
+
+ def tokens_last tokens, n
+ tokens.respond_to?(:last) ? tokens.last(n) : []
end
# Shorthand for scan_until(/\z/).
diff --git a/lib/coderay/scanners/c.rb b/lib/coderay/scanners/c.rb
index 8d24b99..84b6e8e 100644
--- a/lib/coderay/scanners/c.rb
+++ b/lib/coderay/scanners/c.rb
@@ -148,7 +148,7 @@ module Scanners
encoder.text_token match, :char
elsif match = scan(/ \\ | $ /x)
encoder.end_group :string
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
state = :initial
label_expected = false
else
diff --git a/lib/coderay/scanners/cpp.rb b/lib/coderay/scanners/cpp.rb
index 9da62f4..e61f56f 100644
--- a/lib/coderay/scanners/cpp.rb
+++ b/lib/coderay/scanners/cpp.rb
@@ -160,7 +160,7 @@ module Scanners
encoder.text_token match, :char
elsif match = scan(/ \\ | $ /x)
encoder.end_group :string
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
state = :initial
label_expected = false
else
diff --git a/lib/coderay/scanners/css.rb b/lib/coderay/scanners/css.rb
index 7b731ef..55d5239 100644
--- a/lib/coderay/scanners/css.rb
+++ b/lib/coderay/scanners/css.rb
@@ -7,27 +7,25 @@ module Scanners
KINDS_NOT_LOC = [
:comment,
- :class, :pseudo_class, :type,
- :constant, :directive,
+ :class, :pseudo_class, :tag,
+ :id, :directive,
:key, :value, :operator, :color, :float, :string,
- :error, :important,
+ :error, :important, :type,
] # :nodoc:
module RE # :nodoc:
Hex = /[0-9a-fA-F]/
- Unicode = /\\#{Hex}{1,6}(?:\r\n|\s)?/ # differs from standard because it allows uppercase hex too
- Escape = /#{Unicode}|\\[^\r\n\f0-9a-fA-F]/
- NMChar = /[-_a-zA-Z0-9]|#{Escape}/
- NMStart = /[_a-zA-Z]|#{Escape}/
- NL = /\r\n|\r|\n|\f/
- String1 = /"(?:[^\n\r\f\\"]|\\#{NL}|#{Escape})*"?/ # TODO: buggy regexp
- String2 = /'(?:[^\n\r\f\\']|\\#{NL}|#{Escape})*'?/ # TODO: buggy regexp
+ Unicode = /\\#{Hex}{1,6}\b/ # differs from standard because it allows uppercase hex too
+ Escape = /#{Unicode}|\\[^\n0-9a-fA-F]/
+ NMChar = /[-_a-zA-Z0-9]/
+ NMStart = /[_a-zA-Z]/
+ String1 = /"(?:[^\n\\"]+|\\\n|#{Escape})*"?/ # TODO: buggy regexp
+ String2 = /'(?:[^\n\\']+|\\\n|#{Escape})*'?/ # TODO: buggy regexp
String = /#{String1}|#{String2}/
HexColor = /#(?:#{Hex}{6}|#{Hex}{3})/
- Color = /#{HexColor}/
- Num = /-?(?:[0-9]+|[0-9]*\.[0-9]+)/
+ Num = /-?(?:[0-9]*\.[0-9]+|[0-9]+)n?/
Name = /#{NMChar}+/
Ident = /-?#{NMStart}#{NMChar}*/
AtKeyword = /@#{Ident}/
@@ -35,16 +33,15 @@ module Scanners
reldimensions = %w[em ex px]
absdimensions = %w[in cm mm pt pc]
- Unit = Regexp.union(*(reldimensions + absdimensions + %w[s]))
+ Unit = Regexp.union(*(reldimensions + absdimensions + %w[s dpi dppx deg]))
Dimension = /#{Num}#{Unit}/
- Comment = %r! /\* (?: .*? \*/ | .* ) !mx
- Function = /(?:url|alpha|attr|counters?)\((?:[^)\n\r\f]|\\\))*\)?/
+ Function = /(?:url|alpha|attr|counters?)\((?:[^)\n]|\\\))*\)?/
- Id = /##{Name}/
+ Id = /(?!#{HexColor}\b(?!-))##{Name}/
Class = /\.#{Name}/
- PseudoClass = /:#{Name}/
+ PseudoClass = /::?#{Ident}/
AttributeSelector = /\[[^\]]*\]?/
end
@@ -52,11 +49,11 @@ module Scanners
def setup
@state = :initial
- @value_expected = nil
+ @value_expected = false
end
def scan_tokens encoder, options
- states = Array(options[:state] || @state)
+ states = Array(options[:state] || @state).dup
value_expected = @value_expected
until eos?
@@ -67,13 +64,13 @@ module Scanners
elsif case states.last
when :initial, :media
if match = scan(/(?>#{RE::Ident})(?!\()|\*/ox)
- encoder.text_token match, :type
+ encoder.text_token match, :tag
next
elsif match = scan(RE::Class)
encoder.text_token match, :class
next
elsif match = scan(RE::Id)
- encoder.text_token match, :constant
+ encoder.text_token match, :id
next
elsif match = scan(RE::PseudoClass)
encoder.text_token match, :pseudo_class
@@ -148,17 +145,17 @@ module Scanners
start = match[/^\w+\(/]
encoder.text_token start, :delimiter
if match[-1] == ?)
- encoder.text_token match[start.size..-2], :content
+ encoder.text_token match[start.size..-2], :content if match.size > start.size + 1
encoder.text_token ')', :delimiter
else
- encoder.text_token match[start.size..-1], :content
+ encoder.text_token match[start.size..-1], :content if match.size > start.size
end
encoder.end_group :function
elsif match = scan(/(?: #{RE::Dimension} | #{RE::Percentage} | #{RE::Num} )/ox)
encoder.text_token match, :float
- elsif match = scan(/#{RE::Color}/o)
+ elsif match = scan(/#{RE::HexColor}/o)
encoder.text_token match, :color
elsif match = scan(/! *important/)
@@ -170,7 +167,7 @@ module Scanners
elsif match = scan(RE::AtKeyword)
encoder.text_token match, :directive
- elsif match = scan(/ [+>:;,.=()\/] /x)
+ elsif match = scan(/ [+>~:;,.=()\/] /x)
if match == ':'
value_expected = true
elsif match == ';'
diff --git a/lib/coderay/scanners/debug.rb b/lib/coderay/scanners/debug.rb
index 566bfa7..83ede9a 100644
--- a/lib/coderay/scanners/debug.rb
+++ b/lib/coderay/scanners/debug.rb
@@ -1,9 +1,11 @@
+require 'set'
+
module CodeRay
module Scanners
# = Debug Scanner
#
- # Interprets the output of the Encoders::Debug encoder.
+ # Interprets the output of the Encoders::Debug encoder (basically the inverse function).
class Debug < Scanner
register_for :debug
@@ -11,6 +13,11 @@ module Scanners
protected
+ def setup
+ super
+ @known_token_kinds = TokenKinds.keys.map(&:to_s).to_set
+ end
+
def scan_tokens encoder, options
opened_tokens = []
@@ -21,16 +28,19 @@ module Scanners
encoder.text_token match, :space
elsif match = scan(/ (\w+) \( ( [^\)\\]* ( \\. [^\)\\]* )* ) \)? /x)
- kind = self[1].to_sym
- match = self[2].gsub(/\\(.)/m, '\1')
- unless TokenKinds.has_key? kind
- kind = :error
- match = matched
+ if @known_token_kinds.include? self[1]
+ encoder.text_token self[2].gsub(/\\(.)/m, '\1'), self[1].to_sym
+ else
+ encoder.text_token matched, :unknown
end
- encoder.text_token match, kind
elsif match = scan(/ (\w+) ([<\[]) /x)
- kind = self[1].to_sym
+ if @known_token_kinds.include? self[1]
+ kind = self[1].to_sym
+ else
+ kind = :unknown
+ end
+
opened_tokens << kind
case self[2]
when '<'
diff --git a/lib/coderay/scanners/diff.rb b/lib/coderay/scanners/diff.rb
index 9e899c3..fd1aed6 100644
--- a/lib/coderay/scanners/diff.rb
+++ b/lib/coderay/scanners/diff.rb
@@ -20,7 +20,7 @@ module Scanners
line_kind = nil
state = :initial
- deleted_lines = 0
+ deleted_lines_count = 0
scanners = Hash.new do |h, lang|
h[lang] = Scanners[lang].new '', :keep_tokens => true, :keep_state => true
end
@@ -30,7 +30,7 @@ module Scanners
until eos?
if match = scan(/\n/)
- deleted_lines = 0 unless line_kind == :delete
+ deleted_lines_count = 0 unless line_kind == :delete
if line_kind
encoder.end_line line_kind
line_kind = nil
@@ -45,7 +45,7 @@ module Scanners
if match = scan(/--- |\+\+\+ |=+|_+/)
encoder.begin_line line_kind = :head
encoder.text_token match, :head
- if match = scan(/.*?(?=$|[\t\n\x00]| \(revision)/)
+ if match = scan(/[^\x00\n]+?(?=$|[\t\n]| \(revision)/)
encoder.text_token match, :filename
if options[:highlight_code] && match != '/dev/null'
file_type = CodeRay::FileType.fetch(match, :text)
@@ -69,7 +69,7 @@ module Scanners
state = :added
elsif match = scan(/\\ .*/)
encoder.text_token match, :comment
- elsif match = scan(/@@(?>[^@\n]*)@@/)
+ elsif match = scan(/@@(?>[^@\n]+)@@/)
content_scanner.state = :initial unless match?(/\n\+/)
content_scanner_entry_state = nil
if check(/\n|$/)
@@ -99,37 +99,59 @@ module Scanners
end
next
elsif match = scan(/-/)
- deleted_lines += 1
- encoder.begin_line line_kind = :delete
- encoder.text_token match, :delete
- if options[:inline_diff] && deleted_lines == 1 && check(/(?>.*)\n\+(?>.*)$(?!\n\+)/)
- content_scanner_entry_state = content_scanner.state
- skip(/(.*)\n\+(.*)$/)
- head, deletion, insertion, tail = diff self[1], self[2]
- pre, deleted, post = content_scanner.tokenize [head, deletion, tail], :tokens => Tokens.new
- encoder.tokens pre
- unless deleted.empty?
- encoder.begin_group :eyecatcher
- encoder.tokens deleted
- encoder.end_group :eyecatcher
+ deleted_lines_count += 1
+ if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
+ deleted_lines = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
+ inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/) ; scan(/.*/) }
+
+ deleted_lines_tokenized = []
+ inserted_lines_tokenized = []
+ for deleted_line, inserted_line in deleted_lines.zip(inserted_lines)
+ pre, deleted_part, inserted_part, post = diff deleted_line, inserted_line
+ content_scanner_entry_state = content_scanner.state
+ deleted_lines_tokenized << content_scanner.tokenize([pre, deleted_part, post], :tokens => Tokens.new)
+ content_scanner.state = content_scanner_entry_state || :initial
+ inserted_lines_tokenized << content_scanner.tokenize([pre, inserted_part, post], :tokens => Tokens.new)
end
- encoder.tokens post
- encoder.end_line line_kind
- encoder.text_token "\n", :space
- encoder.begin_line line_kind = :insert
- encoder.text_token '+', :insert
- content_scanner.state = content_scanner_entry_state || :initial
- pre, inserted, post = content_scanner.tokenize [head, insertion, tail], :tokens => Tokens.new
- encoder.tokens pre
- unless inserted.empty?
- encoder.begin_group :eyecatcher
- encoder.tokens inserted
- encoder.end_group :eyecatcher
+
+ for pre, deleted_part, post in deleted_lines_tokenized
+ encoder.begin_line :delete
+ encoder.text_token '-', :delete
+ encoder.tokens pre
+ unless deleted_part.empty?
+ encoder.begin_group :eyecatcher
+ encoder.tokens deleted_part
+ encoder.end_group :eyecatcher
+ end
+ encoder.tokens post
+ encoder.end_line :delete
+ encoder.text_token "\n", :space
+ end
+
+ for pre, inserted_part, post in inserted_lines_tokenized
+ encoder.begin_line :insert
+ encoder.text_token '+', :insert
+ encoder.tokens pre
+ unless inserted_part.empty?
+ encoder.begin_group :eyecatcher
+ encoder.tokens inserted_part
+ encoder.end_group :eyecatcher
+ end
+ encoder.tokens post
+ changed_lines_count -= 1
+ if changed_lines_count > 0
+ encoder.end_line :insert
+ encoder.text_token "\n", :space
+ end
end
- encoder.tokens post
+
+ line_kind = :insert
+
elsif match = scan(/.*/)
+ encoder.begin_line line_kind = :delete
+ encoder.text_token '-', :delete
if options[:highlight_code]
- if deleted_lines == 1
+ if deleted_lines_count == 1
content_scanner_entry_state = content_scanner.state
end
content_scanner.tokenize match, :tokens => encoder unless match.empty?
diff --git a/lib/coderay/scanners/c.rb b/lib/coderay/scanners/go.rb
similarity index 63%
copy from lib/coderay/scanners/c.rb
copy to lib/coderay/scanners/go.rb
index 8d24b99..99fdd63 100644
--- a/lib/coderay/scanners/c.rb
+++ b/lib/coderay/scanners/go.rb
@@ -1,82 +1,89 @@
module CodeRay
module Scanners
- # Scanner for C.
- class C < Scanner
-
- register_for :c
- file_extension 'c'
+ class Go < Scanner
+ register_for :go
+ file_extension 'go'
+
+ # http://golang.org/ref/spec#Keywords
KEYWORDS = [
- 'asm', 'break', 'case', 'continue', 'default', 'do',
- 'else', 'enum', 'for', 'goto', 'if', 'return',
- 'sizeof', 'struct', 'switch', 'typedef', 'union', 'while',
- 'restrict', # added in C99
+ 'break', 'default', 'func', 'interface', 'select',
+ 'case', 'defer', 'go', 'map', 'struct',
+ 'chan', 'else', 'goto', 'package', 'switch',
+ 'const', 'fallthrough', 'if', 'range', 'type',
+ 'continue', 'for', 'import', 'return', 'var',
] # :nodoc:
-
+
+ # http://golang.org/ref/spec#Types
PREDEFINED_TYPES = [
- 'int', 'long', 'short', 'char',
- 'signed', 'unsigned', 'float', 'double',
- 'bool', 'complex', # added in C99
+ 'bool',
+ 'uint8', 'uint16', 'uint32', 'uint64',
+ 'int8', 'int16', 'int32', 'int64',
+ 'float32', 'float64',
+ 'complex64', 'complex128',
+ 'byte', 'rune', 'string', 'error',
+ 'uint', 'int', 'uintptr',
] # :nodoc:
-
+
PREDEFINED_CONSTANTS = [
- 'EOF', 'NULL',
- 'true', 'false', # added in C99
+ 'nil', 'iota',
+ 'true', 'false',
] # :nodoc:
- DIRECTIVES = [
- 'auto', 'extern', 'register', 'static', 'void',
- 'const', 'volatile', # added in C89
- 'inline', # added in C99
- ] # :nodoc:
-
+
+ PREDEFINED_FUNCTIONS = %w[
+ append cap close complex copy delete imag len
+ make new panic print println real recover
+ ] # :nodoc:
+
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_TYPES, :predefined_type).
- add(DIRECTIVES, :directive).
- add(PREDEFINED_CONSTANTS, :predefined_constant) # :nodoc:
-
+ add(PREDEFINED_CONSTANTS, :predefined_constant).
+ add(PREDEFINED_FUNCTIONS, :predefined) # :nodoc:
+
ESCAPE = / [rbfntv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
- UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
+ UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
- protected
+ protected
def scan_tokens encoder, options
-
+
state = :initial
label_expected = true
case_expected = false
label_expected_before_preproc_line = nil
in_preproc_line = false
-
+
until eos?
-
+
case state
-
+
when :initial
-
+
if match = scan(/ \s+ | \\\n /x)
if in_preproc_line && match != "\\\n" && match.index(?\n)
in_preproc_line = false
+ case_expected = false
label_expected = label_expected_before_preproc_line
end
encoder.text_token match, :space
-
+
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
encoder.text_token match, :comment
-
- elsif match = scan(/ [-+*=<>?:;,!&^|()\[\]{}~%]+ | \/=? | \.(?!\d) /x)
- label_expected = match =~ /[;\{\}]/
+
+ elsif match = scan(/ <?- (?![\d.]) | [+*=<>?:;,!&^|()\[\]{}~%]+ | \/=? | \.(?!\d) /x)
if case_expected
label_expected = true if match == ':'
case_expected = false
end
encoder.text_token match, :operator
-
+
elsif match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
kind = IDENT_KIND[match]
if kind == :ident && label_expected && !in_preproc_line && scan(/:(?!:)/)
kind = :label
+ label_expected = false
match << matched
else
label_expected = false
@@ -88,7 +95,7 @@ module Scanners
end
end
encoder.text_token match, kind
-
+
elsif match = scan(/L?"/)
encoder.begin_group :string
if match[0] == ?L
@@ -97,45 +104,56 @@ module Scanners
end
encoder.text_token match, :delimiter
state = :string
-
+
+ elsif match = scan(/ ` ([^`]+)? (`)? /x)
+ encoder.begin_group :shell
+ encoder.text_token '`', :delimiter
+ encoder.text_token self[1], :content if self[1]
+ encoder.text_token self[2], :delimiter if self[2]
+ encoder.end_group :shell
+
elsif match = scan(/ \# \s* if \s* 0 /x)
match << scan_until(/ ^\# (?:elif|else|endif) .*? $ | \z /xm) unless eos?
encoder.text_token match, :comment
-
+
elsif match = scan(/#[ \t]*(\w*)/)
encoder.text_token match, :preprocessor
in_preproc_line = true
label_expected_before_preproc_line = label_expected
state = :include_expected if self[1] == 'include'
-
- elsif match = scan(/ L?' (?: [^\'\n\\] | \\ #{ESCAPE} )? '? /ox)
+
+ elsif match = scan(/ L?' (?: [^\'\n\\] | \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) )? '? /ox)
label_expected = false
encoder.text_token match, :char
-
+
elsif match = scan(/\$/)
encoder.text_token match, :ident
- elsif match = scan(/0[xX][0-9A-Fa-f]+/)
+ elsif match = scan(/-?\d*(\.\d*)?([eE][+-]?\d+)?i/)
+ label_expected = false
+ encoder.text_token match, :imaginary
+
+ elsif match = scan(/-?0[xX][0-9A-Fa-f]+/)
label_expected = false
encoder.text_token match, :hex
-
- elsif match = scan(/(?:0[0-7]+)(?![89.eEfF])/)
+
+ elsif match = scan(/-?(?:0[0-7]+)(?![89.eEfF])/)
label_expected = false
encoder.text_token match, :octal
-
- elsif match = scan(/(?:\d+)(?![.eEfF])L?L?/)
- label_expected = false
- encoder.text_token match, :integer
-
- elsif match = scan(/\d[fF]?|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
+
+ elsif match = scan(/-?(?:\d*\.\d+|\d+\.)(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+/)
label_expected = false
encoder.text_token match, :float
-
+
+ elsif match = scan(/-?(?:\d+)(?![.eEfF])L?L?/)
+ label_expected = false
+ encoder.text_token match, :integer
+
else
encoder.text_token getch, :error
-
+
end
-
+
when :string
if match = scan(/[^\\\n"]+/)
encoder.text_token match, :content
@@ -146,44 +164,45 @@ module Scanners
label_expected = false
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
- elsif match = scan(/ \\ | $ /x)
- encoder.end_group :string
+ elsif match = scan(/ \\ /x)
encoder.text_token match, :error
+ elsif match = scan(/$/)
+ encoder.end_group :string
state = :initial
label_expected = false
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
-
+
when :include_expected
if match = scan(/<[^>\n]+>?|"[^"\n\\]*(?:\\.[^"\n\\]*)*"?/)
encoder.text_token match, :include
state = :initial
-
+
elsif match = scan(/\s+/)
encoder.text_token match, :space
state = :initial if match.index ?\n
-
+
else
state = :initial
-
+
end
-
+
else
raise_inspect 'Unknown state', encoder
-
+
end
-
+
end
-
+
if state == :string
encoder.end_group :string
end
-
+
encoder
end
-
+
end
-
+
end
end
diff --git a/lib/coderay/scanners/groovy.rb b/lib/coderay/scanners/groovy.rb
index cf55daf..c64454f 100644
--- a/lib/coderay/scanners/groovy.rb
+++ b/lib/coderay/scanners/groovy.rb
@@ -36,9 +36,12 @@ module Scanners
protected
+ def setup
+ @state = :initial
+ end
+
def scan_tokens encoder, options
-
- state = :initial
+ state = options[:state] || @state
inline_block_stack = []
inline_block_paren_depth = nil
string_delimiter = nil
@@ -223,7 +226,7 @@ module Scanners
encoder.text_token match, :content # TODO: Shouldn't this be :error?
elsif match = scan(/ \\ | \n /x)
- encoder.end_group state
+ encoder.end_group state == :regexp ? :regexp : :string
encoder.text_token match, :error
after_def = value_expected = false
state = :initial
@@ -243,7 +246,17 @@ module Scanners
end
if [:multiline_string, :string, :regexp].include? state
- encoder.end_group state
+ encoder.end_group state == :regexp ? :regexp : :string
+ end
+
+ if options[:keep_state]
+ @state = state
+ end
+
+ until inline_block_stack.empty?
+ state, = *inline_block_stack.pop
+ encoder.end_group :inline
+ encoder.end_group state == :regexp ? :regexp : :string
end
encoder
diff --git a/lib/coderay/scanners/html.rb b/lib/coderay/scanners/html.rb
index 98d06fc..ebe7b01 100644
--- a/lib/coderay/scanners/html.rb
+++ b/lib/coderay/scanners/html.rb
@@ -1,13 +1,13 @@
module CodeRay
module Scanners
-
+
# HTML Scanner
#
# Alias: +xhtml+
#
# See also: Scanners::XML
class HTML < Scanner
-
+
register_for :html
KINDS_NOT_LOC = [
@@ -33,7 +33,8 @@ module Scanners
)
IN_ATTRIBUTE = WordList::CaseIgnoring.new(nil).
- add(EVENT_ATTRIBUTES, :script)
+ add(EVENT_ATTRIBUTES, :script).
+ add(['style'], :style)
ATTR_NAME = /[\w.:-]+/ # :nodoc:
TAG_END = /\/?>/ # :nodoc:
@@ -69,21 +70,28 @@ module Scanners
def setup
@state = :initial
@plain_string_content = nil
+ @in_tag = nil
end
def scan_java_script encoder, code
if code && !code.empty?
@java_script_scanner ||= Scanners::JavaScript.new '', :keep_tokens => true
- # encoder.begin_group :inline
@java_script_scanner.tokenize code, :tokens => encoder
- # encoder.end_group :inline
+ end
+ end
+
+ def scan_css encoder, code, state = [:initial]
+ if code && !code.empty?
+ @css_scanner ||= Scanners::CSS.new '', :keep_tokens => true
+ @css_scanner.tokenize code, :tokens => encoder, :state => state
end
end
def scan_tokens encoder, options
state = options[:state] || @state
plain_string_content = @plain_string_content
- in_tag = in_attribute = nil
+ in_tag = @in_tag
+ in_attribute = nil
encoder.begin_group :string if state == :attribute_value_string
@@ -97,9 +105,17 @@ module Scanners
case state
when :initial
- if match = scan(/<!--(?:.*?-->|.*)/m)
+ if match = scan(/<!\[CDATA\[/)
+ encoder.text_token match, :inline_delimiter
+ if match = scan(/.*?\]\]>/m)
+ encoder.text_token match[0..-4], :plain
+ encoder.text_token ']]>', :inline_delimiter
+ elsif match = scan(/.+/)
+ encoder.text_token match, :error
+ end
+ elsif match = scan(/<!--(?:.*?-->|.*)/m)
encoder.text_token match, :comment
- elsif match = scan(/<!DOCTYPE(?:.*?>|.*)/m)
+ elsif match = scan(/<!(\w+)(?:.*?>|.*)|\]>/m)
encoder.text_token match, :doctype
elsif match = scan(/<\?xml(?:.*?\?>|.*)/m)
encoder.text_token match, :preprocessor
@@ -108,7 +124,7 @@ module Scanners
elsif match = scan(/<\/[-\w.:]*>?/m)
in_tag = nil
encoder.text_token match, :tag
- elsif match = scan(/<(?:(script)|[-\w.:]+)(>)?/m)
+ elsif match = scan(/<(?:(script|style)|[-\w.:]+)(>)?/m)
encoder.text_token match, :tag
in_tag = self[1]
if self[2]
@@ -149,12 +165,9 @@ module Scanners
if match = scan(/=/) #/
encoder.text_token match, :operator
state = :attribute_value
- elsif scan(/#{ATTR_NAME}/o) || scan(/#{TAG_END}/o)
- state = :attribute
- next
else
- encoder.text_token getch, :error
state = :attribute
+ next
end
when :attribute_value
@@ -162,17 +175,21 @@ module Scanners
encoder.text_token match, :attribute_value
state = :attribute
elsif match = scan(/["']/)
- if in_attribute == :script
- encoder.begin_group :inline
- encoder.text_token match, :inline_delimiter
+ if in_attribute == :script || in_attribute == :style
+ encoder.begin_group :string
+ encoder.text_token match, :delimiter
if scan(/javascript:[ \t]*/)
encoder.text_token matched, :comment
end
code = scan_until(match == '"' ? /(?="|\z)/ : /(?='|\z)/)
- scan_java_script encoder, code
+ if in_attribute == :script
+ scan_java_script encoder, code
+ else
+ scan_css encoder, code, [:block]
+ end
match = scan(/["']/)
- encoder.text_token match, :inline_delimiter if match
- encoder.end_group :inline
+ encoder.text_token match, :delimiter if match
+ encoder.end_group :string
state = :attribute
in_attribute = nil
else
@@ -207,19 +224,23 @@ module Scanners
when :in_special_tag
case in_tag
- when 'script'
+ when 'script', 'style'
encoder.text_token match, :space if match = scan(/[ \t]*\n/)
if scan(/(\s*<!--)(?:(.*?)(-->)|(.*))/m)
code = self[2] || self[4]
closing = self[3]
encoder.text_token self[1], :comment
else
- code = scan_until(/(?=(?:\n\s*)?<\/script>)|\z/)
+ code = scan_until(/(?=(?:\n\s*)?<\/#{in_tag}>)|\z/)
closing = false
end
unless code.empty?
encoder.begin_group :inline
- scan_java_script encoder, code
+ if in_tag == 'script'
+ scan_java_script encoder, code
+ else
+ scan_css encoder, code
+ end
encoder.end_group :inline
end
encoder.text_token closing, :comment if closing
@@ -240,6 +261,7 @@ module Scanners
if options[:keep_state]
@state = state
@plain_string_content = plain_string_content
+ @in_tag = in_tag
end
encoder.end_group :string if state == :attribute_value_string
diff --git a/lib/coderay/scanners/java.rb b/lib/coderay/scanners/java.rb
index c1490ac..b282864 100644
--- a/lib/coderay/scanners/java.rb
+++ b/lib/coderay/scanners/java.rb
@@ -147,7 +147,7 @@ module Scanners
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
state = :initial
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
diff --git a/lib/coderay/scanners/java_script.rb b/lib/coderay/scanners/java_script.rb
index 43ecb18..9eb0a0a 100644
--- a/lib/coderay/scanners/java_script.rb
+++ b/lib/coderay/scanners/java_script.rb
@@ -54,10 +54,17 @@ module Scanners
protected
+ def setup
+ @state = :initial
+ end
+
def scan_tokens encoder, options
- state = :initial
- string_delimiter = nil
+ state, string_delimiter = options[:state] || @state
+ if string_delimiter
+ encoder.begin_group state
+ end
+
value_expected = true
key_expected = false
function_expected = false
@@ -72,9 +79,10 @@ module Scanners
value_expected = true if !value_expected && match.index(?\n)
encoder.text_token match, :space
- elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
+ elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .*() ) !mx)
value_expected = true
encoder.text_token match, :comment
+ state = :open_multi_line_comment if self[1]
elsif check(/\.?\d/)
key_expected = value_expected = false
@@ -175,20 +183,36 @@ module Scanners
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
+ string_delimiter = nil
key_expected = value_expected = false
state = :initial
else
- raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
+ raise_inspect "else case #{string_delimiter} reached; %p not handled." % peek(1), encoder
end
+ when :open_multi_line_comment
+ if match = scan(%r! .*? \*/ !mx)
+ state = :initial
+ else
+ match = scan(%r! .+ !mx)
+ end
+ value_expected = true
+ encoder.text_token match, :comment if match
+
else
- raise_inspect 'Unknown state', encoder
+ #:nocov:
+ raise_inspect 'Unknown state: %p' % [state], encoder
+ #:nocov:
end
end
+ if options[:keep_state]
+ @state = state, string_delimiter
+ end
+
if [:string, :regexp].include? state
encoder.end_group state
end
diff --git a/lib/coderay/scanners/json.rb b/lib/coderay/scanners/json.rb
index 0c90c34..b09970c 100644
--- a/lib/coderay/scanners/json.rb
+++ b/lib/coderay/scanners/json.rb
@@ -14,15 +14,21 @@ module Scanners
ESCAPE = / [bfnrt\\"\/] /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x # :nodoc:
+ KEY = / (?> (?: [^\\"]+ | \\. )* ) " \s* : /x
protected
+ def setup
+ @state = :initial
+ end
+
# See http://json.org/ for a definition of the JSON lexic/grammar.
def scan_tokens encoder, options
+ state = options[:state] || @state
- state = :initial
- stack = []
- key_expected = false
+ if [:string, :key].include? state
+ encoder.begin_group state
+ end
until eos?
@@ -32,18 +38,11 @@ module Scanners
if match = scan(/ \s+ /x)
encoder.text_token match, :space
elsif match = scan(/"/)
- state = key_expected ? :key : :string
+ state = check(/#{KEY}/o) ? :key : :string
encoder.begin_group state
encoder.text_token match, :delimiter
elsif match = scan(/ [:,\[{\]}] /x)
encoder.text_token match, :operator
- case match
- when ':' then key_expected = false
- when ',' then key_expected = true if stack.last == :object
- when '{' then stack << :object; key_expected = true
- when '[' then stack << :array
- when '}', ']' then stack.pop # no error recovery, but works for valid JSON
- end
elsif match = scan(/ true | false | null /x)
encoder.text_token match, :value
elsif match = scan(/ -? (?: 0 | [1-9]\d* ) /x)
@@ -70,7 +69,7 @@ module Scanners
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
@@ -82,6 +81,10 @@ module Scanners
end
end
+ if options[:keep_state]
+ @state = state
+ end
+
if [:string, :key].include? state
encoder.end_group state
end
diff --git a/lib/coderay/scanners/lua.rb b/lib/coderay/scanners/lua.rb
new file mode 100644
index 0000000..fb1e45a
--- /dev/null
+++ b/lib/coderay/scanners/lua.rb
@@ -0,0 +1,280 @@
+# encoding: utf-8
+
+module CodeRay
+module Scanners
+
+ # Scanner for the Lua[http://lua.org] programming lanuage.
+ #
+ # The language’s complete syntax is defined in
+ # {the Lua manual}[http://www.lua.org/manual/5.2/manual.html],
+ # which is what this scanner tries to conform to.
+ class Lua < Scanner
+
+ register_for :lua
+ file_extension 'lua'
+ title 'Lua'
+
+ # Keywords used in Lua.
+ KEYWORDS = %w[and break do else elseif end
+ for function goto if in
+ local not or repeat return
+ then until while
+ ]
+
+ # Constants set by the Lua core.
+ PREDEFINED_CONSTANTS = %w[false true nil]
+
+ # The expressions contained in this array are parts of Lua’s `basic'
+ # library. Although it’s not entirely necessary to load that library,
+ # it is highly recommended and one would have to provide own implementations
+ # of some of these expressions if one does not do so. They however aren’t
+ # keywords, neither are they constants, but nearly predefined, so they
+ # get tagged as `predefined' rather than anything else.
+ #
+ # This list excludes values of form `_UPPERCASE' because the Lua manual
+ # requires such identifiers to be reserved by Lua anyway and they are
+ # highlighted directly accordingly, without the need for specific
+ # identifiers to be listed here.
+ PREDEFINED_EXPRESSIONS = %w[
+ assert collectgarbage dofile error getmetatable
+ ipairs load loadfile next pairs pcall print
+ rawequal rawget rawlen rawset select setmetatable
+ tonumber tostring type xpcall
+ ]
+
+ # Automatic token kind selection for normal words.
+ IDENT_KIND = CodeRay::WordList.new(:ident).
+ add(KEYWORDS, :keyword).
+ add(PREDEFINED_CONSTANTS, :predefined_constant).
+ add(PREDEFINED_EXPRESSIONS, :predefined)
+
+ protected
+
+ # Scanner initialization.
+ def setup
+ @state = :initial
+ @brace_depth = 0
+ end
+
+ # CodeRay entry hook. Starts parsing.
+ def scan_tokens(encoder, options)
+ state = options[:state] || @state
+ brace_depth = @brace_depth
+ num_equals = nil
+
+ until eos?
+ case state
+
+ when :initial
+ if match = scan(/\-\-\[\=*\[/) #--[[ long (possibly multiline) comment ]]
+ num_equals = match.count("=") # Number must match for comment end
+ encoder.begin_group(:comment)
+ encoder.text_token(match, :delimiter)
+ state = :long_comment
+
+ elsif match = scan(/--.*$/) # --Lua comment
+ encoder.text_token(match, :comment)
+
+ elsif match = scan(/\[=*\[/) # [[ long (possibly multiline) string ]]
+ num_equals = match.count("=") # Number must match for comment end
+ encoder.begin_group(:string)
+ encoder.text_token(match, :delimiter)
+ state = :long_string
+
+ elsif match = scan(/::\s*[a-zA-Z_][a-zA-Z0-9_]+\s*::/) # ::goto_label::
+ encoder.text_token(match, :label)
+
+ elsif match = scan(/_[A-Z]+/) # _UPPERCASE are names reserved for Lua
+ encoder.text_token(match, :predefined)
+
+ elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/) # Normal letters (or letters followed by digits)
+ kind = IDENT_KIND[match]
+
+ # Extra highlighting for entities following certain keywords
+ if kind == :keyword and match == "function"
+ state = :function_expected
+ elsif kind == :keyword and match == "goto"
+ state = :goto_label_expected
+ elsif kind == :keyword and match == "local"
+ state = :local_var_expected
+ end
+
+ encoder.text_token(match, kind)
+
+ elsif match = scan(/\{/) # Opening table brace {
+ encoder.begin_group(:map)
+ encoder.text_token(match, brace_depth >= 1 ? :inline_delimiter : :delimiter)
+ brace_depth += 1
+ state = :map
+
+ elsif match = scan(/\}/) # Closing table brace }
+ if brace_depth == 1
+ brace_depth = 0
+ encoder.text_token(match, :delimiter)
+ encoder.end_group(:map)
+ elsif brace_depth == 0 # Mismatched brace
+ encoder.text_token(match, :error)
+ else
+ brace_depth -= 1
+ encoder.text_token(match, :inline_delimiter)
+ encoder.end_group(:map)
+ state = :map
+ end
+
+ elsif match = scan(/["']/) # String delimiters " and '
+ encoder.begin_group(:string)
+ encoder.text_token(match, :delimiter)
+ start_delim = match
+ state = :string
+
+ # ↓Prefix hex number ←|→ decimal number
+ elsif match = scan(/-? (?:0x\h* \. \h+ (?:p[+\-]?\d+)? | \d*\.\d+ (?:e[+\-]?\d+)?)/ix) # hexadecimal constants have no E power, decimal ones no P power
+ encoder.text_token(match, :float)
+
+ # ↓Prefix hex number ←|→ decimal number
+ elsif match = scan(/-? (?:0x\h+ (?:p[+\-]?\d+)? | \d+ (?:e[+\-]?\d+)?)/ix) # hexadecimal constants have no E power, decimal ones no P power
+ encoder.text_token(match, :integer)
+
+ elsif match = scan(/[\+\-\*\/%^\#=~<>\(\)\[\]:;,] | \.(?!\d)/x) # Operators
+ encoder.text_token(match, :operator)
+
+ elsif match = scan(/\s+/) # Space
+ encoder.text_token(match, :space)
+
+ else # Invalid stuff. Note that Lua doesn’t accept multibyte chars outside of strings, hence these are also errors.
+ encoder.text_token(getch, :error)
+ end
+
+ # It may be that we’re scanning a full-blown subexpression of a table
+ # (tables can contain full expressions in parts).
+ # If this is the case, return to :map scanning state.
+ state = :map if state == :initial && brace_depth >= 1
+
+ when :function_expected
+ if match = scan(/\(.*?\)/m) # x = function() # "Anonymous" function without explicit name
+ encoder.text_token(match, :operator)
+ state = :initial
+ elsif match = scan(/[a-zA-Z_] (?:[a-zA-Z0-9_\.] (?!\.\d))* [\.\:]/x) # function tbl.subtbl.foo() | function tbl:foo() # Colon only allowed as last separator
+ encoder.text_token(match, :ident)
+ elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/) # function foo()
+ encoder.text_token(match, :function)
+ state = :initial
+ elsif match = scan(/\s+/) # Between the `function' keyword and the ident may be any amount of whitespace
+ encoder.text_token(match, :space)
+ else
+ encoder.text_token(getch, :error)
+ state = :initial
+ end
+
+ when :goto_label_expected
+ if match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/)
+ encoder.text_token(match, :label)
+ state = :initial
+ elsif match = scan(/\s+/) # Between the `goto' keyword and the label may be any amount of whitespace
+ encoder.text_token(match, :space)
+ else
+ encoder.text_token(getch, :error)
+ end
+
+ when :local_var_expected
+ if match = scan(/function/) # local function ...
+ encoder.text_token(match, :keyword)
+ state = :function_expected
+ elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]*/)
+ encoder.text_token(match, :local_variable)
+ elsif match = scan(/,/)
+ encoder.text_token(match, :operator)
+ elsif match = scan(/\=/)
+ encoder.text_token(match, :operator)
+ # After encountering the equal sign, arbitrary expressions are
+ # allowed again, so just return to the main state for further
+ # parsing.
+ state = :initial
+ elsif match = scan(/\n/)
+ encoder.text_token(match, :space)
+ state = :initial
+ elsif match = scan(/\s+/)
+ encoder.text_token(match, :space)
+ else
+ encoder.text_token(getch, :error)
+ end
+
+ when :long_comment
+ if match = scan(/.*?(?=\]={#{num_equals}}\])/m)
+ encoder.text_token(match, :content)
+
+ delim = scan(/\]={#{num_equals}}\]/)
+ encoder.text_token(delim, :delimiter)
+ else # No terminator found till EOF
+ encoder.text_token(rest, :error)
+ terminate
+ end
+ encoder.end_group(:comment)
+ state = :initial
+
+ when :long_string
+ if match = scan(/.*?(?=\]={#{num_equals}}\])/m) # Long strings do not interpret any escape sequences
+ encoder.text_token(match, :content)
+
+ delim = scan(/\]={#{num_equals}}\]/)
+ encoder.text_token(delim, :delimiter)
+ else # No terminator found till EOF
+ encoder.text_token(rest, :error)
+ terminate
+ end
+ encoder.end_group(:string)
+ state = :initial
+
+ when :string
+ if match = scan(/[^\\#{start_delim}\n]+/) # Everything except \ and the start delimiter character is string content (newlines are only allowed if preceeded by \ or \z)
+ encoder.text_token(match, :content)
+ elsif match = scan(/\\(?:['"abfnrtv\\]|z\s*|x\h\h|\d{1,3}|\n)/m)
+ encoder.text_token(match, :char)
+ elsif match = scan(Regexp.compile(start_delim))
+ encoder.text_token(match, :delimiter)
+ encoder.end_group(:string)
+ state = :initial
+ elsif match = scan(/\n/) # Lua forbids unescaped newlines in normal non-long strings
+ encoder.text_token("\\n\n", :error) # Visually appealing error indicator--otherwise users may wonder whether the highlighter cannot highlight multine strings
+ encoder.end_group(:string)
+ state = :initial
+ else
+ encoder.text_token(getch, :error)
+ end
+
+ when :map
+ if match = scan(/[,;]/)
+ encoder.text_token(match, :operator)
+ elsif match = scan(/[a-zA-Z_][a-zA-Z0-9_]* (?=\s*=)/x)
+ encoder.text_token(match, :key)
+ encoder.text_token(scan(/\s+/), :space) if check(/\s+/)
+ encoder.text_token(scan(/\=/), :operator)
+ state = :initial
+ elsif match = scan(/\s+/m)
+ encoder.text_token(match, :space)
+ else
+ # Note this clause doesn’t advance the scan pointer, it’s a kind of
+ # "retry with other options" (the :initial state then of course
+ # advances the pointer).
+ state = :initial
+ end
+ else
+ raise
+ end
+
+ end
+
+ if options[:keep_state]
+ @state = state
+ end
+
+ encoder.end_group :string if [:string].include? state
+ brace_depth.times { encoder.end_group :map }
+
+ encoder
+ end
+
+ end
+
+end
+end
diff --git a/lib/coderay/scanners/php.rb b/lib/coderay/scanners/php.rb
index 8acfff5..7a8d75d 100644
--- a/lib/coderay/scanners/php.rb
+++ b/lib/coderay/scanners/php.rb
@@ -1,4 +1,4 @@
-# encoding: ASCII-8BIT
+# encoding: utf-8
module CodeRay
module Scanners
@@ -11,7 +11,6 @@ module Scanners
register_for :php
file_extension 'php'
- encoding 'BINARY'
KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
@@ -211,7 +210,7 @@ module Scanners
HTML_INDICATOR = /<!DOCTYPE html|<(?:html|body|div|p)[> ]/i
- IDENTIFIER = /[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*/i
+ IDENTIFIER = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : Regexp.new('[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*', true)
VARIABLE = /\$#{IDENTIFIER}/
OPERATOR = /
@@ -266,7 +265,7 @@ module Scanners
@html_scanner.tokenize match unless match.empty?
end
- when :php
+ when :php, :php_inline
if match = scan(/\s+/)
encoder.text_token match, :space
@@ -333,7 +332,7 @@ module Scanners
if states.size == 1
encoder.text_token match, :error
else
- states.pop
+ state = states.pop
if states.last.is_a?(::Array)
delimiter = states.last[1]
states[-1] = states.last[0]
@@ -341,6 +340,7 @@ module Scanners
encoder.end_group :inline
else
encoder.text_token match, :operator
+ encoder.end_group :inline if state == :php_inline
label_expected = true
end
end
@@ -351,7 +351,14 @@ module Scanners
elsif match = scan(RE::PHP_END)
encoder.text_token match, :inline_delimiter
- states = [:initial]
+ while state = states.pop
+ encoder.end_group :string if [:sqstring, :dqstring].include? state
+ if state.is_a? Array
+ encoder.end_group :inline
+ encoder.end_group :string if [:sqstring, :dqstring].include? state.first
+ end
+ end
+ states << :initial
elsif match = scan(/<<<(?:(#{RE::IDENTIFIER})|"(#{RE::IDENTIFIER})"|'(#{RE::IDENTIFIER})')/o)
encoder.begin_group :string
@@ -401,6 +408,7 @@ module Scanners
elsif match = scan(/\\/)
encoder.text_token match, :error
else
+ encoder.end_group :string
states.pop
end
@@ -460,7 +468,7 @@ module Scanners
encoder.begin_group :inline
states[-1] = [states.last, delimiter]
delimiter = nil
- states.push :php
+ states.push :php_inline
encoder.text_token match, :delimiter
else
encoder.text_token match, :content
@@ -470,6 +478,7 @@ module Scanners
elsif match = scan(/\$/)
encoder.text_token match, :content
else
+ encoder.end_group :string
states.pop
end
@@ -501,6 +510,14 @@ module Scanners
end
+ while state = states.pop
+ encoder.end_group :string if [:sqstring, :dqstring].include? state
+ if state.is_a? Array
+ encoder.end_group :inline
+ encoder.end_group :string if [:sqstring, :dqstring].include? state.first
+ end
+ end
+
encoder
end
diff --git a/lib/coderay/scanners/python.rb b/lib/coderay/scanners/python.rb
index cbdbbdb..09c8b6e 100644
--- a/lib/coderay/scanners/python.rb
+++ b/lib/coderay/scanners/python.rb
@@ -133,7 +133,7 @@ module Scanners
elsif match = scan(/ \\ | $ /x)
encoder.end_group string_type
string_type = nil
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder, state
@@ -157,12 +157,12 @@ module Scanners
encoder.text_token match, :operator
elsif match = scan(/(u?r?|b)?("""|"|'''|')/i)
+ modifiers = self[1]
string_delimiter = self[2]
- string_type = docstring_coming ? :docstring : :string
+ string_type = docstring_coming ? :docstring : (modifiers == 'b' ? :binary : :string)
docstring_coming = false if docstring_coming
encoder.begin_group string_type
string_raw = false
- modifiers = self[1]
unless modifiers.empty?
string_raw = !!modifiers.index(?r)
encoder.text_token modifiers, :modifier
diff --git a/lib/coderay/scanners/raydebug.rb b/lib/coderay/scanners/raydebug.rb
index 7a21354..1effdc8 100644
--- a/lib/coderay/scanners/raydebug.rb
+++ b/lib/coderay/scanners/raydebug.rb
@@ -1,23 +1,30 @@
+require 'set'
+
module CodeRay
module Scanners
-
- # = Debug Scanner
+
+ # = Raydebug Scanner
#
- # Parses the output of the Encoders::Debug encoder.
+ # Highlights the output of the Encoders::Debug encoder.
class Raydebug < Scanner
-
+
register_for :raydebug
file_extension 'raydebug'
title 'CodeRay Token Dump'
protected
+ def setup
+ super
+ @known_token_kinds = TokenKinds.keys.map(&:to_s).to_set
+ end
+
def scan_tokens encoder, options
-
+
opened_tokens = []
-
+
until eos?
-
+
if match = scan(/\s+/)
encoder.text_token match, :space
@@ -26,20 +33,22 @@ module Scanners
encoder.text_token kind, :class
encoder.text_token '(', :operator
match = self[2]
- encoder.text_token match, kind.to_sym
+ unless match.empty?
+ if @known_token_kinds.include? kind
+ encoder.text_token match, kind.to_sym
+ else
+ encoder.text_token match, :plain
+ end
+ end
encoder.text_token match, :operator if match = scan(/\)/)
elsif match = scan(/ (\w+) ([<\[]) /x)
- kind = self[1]
- case self[2]
- when '<'
- encoder.text_token kind, :class
- when '['
- encoder.text_token kind, :class
+ encoder.text_token self[1], :class
+ if @known_token_kinds.include? self[1]
+ kind = self[1].to_sym
else
- raise 'CodeRay bug: This case should not be reached.'
+ kind = :unknown
end
- kind = kind.to_sym
opened_tokens << kind
encoder.begin_group kind
encoder.text_token self[2], :operator
@@ -59,8 +68,8 @@ module Scanners
encoder
end
-
+
end
-
+
end
end
diff --git a/lib/coderay/scanners/ruby.rb b/lib/coderay/scanners/ruby.rb
index 2be98a6..80165ca 100644
--- a/lib/coderay/scanners/ruby.rb
+++ b/lib/coderay/scanners/ruby.rb
@@ -94,18 +94,27 @@ module Scanners
if !method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_NAME}/uo :
/#{patterns::METHOD_NAME}/o)
- value_expected = false
+
kind = patterns::IDENT_KIND[match]
- if kind == :ident
- if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
- kind = :constant
+ if value_expected != :colon_expected && scan(/:(?!:)/)
+ value_expected = true
+ encoder.text_token match, :key
+ encoder.text_token ':', :operator
+ else
+ value_expected = false
+ if kind == :ident
+ if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
+ kind = :constant
+ end
+ elsif kind == :keyword
+ state = patterns::KEYWORD_NEW_STATE[match]
+ if patterns::KEYWORDS_EXPECTING_VALUE[match]
+ value_expected = match == 'when' ? :colon_expected : true
+ end
end
- elsif kind == :keyword
- state = patterns::KEYWORD_NEW_STATE[match]
- value_expected = true if patterns::KEYWORDS_EXPECTING_VALUE[match]
+ value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
+ encoder.text_token match, kind
end
- value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
- encoder.text_token match, kind
elsif method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_AFTER_DOT}/uo :
@@ -119,9 +128,9 @@ module Scanners
value_expected = check(/#{patterns::VALUE_FOLLOWS}/o)
# OPERATORS #
- elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | (?: \.\.\.? | ==?=? | [,\(\[\{] )() | [\)\]\}] /x)
+ elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | ( \.\.\.? | ==?=? | [,\(\[\{] ) | [\)\]\}] /x)
method_call_expected = self[1]
- value_expected = !method_call_expected && self[2]
+ value_expected = !method_call_expected && !!self[2]
if inline_block_stack
case match
when '{'
@@ -213,7 +222,7 @@ module Scanners
encoder.text_token match, :integer
elsif match = scan(/ %=? | <(?:<|=>?)? | \? /x)
- value_expected = true
+ value_expected = match == '?' ? :colon_expected : true
encoder.text_token match, :operator
elsif match = scan(/`/)
@@ -260,7 +269,7 @@ module Scanners
end
if last_state
- state = last_state
+ state = last_state unless state.is_a?(StringState) # otherwise, a simple 'def"' results in unclosed tokens
last_state = nil
end
diff --git a/lib/coderay/scanners/ruby/patterns.rb b/lib/coderay/scanners/ruby/patterns.rb
index a52198e..0b36e13 100644
--- a/lib/coderay/scanners/ruby/patterns.rb
+++ b/lib/coderay/scanners/ruby/patterns.rb
@@ -1,9 +1,9 @@
# encoding: utf-8
module CodeRay
module Scanners
-
+
module Ruby::Patterns # :nodoc: all
-
+
KEYWORDS = %w[
and def end in or unless begin
defined? ensure module redo super until
@@ -12,7 +12,7 @@ module Scanners
while alias class elsif if not return
undef yield
]
-
+
# See http://murfy.de/ruby-constants.
PREDEFINED_CONSTANTS = %w[
nil true false self
@@ -24,19 +24,19 @@ module Scanners
RUBY_PLATFORM RUBY_RELEASE_DATE RUBY_REVISION RUBY_VERSION
__FILE__ __LINE__ __ENCODING__
]
-
+
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_CONSTANTS, :predefined_constant)
-
+
KEYWORD_NEW_STATE = WordList.new(:initial).
add(%w[ def ], :def_expected).
add(%w[ undef ], :undef_expected).
add(%w[ alias ], :alias_expected).
add(%w[ class module ], :module_expected)
-
- IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? /[[:alpha:]_][[:alnum:]_]*/ : /[^\W\d]\w*/
-
+
+ IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? Regexp.new('[[:alpha:]_[^\0-\177]][[:alnum:]_[^\0-\177]]*') : /[^\W\d]\w*/
+
METHOD_NAME = / #{IDENT} [?!]? /ox
METHOD_NAME_OPERATOR = /
\*\*? # multiplication and power
@@ -57,25 +57,25 @@ module Scanners
GLOBAL_VARIABLE = / \$ (?: #{IDENT} | [1-9]\d* | 0\w* | [~&+`'=\/,;_.<>!@$?*":\\] | -[a-zA-Z_0-9] ) /ox
PREFIX_VARIABLE = / #{GLOBAL_VARIABLE} | #{OBJECT_VARIABLE} /ox
VARIABLE = / @?@? #{IDENT} | #{GLOBAL_VARIABLE} /ox
-
+
QUOTE_TO_TYPE = {
'`' => :shell,
'/'=> :regexp,
}
QUOTE_TO_TYPE.default = :string
-
+
REGEXP_MODIFIERS = /[mousenix]*/
-
+
DECIMAL = /\d+(?:_\d+)*/
OCTAL = /0_?[0-7]+(?:_[0-7]+)*/
HEXADECIMAL = /0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*/
BINARY = /0b[01]+(?:_[01]+)*/
-
+
EXPONENT = / [eE] [+-]? #{DECIMAL} /ox
FLOAT_SUFFIX = / #{EXPONENT} | \. #{DECIMAL} #{EXPONENT}? /ox
FLOAT_OR_INT = / #{DECIMAL} (?: #{FLOAT_SUFFIX} () )? /ox
NUMERIC = / (?: (?=0) (?: #{OCTAL} | #{HEXADECIMAL} | #{BINARY} ) | #{FLOAT_OR_INT} ) /ox
-
+
SYMBOL = /
:
(?:
@@ -85,7 +85,7 @@ module Scanners
)
/ox
METHOD_NAME_OR_SYMBOL = / #{METHOD_NAME_EX} | #{SYMBOL} /ox
-
+
SIMPLE_ESCAPE = /
[abefnrstv]
| [0-7]{1,3}
@@ -110,7 +110,7 @@ module Scanners
| \\ #{ESCAPE}
)
/mox
-
+
# NOTE: This is not completely correct, but
# nobody needs heredoc delimiters ending with \n.
HEREDOC_OPEN = /
@@ -122,13 +122,13 @@ module Scanners
( [^\n]*? ) \3 # $4 = delim
)
/mx
-
+
RUBYDOC = /
=begin (?!\S)
.*?
(?: \Z | ^=end (?!\S) [^\n]* )
/mx
-
+
DATA = /
__END__$
.*?
@@ -136,7 +136,7 @@ module Scanners
/mx
RUBYDOC_OR_DATA = / #{RUBYDOC} | #{DATA} /xo
-
+
# Checks for a valid value to follow. This enables
# value_expected in method calls without parentheses.
VALUE_FOLLOWS = /
@@ -157,13 +157,16 @@ module Scanners
yield
])
- FANCY_STRING_START = / % ( [QqrsWwx] | (?![a-zA-Z0-9]) ) ([^a-zA-Z0-9]) /x
+ FANCY_STRING_START = / % ( [iIqQrswWx] | (?![a-zA-Z0-9]) ) ([^a-zA-Z0-9]) /x
FANCY_STRING_KIND = Hash.new(:string).merge({
+ 'i' => :symbol,
+ 'I' => :symbol,
'r' => :regexp,
's' => :symbol,
'x' => :shell,
})
FANCY_STRING_INTERPRETED = Hash.new(true).merge({
+ 'i' => false,
'q' => false,
's' => false,
'w' => false,
diff --git a/lib/coderay/scanners/ruby/string_state.rb b/lib/coderay/scanners/ruby/string_state.rb
index 2f398d1..28ddd6c 100644
--- a/lib/coderay/scanners/ruby/string_state.rb
+++ b/lib/coderay/scanners/ruby/string_state.rb
@@ -16,7 +16,6 @@ module Scanners
STRING_PATTERN = Hash.new do |h, k|
delim, interpreted = *k
- # delim = delim.dup # workaround for old Ruby
delim_pattern = Regexp.escape(delim)
if closing_paren = CLOSING_PAREN[delim]
delim_pattern << Regexp.escape(closing_paren)
@@ -29,12 +28,13 @@ module Scanners
# '| [|?*+(){}\[\].^$]'
# end
- h[k] =
- if interpreted && delim != '#'
- / (?= [#{delim_pattern}] | \# [{$@] ) /mx
- else
- / (?= [#{delim_pattern}] ) /mx
- end
+ if interpreted && delim != '#'
+ / (?= [#{delim_pattern}] | \# [{$@] ) /mx
+ else
+ / (?= [#{delim_pattern}] ) /mx
+ end.tap do |pattern|
+ h[k] = pattern if (delim.respond_to?(:ord) ? delim.ord : delim[0]) < 256
+ end
end
def initialize kind, interpreted, delim, heredoc = false
diff --git a/lib/coderay/scanners/sass.rb b/lib/coderay/scanners/sass.rb
new file mode 100644
index 0000000..e3296b9
--- /dev/null
+++ b/lib/coderay/scanners/sass.rb
@@ -0,0 +1,232 @@
+module CodeRay
+module Scanners
+
+ # A scanner for Sass.
+ class Sass < CSS
+
+ register_for :sass
+ file_extension 'sass'
+
+ protected
+
+ def setup
+ @state = :initial
+ end
+
+ def scan_tokens encoder, options
+ states = Array(options[:state] || @state).dup
+
+ encoder.begin_group :string if states.last == :sqstring || states.last == :dqstring
+
+ until eos?
+
+ if bol? && (match = scan(/(?>( +)?(\/[\*\/])(.+)?)(?=\n)/))
+ encoder.text_token self[1], :space if self[1]
+ encoder.begin_group :comment
+ encoder.text_token self[2], :delimiter
+ encoder.text_token self[3], :content if self[3]
+ if match = scan(/(?:\n+#{self[1]} .*)+/)
+ encoder.text_token match, :content
+ end
+ encoder.end_group :comment
+ elsif match = scan(/\n|[^\n\S]+\n?/)
+ encoder.text_token match, :space
+ if match.index(/\n/)
+ value_expected = false
+ states.pop if states.last == :include
+ end
+
+ elsif states.last == :sass_inline && (match = scan(/\}/))
+ encoder.text_token match, :inline_delimiter
+ encoder.end_group :inline
+ states.pop
+
+ elsif case states.last
+ when :initial, :media, :sass_inline
+ if match = scan(/(?>#{RE::Ident})(?!\()/ox)
+ encoder.text_token match, value_expected ? :value : (check(/.*:(?![a-z])/) ? :key : :tag)
+ next
+ elsif !value_expected && (match = scan(/\*/))
+ encoder.text_token match, :tag
+ next
+ elsif match = scan(RE::Class)
+ encoder.text_token match, :class
+ next
+ elsif match = scan(RE::Id)
+ encoder.text_token match, :id
+ next
+ elsif match = scan(RE::PseudoClass)
+ encoder.text_token match, :pseudo_class
+ next
+ elsif match = scan(RE::AttributeSelector)
+ # TODO: Improve highlighting inside of attribute selectors.
+ encoder.text_token match[0,1], :operator
+ encoder.text_token match[1..-2], :attribute_name if match.size > 2
+ encoder.text_token match[-1,1], :operator if match[-1] == ?]
+ next
+ elsif match = scan(/(\=|@mixin +)#{RE::Ident}/o)
+ encoder.text_token match, :function
+ next
+ elsif match = scan(/@import\b/)
+ encoder.text_token match, :directive
+ states << :include
+ next
+ elsif match = scan(/@media\b/)
+ encoder.text_token match, :directive
+ # states.push :media_before_name
+ next
+ end
+
+ when :block
+ if match = scan(/(?>#{RE::Ident})(?!\()/ox)
+ if value_expected
+ encoder.text_token match, :value
+ else
+ encoder.text_token match, :key
+ end
+ next
+ end
+
+ when :sqstring, :dqstring
+ if match = scan(states.last == :sqstring ? /(?:[^\n\'\#]+|\\\n|#{RE::Escape}|#(?!\{))+/o : /(?:[^\n\"\#]+|\\\n|#{RE::Escape}|#(?!\{))+/o)
+ encoder.text_token match, :content
+ elsif match = scan(/['"]/)
+ encoder.text_token match, :delimiter
+ encoder.end_group :string
+ states.pop
+ elsif match = scan(/#\{/)
+ encoder.begin_group :inline
+ encoder.text_token match, :inline_delimiter
+ states.push :sass_inline
+ elsif match = scan(/ \\ | $ /x)
+ encoder.end_group states.last
+ encoder.text_token match, :error unless match.empty?
+ states.pop
+ else
+ raise_inspect "else case #{states.last} reached; %p not handled." % peek(1), encoder
+ end
+
+ when :include
+ if match = scan(/[^\s'",]+/)
+ encoder.text_token match, :include
+ next
+ end
+
+ else
+ #:nocov:
+ raise_inspect 'Unknown state: %p' % [states.last], encoder
+ #:nocov:
+
+ end
+
+ elsif match = scan(/\$#{RE::Ident}/o)
+ encoder.text_token match, :variable
+ next
+
+ elsif match = scan(/&/)
+ encoder.text_token match, :local_variable
+
+ elsif match = scan(/\+#{RE::Ident}/o)
+ encoder.text_token match, :include
+ value_expected = true
+
+ elsif match = scan(/\/\*(?:.*?\*\/|.*)|\/\/.*/)
+ encoder.text_token match, :comment
+
+ elsif match = scan(/#\{/)
+ encoder.begin_group :inline
+ encoder.text_token match, :inline_delimiter
+ states.push :sass_inline
+
+ elsif match = scan(/\{/)
+ value_expected = false
+ encoder.text_token match, :operator
+ states.push :block
+
+ elsif match = scan(/\}/)
+ value_expected = false
+ encoder.text_token match, :operator
+ if states.last == :block || states.last == :media
+ states.pop
+ end
+
+ elsif match = scan(/['"]/)
+ encoder.begin_group :string
+ encoder.text_token match, :delimiter
+ if states.include? :sass_inline
+ # no nesting, just scan the string until delimiter
+ content = scan_until(/(?=#{match}|\}|\z)/)
+ encoder.text_token content, :content unless content.empty?
+ encoder.text_token match, :delimiter if scan(/#{match}/)
+ encoder.end_group :string
+ else
+ states.push match == "'" ? :sqstring : :dqstring
+ end
+
+ elsif match = scan(/#{RE::Function}/o)
+ encoder.begin_group :function
+ start = match[/^[-\w]+\(/]
+ encoder.text_token start, :delimiter
+ if match[-1] == ?)
+ encoder.text_token match[start.size..-2], :content
+ encoder.text_token ')', :delimiter
+ else
+ encoder.text_token match[start.size..-1], :content if start.size < match.size
+ end
+ encoder.end_group :function
+
+ elsif match = scan(/[a-z][-a-z_]*(?=\()/o)
+ encoder.text_token match, :predefined
+
+ elsif match = scan(/(?: #{RE::Dimension} | #{RE::Percentage} | #{RE::Num} )/ox)
+ encoder.text_token match, :float
+
+ elsif match = scan(/#{RE::HexColor}/o)
+ encoder.text_token match, :color
+
+ elsif match = scan(/! *(?:important|optional)/)
+ encoder.text_token match, :important
+
+ elsif match = scan(/(?:rgb|hsl)a?\([^()\n]*\)?/)
+ encoder.text_token match, :color
+
+ elsif match = scan(/@else if\b|#{RE::AtKeyword}/o)
+ encoder.text_token match, :directive
+ value_expected = true
+
+ elsif match = scan(/ == | != | [-+*\/>~:;,.=()] /x)
+ if match == ':'
+ value_expected = true
+ elsif match == ';'
+ value_expected = false
+ end
+ encoder.text_token match, :operator
+
+ else
+ encoder.text_token getch, :error
+
+ end
+
+ end
+
+ states.pop if states.last == :include
+
+ if options[:keep_state]
+ @state = states.dup
+ end
+
+ while state = states.pop
+ if state == :sass_inline
+ encoder.end_group :inline
+ elsif state == :sqstring || state == :dqstring
+ encoder.end_group :string
+ end
+ end
+
+ encoder
+ end
+
+ end
+
+end
+end
diff --git a/lib/coderay/scanners/sql.rb b/lib/coderay/scanners/sql.rb
index bcbffd5..93aeaf3 100644
--- a/lib/coderay/scanners/sql.rb
+++ b/lib/coderay/scanners/sql.rb
@@ -1,8 +1,9 @@
-module CodeRay module Scanners
+module CodeRay
+module Scanners
# by Josh Goebel
class SQL < Scanner
-
+
register_for :sql
KEYWORDS = %w(
@@ -148,7 +149,8 @@ module CodeRay module Scanners
encoder.text_token string_content, :content
string_content = ''
end
- encoder.text_token match, :error
+ encoder.text_token match, :error unless match.empty?
+ encoder.end_group :string
state = :initial
else
raise "else case \" reached; %p not handled." % peek(1), encoder
@@ -171,4 +173,5 @@ module CodeRay module Scanners
end
-end end
\ No newline at end of file
+end
+end
diff --git a/lib/coderay/scanners/taskpaper.rb b/lib/coderay/scanners/taskpaper.rb
new file mode 100644
index 0000000..42670bc
--- /dev/null
+++ b/lib/coderay/scanners/taskpaper.rb
@@ -0,0 +1,36 @@
+module CodeRay
+module Scanners
+
+ class Taskpaper < Scanner
+
+ register_for :taskpaper
+ file_extension 'taskpaper'
+
+ protected
+
+ def scan_tokens encoder, options
+ until eos?
+ if match = scan(/\S.*:.*$/) # project
+ encoder.text_token(match, :namespace)
+ elsif match = scan(/-.+ at done.*/) # completed task
+ encoder.text_token(match, :done)
+ elsif match = scan(/-(?:[^@\n]+|@(?!due))*/) # task
+ encoder.text_token(match, :plain)
+ elsif match = scan(/@due.*/) # comment
+ encoder.text_token(match, :important)
+ elsif match = scan(/.+/) # comment
+ encoder.text_token(match, :comment)
+ elsif match = scan(/\s+/) # space
+ encoder.text_token(match, :space)
+ else # other
+ encoder.text_token getch, :error
+ end
+ end
+
+ encoder
+ end
+
+ end
+
+end
+end
diff --git a/lib/coderay/scanners/yaml.rb b/lib/coderay/scanners/yaml.rb
index 96f4e93..32c8e2c 100644
--- a/lib/coderay/scanners/yaml.rb
+++ b/lib/coderay/scanners/yaml.rb
@@ -47,7 +47,7 @@ module Scanners
when !check(/(?:"[^"]*")(?=: |:$)/) && match = scan(/"/)
encoder.begin_group :string
encoder.text_token match, :delimiter
- encoder.text_token match, :content if match = scan(/ [^"\\]* (?: \\. [^"\\]* )* /mx)
+ encoder.text_token match, :content if (match = scan(/ [^"\\]* (?: \\. [^"\\]* )* /mx)) && !match.empty?
encoder.text_token match, :delimiter if match = scan(/"/)
encoder.end_group :string
next
@@ -84,7 +84,7 @@ module Scanners
when match = scan(/(?:"[^"\n]*"|'[^'\n]*')(?= *:(?: |$))/)
encoder.begin_group :key
encoder.text_token match[0,1], :delimiter
- encoder.text_token match[1..-2], :content
+ encoder.text_token match[1..-2], :content if match.size > 2
encoder.text_token match[-1,1], :delimiter
encoder.end_group :key
key_indent = column(pos - match.size) - 1
diff --git a/lib/coderay/styles/alpha.rb b/lib/coderay/styles/alpha.rb
index 8506d10..d304dc4 100644
--- a/lib/coderay/styles/alpha.rb
+++ b/lib/coderay/styles/alpha.rb
@@ -3,14 +3,14 @@ module Styles
# A colorful theme using CSS 3 colors (with alpha channel).
class Alpha < Style
-
+
register_for :alpha
-
+
code_background = 'hsl(0,0%,95%)'
numbers_background = 'hsl(180,65%,90%)'
border_color = 'silver'
normal_color = 'black'
-
+
CSS_MAIN_STYLES = <<-MAIN # :nodoc:
.CodeRay {
background-color: #{code_background};
@@ -39,6 +39,9 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
color: gray !important;
text-decoration: none !important;
}
+.CodeRay .line-numbers pre {
+ word-break: normal;
+}
.CodeRay .line-numbers a:target { color: blue !important; }
.CodeRay .line-numbers .highlighted { color: red !important; }
.CodeRay .line-numbers .highlighted a { color: red !important; }
@@ -53,77 +56,84 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
.annotation { color:#007 }
.attribute-name { color:#b48 }
.attribute-value { color:#700 }
-.binary { color:#509 }
+.binary { color:#549 }
+.binary .char { color:#325 }
+.binary .delimiter { color:#325 }
+.char { color:#D20 }
.char .content { color:#D20 }
.char .delimiter { color:#710 }
-.char { color:#D20 }
.class { color:#B06; font-weight:bold }
.class-variable { color:#369 }
.color { color:#0A0 }
.comment { color:#777 }
.comment .char { color:#444 }
.comment .delimiter { color:#444 }
-.complex { color:#A08 }
.constant { color:#036; font-weight:bold }
.decorator { color:#B0B }
.definition { color:#099; font-weight:bold }
.delimiter { color:black }
.directive { color:#088; font-weight:bold }
-.doc { color:#970 }
-.doc-string { color:#D42; font-weight:bold }
+.docstring { color:#D42; }
.doctype { color:#34b }
+.done { text-decoration: line-through; color: gray }
.entity { color:#800; font-weight:bold }
.error { color:#F00; background-color:#FAA }
.escape { color:#666 }
.exception { color:#C00; font-weight:bold }
.float { color:#60E }
.function { color:#06B; font-weight:bold }
+.function .delimiter { color:#024; font-weight:bold }
.global-variable { color:#d70 }
.hex { color:#02b }
-.imaginary { color:#f00 }
+.id { color:#33D; font-weight:bold }
.include { color:#B44; font-weight:bold }
.inline { background-color: hsla(0,0%,0%,0.07); color: black }
.inline-delimiter { font-weight: bold; color: #666 }
.instance-variable { color:#33B }
.integer { color:#00D }
+.imaginary { color:#f00 }
+.important { color:#D00 }
+.key { color: #606 }
.key .char { color: #60f }
.key .delimiter { color: #404 }
-.key { color: #606 }
.keyword { color:#080; font-weight:bold }
.label { color:#970; font-weight:bold }
-.local-variable { color:#963 }
+.local-variable { color:#950 }
+.map .content { color:#808 }
+.map .delimiter { color:#40A}
+.map { background-color:hsla(200,100%,50%,0.06); }
.namespace { color:#707; font-weight:bold }
.octal { color:#40E }
.operator { }
.predefined { color:#369; font-weight:bold }
.predefined-constant { color:#069 }
-.predefined-type { color:#0a5; font-weight:bold }
+.predefined-type { color:#0a8; font-weight:bold }
.preprocessor { color:#579 }
.pseudo-class { color:#00C; font-weight:bold }
+.regexp { background-color:hsla(300,100%,50%,0.06); }
.regexp .content { color:#808 }
.regexp .delimiter { color:#404 }
.regexp .modifier { color:#C2C }
-.regexp { background-color:hsla(300,100%,50%,0.06); }
.reserved { color:#080; font-weight:bold }
+.shell { background-color:hsla(120,100%,50%,0.06); }
.shell .content { color:#2B2 }
.shell .delimiter { color:#161 }
-.shell { background-color:hsla(120,100%,50%,0.06); }
+.string { background-color:hsla(0,100%,50%,0.05); }
.string .char { color: #b0b }
.string .content { color: #D20 }
.string .delimiter { color: #710 }
.string .modifier { color: #E40 }
-.string { background-color:hsla(0,100%,50%,0.05); }
-.symbol .content { color:#A60 }
-.symbol .delimiter { color:#630 }
.symbol { color:#A60 }
-.tag { color:#070 }
+.symbol .content { color:#A60 }
+.symbol .delimiter { color:#740 }
+.tag { color:#070; font-weight:bold }
.type { color:#339; font-weight:bold }
-.value { color: #088; }
-.variable { color:#037 }
+.value { color: #088 }
+.variable { color:#037 }
.insert { background: hsla(120,100%,50%,0.12) }
.delete { background: hsla(0,100%,50%,0.12) }
-.change { color: #bbf; background: #007; }
+.change { color: #bbf; background: #007 }
.head { color: #f8f; background: #505 }
.head .filename { color: white; }
@@ -135,8 +145,8 @@ table.CodeRay td { padding: 2px 4px; vertical-align: top; }
.change .change { color: #88f }
.head .head { color: #f4f }
TOKENS
-
+
end
-
+
end
end
diff --git a/lib/coderay/token_kinds.rb b/lib/coderay/token_kinds.rb
index 3b8d07e..f911862 100755
--- a/lib/coderay/token_kinds.rb
+++ b/lib/coderay/token_kinds.rb
@@ -1,90 +1,85 @@
module CodeRay
# A Hash of all known token kinds and their associated CSS classes.
- TokenKinds = Hash.new do |h, k|
- warn 'Undefined Token kind: %p' % [k] if $CODERAY_DEBUG
- false
- end
+ TokenKinds = Hash.new(false)
# speedup
TokenKinds.compare_by_identity if TokenKinds.respond_to? :compare_by_identity
TokenKinds.update( # :nodoc:
- :annotation => 'annotation',
- :attribute_name => 'attribute-name',
- :attribute_value => 'attribute-value',
- :binary => 'bin',
- :char => 'char',
- :class => 'class',
- :class_variable => 'class-variable',
- :color => 'color',
- :comment => 'comment',
- :complex => 'complex',
- :constant => 'constant',
- :content => 'content',
- :debug => 'debug',
- :decorator => 'decorator',
- :definition => 'definition',
- :delimiter => 'delimiter',
- :directive => 'directive',
- :doc => 'doc',
- :doctype => 'doctype',
- :doc_string => 'doc-string',
- :entity => 'entity',
- :error => 'error',
- :escape => 'escape',
- :exception => 'exception',
- :filename => 'filename',
- :float => 'float',
- :function => 'function',
- :global_variable => 'global-variable',
- :hex => 'hex',
- :imaginary => 'imaginary',
- :important => 'important',
- :include => 'include',
- :inline => 'inline',
- :inline_delimiter => 'inline-delimiter',
- :instance_variable => 'instance-variable',
- :integer => 'integer',
- :key => 'key',
- :keyword => 'keyword',
- :label => 'label',
- :local_variable => 'local-variable',
- :modifier => 'modifier',
- :namespace => 'namespace',
- :octal => 'octal',
- :predefined => 'predefined',
- :predefined_constant => 'predefined-constant',
- :predefined_type => 'predefined-type',
- :preprocessor => 'preprocessor',
- :pseudo_class => 'pseudo-class',
- :regexp => 'regexp',
- :reserved => 'reserved',
- :shell => 'shell',
- :string => 'string',
- :symbol => 'symbol',
- :tag => 'tag',
- :type => 'type',
- :value => 'value',
- :variable => 'variable',
+ :debug => 'debug', # highlight for debugging (white on blue background)
- :change => 'change',
- :delete => 'delete',
- :head => 'head',
- :insert => 'insert',
+ :annotation => 'annotation', # Groovy, Java
+ :attribute_name => 'attribute-name', # HTML, CSS
+ :attribute_value => 'attribute-value', # HTML
+ :binary => 'binary', # Python, Ruby
+ :char => 'char', # most scanners, also inside of strings
+ :class => 'class', # lots of scanners, for different purposes also in CSS
+ :class_variable => 'class-variable', # Ruby, YAML
+ :color => 'color', # CSS
+ :comment => 'comment', # most scanners
+ :constant => 'constant', # PHP, Ruby
+ :content => 'content', # inside of strings, most scanners
+ :decorator => 'decorator', # Python
+ :definition => 'definition', # CSS
+ :delimiter => 'delimiter', # inside strings, comments and other types
+ :directive => 'directive', # lots of scanners
+ :doctype => 'doctype', # Goorvy, HTML, Ruby, YAML
+ :docstring => 'docstring', # Python
+ :done => 'done', # Taskpaper
+ :entity => 'entity', # HTML
+ :error => 'error', # invalid token, most scanners
+ :escape => 'escape', # Ruby (string inline variables like #$foo, #@bar)
+ :exception => 'exception', # Java, PHP, Python
+ :filename => 'filename', # Diff
+ :float => 'float', # most scanners
+ :function => 'function', # CSS, JavaScript, PHP
+ :global_variable => 'global-variable', # Ruby, YAML
+ :hex => 'hex', # hexadecimal number; lots of scanners
+ :id => 'id', # CSS
+ :imaginary => 'imaginary', # Python
+ :important => 'important', # CSS, Taskpaper
+ :include => 'include', # C, Groovy, Java, Python, Sass
+ :inline => 'inline', # nested code, eg. inline string evaluation; lots of scanners
+ :inline_delimiter => 'inline-delimiter', # used instead of :inline > :delimiter FIXME: Why use inline_delimiter?
+ :instance_variable => 'instance-variable', # Ruby
+ :integer => 'integer', # most scanners
+ :key => 'key', # lots of scanners, used together with :value
+ :keyword => 'keyword', # reserved word that's actually implemented; most scanners
+ :label => 'label', # C, PHP
+ :local_variable => 'local-variable', # local and magic variables; some scanners
+ :map => 'map', # Lua tables
+ :modifier => 'modifier', # used inside on strings; lots of scanners
+ :namespace => 'namespace', # Clojure, Java, Taskpaper
+ :octal => 'octal', # lots of scanners
+ :predefined => 'predefined', # predefined function: lots of scanners
+ :predefined_constant => 'predefined-constant',# lots of scanners
+ :predefined_type => 'predefined-type', # C, Java, PHP
+ :preprocessor => 'preprocessor', # C, Delphi, HTML
+ :pseudo_class => 'pseudo-class', # CSS
+ :regexp => 'regexp', # Groovy, JavaScript, Ruby
+ :reserved => 'reserved', # most scanners
+ :shell => 'shell', # Ruby
+ :string => 'string', # most scanners
+ :symbol => 'symbol', # Clojure, Ruby, YAML
+ :tag => 'tag', # CSS, HTML
+ :type => 'type', # CSS, Java, SQL, YAML
+ :value => 'value', # used together with :key; CSS, JSON, YAML
+ :variable => 'variable', # Sass, SQL, YAML
- :eyecatcher => 'eyecatcher',
+ :change => 'change', # Diff
+ :delete => 'delete', # Diff
+ :head => 'head', # Diff, YAML
+ :insert => 'insert', # Diff
+ :eyecatcher => 'eyecatcher', # Diff
- :ident => false,
- :operator => false,
+ :ident => false, # almost all scanners
+ :operator => false, # almost all scanners
- :space => false,
- :plain => false
+ :space => false, # almost all scanners
+ :plain => false # almost all scanners
)
- TokenKinds[:method] = TokenKinds[:function]
- TokenKinds[:escape] = TokenKinds[:delimiter]
- TokenKinds[:docstring] = TokenKinds[:comment]
-
- TokenKinds.freeze
+ TokenKinds[:method] = TokenKinds[:function]
+ TokenKinds[:unknown] = TokenKinds[:plain]
end
diff --git a/lib/coderay/tokens.rb b/lib/coderay/tokens.rb
index c747017..e7bffce 100644
--- a/lib/coderay/tokens.rb
+++ b/lib/coderay/tokens.rb
@@ -1,55 +1,43 @@
module CodeRay
- # GZip library for writing and reading token dumps.
- autoload :GZip, coderay_path('helpers', 'gzip')
-
- # = Tokens TODO: Rewrite!
- #
- # The Tokens class represents a list of tokens returnd from
- # a Scanner.
+ # The Tokens class represents a list of tokens returned from
+ # a Scanner. It's actually just an Array with a few helper methods.
#
- # A token is not a special object, just a two-element Array
- # consisting of
+ # A token itself is not a special object, just two elements in an Array:
# * the _token_ _text_ (the original source of the token in a String) or
# a _token_ _action_ (begin_group, end_group, begin_line, end_line)
# * the _token_ _kind_ (a Symbol representing the type of the token)
#
- # A token looks like this:
+ # It looks like this:
#
- # ['# It looks like this', :comment]
- # ['3.1415926', :float]
- # ['$^', :error]
+ # ..., '# It looks like this', :comment, ...
+ # ..., '3.1415926', :float, ...
+ # ..., '$^', :error, ...
#
# Some scanners also yield sub-tokens, represented by special
- # token actions, namely begin_group and end_group.
+ # token actions, for example :begin_group and :end_group.
#
# The Ruby scanner, for example, splits "a string" into:
#
# [
- # [:begin_group, :string],
- # ['"', :delimiter],
- # ['a string', :content],
- # ['"', :delimiter],
- # [:end_group, :string]
+ # :begin_group, :string,
+ # '"', :delimiter,
+ # 'a string', :content,
+ # '"', :delimiter,
+ # :end_group, :string
# ]
#
- # Tokens is the interface between Scanners and Encoders:
- # The input is split and saved into a Tokens object. The Encoder
- # then builds the output from this object.
- #
- # Thus, the syntax below becomes clear:
+ # Tokens can be used to save the output of a Scanners in a simple
+ # Ruby object that can be send to an Encoder later:
#
- # CodeRay.scan('price = 2.59', :ruby).html
- # # the Tokens object is here -------^
- #
- # See how small it is? ;)
+ # tokens = CodeRay.scan('price = 2.59', :ruby).tokens
+ # tokens.encode(:html)
+ # tokens.html
+ # CodeRay.encoder(:html).encode_tokens(tokens)
#
# Tokens gives you the power to handle pre-scanned code very easily:
- # You can convert it to a webpage, a YAML file, or dump it into a gzip'ed string
- # that you put in your DB.
- #
- # It also allows you to generate tokens directly (without using a scanner),
- # to load them from a file, and still use any Encoder that CodeRay provides.
+ # You can serialize it to a JSON string and store it in a database, pass it
+ # around to encode it more than once, send it to other algorithms...
class Tokens < Array
# The Scanner instance that created the tokens.
@@ -58,8 +46,7 @@ module CodeRay
# Encode the tokens using encoder.
#
# encoder can be
- # * a symbol like :html oder :statistic
- # * an Encoder class
+ # * a plugin name like :html oder 'statistic'
# * an Encoder object
#
# options are passed to the encoder.
@@ -93,6 +80,7 @@ module CodeRay
# This method is used by @Scanner#tokenize@ when called with an Array
# of source strings. The Diff encoder uses it for inline highlighting.
def split_into_parts *sizes
+ return Array.new(sizes.size) { Tokens.new } if size == 2 && first == ''
parts = []
opened = []
content = nil
@@ -156,53 +144,11 @@ module CodeRay
parts
end
- # Dumps the object into a String that can be saved
- # in files or databases.
- #
- # The dump is created with Marshal.dump;
- # In addition, it is gzipped using GZip.gzip.
- #
- # The returned String object includes Undumping
- # so it has an #undump method. See Tokens.load.
- #
- # You can configure the level of compression,
- # but the default value 7 should be what you want
- # in most cases as it is a good compromise between
- # speed and compression rate.
- #
- # See GZip module.
- def dump gzip_level = 7
- dump = Marshal.dump self
- dump = GZip.gzip dump, gzip_level
- dump.extend Undumping
- end
-
# Return the actual number of tokens.
def count
size / 2
end
- # Include this module to give an object an #undump
- # method.
- #
- # The string returned by Tokens.dump includes Undumping.
- module Undumping
- # Calls Tokens.load with itself.
- def undump
- Tokens.load self
- end
- end
-
- # Undump the object using Marshal.load, then
- # unzip it using GZip.gunzip.
- #
- # The result is commonly a Tokens object, but
- # this is not guaranteed.
- def Tokens.load dump
- dump = GZip.gunzip dump
- @dump = Marshal.load dump
- end
-
alias text_token push
def begin_group kind; push :begin_group, kind end
def end_group kind; push :end_group, kind end
diff --git a/lib/coderay/version.rb b/lib/coderay/version.rb
index 368b963..4b4f085 100644
--- a/lib/coderay/version.rb
+++ b/lib/coderay/version.rb
@@ -1,3 +1,3 @@
module CodeRay
- VERSION = '1.0.6'
+ VERSION = '1.1.0'
end
diff --git a/metadata.yml b/metadata.yml
index 5d0717e..3686c2d 100644
--- a/metadata.yml
+++ b/metadata.yml
@@ -1,15 +1,14 @@
--- !ruby/object:Gem::Specification
name: coderay
version: !ruby/object:Gem::Version
- version: 1.0.6
- prerelease:
+ version: 1.1.0
platform: ruby
authors:
- Kornelius Kalnbach
autorequire:
bindir: bin
cert_chain: []
-date: 2012-04-05 00:00:00.000000000 Z
+date: 2013-08-31 00:00:00.000000000 Z
dependencies: []
description: Fast and easy syntax highlighting for selected languages, written in
Ruby. Comes with RedCloth integration and LOC counter.
@@ -21,7 +20,6 @@ extensions: []
extra_rdoc_files:
- README_INDEX.rdoc
files:
-- LICENSE
- README_INDEX.rdoc
- Rakefile
- lib/coderay.rb
@@ -31,6 +29,7 @@ files:
- lib/coderay/encoders/comment_filter.rb
- lib/coderay/encoders/count.rb
- lib/coderay/encoders/debug.rb
+- lib/coderay/encoders/debug_lint.rb
- lib/coderay/encoders/div.rb
- lib/coderay/encoders/filter.rb
- lib/coderay/encoders/html.rb
@@ -39,6 +38,7 @@ files:
- lib/coderay/encoders/html/output.rb
- lib/coderay/encoders/json.rb
- lib/coderay/encoders/lines_of_code.rb
+- lib/coderay/encoders/lint.rb
- lib/coderay/encoders/null.rb
- lib/coderay/encoders/page.rb
- lib/coderay/encoders/span.rb
@@ -50,7 +50,6 @@ files:
- lib/coderay/encoders/yaml.rb
- lib/coderay/for_redcloth.rb
- lib/coderay/helpers/file_type.rb
-- lib/coderay/helpers/gzip.rb
- lib/coderay/helpers/plugin.rb
- lib/coderay/helpers/word_list.rb
- lib/coderay/scanner.rb
@@ -63,6 +62,7 @@ files:
- lib/coderay/scanners/delphi.rb
- lib/coderay/scanners/diff.rb
- lib/coderay/scanners/erb.rb
+- lib/coderay/scanners/go.rb
- lib/coderay/scanners/groovy.rb
- lib/coderay/scanners/haml.rb
- lib/coderay/scanners/html.rb
@@ -70,13 +70,16 @@ files:
- lib/coderay/scanners/java/builtin_types.rb
- lib/coderay/scanners/java_script.rb
- lib/coderay/scanners/json.rb
+- lib/coderay/scanners/lua.rb
- lib/coderay/scanners/php.rb
- lib/coderay/scanners/python.rb
- lib/coderay/scanners/raydebug.rb
- lib/coderay/scanners/ruby.rb
- lib/coderay/scanners/ruby/patterns.rb
- lib/coderay/scanners/ruby/string_state.rb
+- lib/coderay/scanners/sass.rb
- lib/coderay/scanners/sql.rb
+- lib/coderay/scanners/taskpaper.rb
- lib/coderay/scanners/text.rb
- lib/coderay/scanners/xml.rb
- lib/coderay/scanners/yaml.rb
@@ -91,10 +94,11 @@ files:
- test/functional/examples.rb
- test/functional/for_redcloth.rb
- test/functional/suite.rb
-- !binary |-
- YmluL2NvZGVyYXk=
+- bin/coderay
homepage: http://coderay.rubychan.de
-licenses: []
+licenses:
+- MIT
+metadata: {}
post_install_message:
rdoc_options:
- -SNw2
@@ -103,22 +107,20 @@ rdoc_options:
require_paths:
- lib
required_ruby_version: !ruby/object:Gem::Requirement
- none: false
requirements:
- - - ! '>='
+ - - '>='
- !ruby/object:Gem::Version
version: 1.8.6
required_rubygems_version: !ruby/object:Gem::Requirement
- none: false
requirements:
- - - ! '>='
+ - - '>='
- !ruby/object:Gem::Version
version: '0'
requirements: []
rubyforge_project: coderay
-rubygems_version: 1.8.21
+rubygems_version: 2.0.3
signing_key:
-specification_version: 3
+specification_version: 4
summary: Fast syntax highlighting for selected languages.
test_files:
- test/functional/basic.rb
diff --git a/test/functional/basic.rb b/test/functional/basic.rb
index 3053b54..752d4ba 100755
--- a/test/functional/basic.rb
+++ b/test/functional/basic.rb
@@ -164,9 +164,7 @@ more code # and another comment, in-line.
end
end
assert_equal 'reserved', CodeRay::TokenKinds[:reserved]
- assert_warning 'Undefined Token kind: :shibboleet' do
- assert_equal false, CodeRay::TokenKinds[:shibboleet]
- end
+ assert_equal false, CodeRay::TokenKinds[:shibboleet]
end
class Milk < CodeRay::Encoders::Encoder
diff --git a/test/functional/examples.rb b/test/functional/examples.rb
index ff64af3..985ef87 100755
--- a/test/functional/examples.rb
+++ b/test/functional/examples.rb
@@ -22,7 +22,7 @@ end
CODE
assert_equal <<-DIV, div
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><a href="#n1" name="n1">1</a>
+ <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
<a href="#n2" name="n2">2</a>
<a href="#n3" name="n3">3</a>
</pre></td>
@@ -38,7 +38,7 @@ end
<body>
<table class="CodeRay"><tr>
- <td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>
+ <td class="line-numbers"><pre><a href="#n1" name="n1">1</a>
</pre></td>
<td class="code"><pre>puts <span class="string"><span class="delimiter">"</span><span class="content">Hello, world!</span><span class="delimiter">"</span></span></pre></td>
</tr></table>
@@ -114,7 +114,7 @@ Token Types (7):
# format the tokens
term = terminal_encoder.encode_tokens(tokens)
- assert_equal "\e[1;31mimport\e[0m \e[33mthis\e[0m; \e[37m# The Zen of Python\e[0m", term
+ assert_equal "\e[32mimport\e[0m \e[31mthis\e[0m; \e[1;30m# The Zen of Python\e[0m", term
# re-using scanner and encoder
ruby_highlighter = CodeRay::Duo[:ruby, :div]
diff --git a/test/functional/for_redcloth.rb b/test/functional/for_redcloth.rb
index e980667..9fd244e 100644
--- a/test/functional/for_redcloth.rb
+++ b/test/functional/for_redcloth.rb
@@ -1,5 +1,4 @@
require 'test/unit'
-require File.expand_path('../../lib/assert_warning', __FILE__)
$:.unshift File.expand_path('../../../lib', __FILE__)
require 'coderay'
@@ -66,19 +65,14 @@ class BasicTest < Test::Unit::TestCase
# See http://jgarber.lighthouseapp.com/projects/13054/tickets/124-code-markup-does-not-allow-brackets.
def test_for_redcloth_false_positive
require 'coderay/for_redcloth'
- assert_warning 'CodeRay::Scanners could not load plugin :project; falling back to :text' do
- assert_equal '<p><code>[project]_dff.skjd</code></p>',
- RedCloth.new('@[project]_dff.skjd@').to_html
- end
+ assert_equal '<p><code>[project]_dff.skjd</code></p>',
+ RedCloth.new('@[project]_dff.skjd@').to_html
# false positive, but expected behavior / known issue
assert_equal "<p><span lang=\"ruby\" class=\"CodeRay\">_dff.skjd</span></p>",
RedCloth.new('@[ruby]_dff.skjd@').to_html
- assert_warning 'CodeRay::Scanners could not load plugin :project; falling back to :text' do
- assert_equal <<-BLOCKCODE.chomp,
+ assert_equal <<-BLOCKCODE.chomp, RedCloth.new('bc. [project]_dff.skjd').to_html
<pre><code>[project]_dff.skjd</code></pre>
- BLOCKCODE
- RedCloth.new('bc. [project]_dff.skjd').to_html
- end
+ BLOCKCODE
end
end if defined? RedCloth
\ No newline at end of file
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-ruby-extras/coderay.git
More information about the Pkg-ruby-extras-commits
mailing list