[DRE-commits] [ruby-parslet] 01/03: Imported Upstream version 1.6.1
Youhei SASAKI
uwabami-guest at moszumanska.debian.org
Wed Jul 23 11:32:36 UTC 2014
This is an automated email from the git hooks/post-receive script.
uwabami-guest pushed a commit to branch master
in repository ruby-parslet.
commit 0cc156054413daa965f602cea9a5f4afa6e2de76
Author: Youhei SASAKI <uwabami at gfd-dennou.org>
Date: Wed Jul 23 17:57:59 2014 +0900
Imported Upstream version 1.6.1
---
HISTORY.txt | 27 +++-
LICENSE | 3 +-
README | 21 +--
checksums.yaml.gz | Bin 0 -> 269 bytes
example/big.erb | 73 ++++++++++
example/email_parser.rb | 8 +-
example/optimized_erb.rb | 42 ++++++
example/output/optimized_erb.out | 1 +
example/output/prec_calc.out | 5 +
example/prec_calc.rb | 71 ++++++++++
lib/parslet.rb | 34 ++++-
lib/parslet/accelerator.rb | 161 ++++++++++++++++++++++
lib/parslet/accelerator/application.rb | 62 +++++++++
lib/parslet/accelerator/engine.rb | 112 +++++++++++++++
lib/parslet/atoms.rb | 1 +
lib/parslet/atoms/base.rb | 8 +-
lib/parslet/atoms/context.rb | 14 +-
lib/parslet/atoms/infix.rb | 121 +++++++++++++++++
lib/parslet/atoms/lookahead.rb | 9 +-
lib/parslet/atoms/re.rb | 2 +-
lib/parslet/atoms/repetition.rb | 7 +-
lib/parslet/atoms/str.rb | 3 +-
lib/parslet/{transform => }/context.rb | 5 +-
lib/parslet/graphviz.rb | 97 +++++++++++++
lib/parslet/pattern.rb | 2 +-
lib/parslet/position.rb | 21 +++
lib/parslet/rig/rspec.rb | 12 +-
lib/parslet/slice.rb | 17 ++-
lib/parslet/source.rb | 70 +++++++---
lib/parslet/source/line_cache.rb | 27 ++--
lib/parslet/transform.rb | 6 +-
metadata.yml | 242 +++++++++------------------------
32 files changed, 1029 insertions(+), 255 deletions(-)
diff --git a/HISTORY.txt b/HISTORY.txt
index a4491e2..ad81ce2 100644
--- a/HISTORY.txt
+++ b/HISTORY.txt
@@ -2,8 +2,33 @@
- prsnt? and absnt? are now finally banned into oblivion. Wasting vocals for
the win.
+
+= 1.7 / ???
+
+ ! Small speed gains from improvements on the hot spots.
+
+= 1.6 / 1May2014
+
+ + EXPERIMENTAL: Parslet accelerators permit replacing parts of your parser
+ with optimized atoms using pattern matching. Look at
+ examples/optimized_erb.rb or the introduction to the feature in
+ qed/accelerators.md.
+
+ + infix_expression permits to declare an infix expression parser (think
+ calculator) directly. This will solve many of the problems we have
+ more elegantly.
+
+ + Rspec 3 syntax, though hideous, should now work.
+
+ - Drops 1.8.7 compatibility.
+
+ ! A performance anomaly when parsing multibyte characters has been detected
+ and fixed with the help of Zach Moazeni (@zmoazeni).
+
+ ! A few small bug fixes and optimisations have been introduced. API should
+ remain unchanged.
-= 1.5 / ??
+= 1.5 / 27Dec2012
+ Handles unconsumed input at end of parse completely differently. Instead
of generating a toplevel error, it now raises an error in every branch
diff --git a/LICENSE b/LICENSE
index e57eec2..7384060 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,5 +1,4 @@
-
- Copyright (c) 2010 Kaspar Schiess
+ Copyright (c) 2010-2014 Kaspar Schiess
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
diff --git a/README b/README
index 5eb3f0f..90444b1 100644
--- a/README
+++ b/README
@@ -51,20 +51,23 @@ SYNOPSIS
# and then
Smalltalk.new.parse('smalltalk')
-COMPATIBILITY
+FEATURES
-This library should work with most rubies. I've tested it with MRI 1.8
-(except 1.8.6), 1.9, rbx-head, jruby. Please report as a bug if you encounter
-issues.
+ * Tools for every part of the parser chain
+ * Transformers generate Abstract Syntax Trees
+ * Accelerators transform parsers, making them quite a bit faster
+ * Pluggable error reporters
+ * Graphviz export for your parser
+ * Rspec testing support rig
+ * Simply Ruby, composable and hackable
-Note that due to Ruby 1.8 internals, Unicode parsing is not supported on that
-version.
+COMPATIBILITY
-On Mac OS X Lion, ruby-1.8.7-p352 has been known to segfault. Use
-ruby-1.8.7-p334 for better results.
+This library is intended to work with Ruby variants >= 1.9. I've tested it on
+MRI 1.9, rbx-head, jruby. Please report as a bug if you encounter issues.
STATUS
Production worthy.
-(c) 2010, 2011, 2012 Kaspar Schiess
\ No newline at end of file
+(c) 2010-2014 Kaspar Schiess
diff --git a/checksums.yaml.gz b/checksums.yaml.gz
new file mode 100644
index 0000000..7fea2e9
Binary files /dev/null and b/checksums.yaml.gz differ
diff --git a/example/big.erb b/example/big.erb
new file mode 100644
index 0000000..6a36800
--- /dev/null
+++ b/example/big.erb
@@ -0,0 +1,73 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+<%= erb tag %>
+
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
+cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
+proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
diff --git a/example/email_parser.rb b/example/email_parser.rb
index e1a858e..1bfcfea 100644
--- a/example/email_parser.rb
+++ b/example/email_parser.rb
@@ -44,9 +44,11 @@ end
parser = EmailParser.new
sanitizer = EmailSanitizer.new
-unless ARGV[0]
+input = ARGV[0] || begin
+ default = "a.b.c.d at gmail.com"
STDERR.puts "usage: #{$0} \"EMAIL_ADDR\""
- STDOUT.puts "since you haven't specified any EMAIL_ADDR, for testing purposes we're using a.b.c.d at gmail.com"
+ STDOUT.puts "since you haven't specified any EMAIL_ADDR, for testing purposes we're using #{default}"
+ default
end
-p sanitizer.apply(parser.parse_with_debug(ARGV[0] || 'a.b.c.d at gmail.com'))
+p sanitizer.apply(parser.parse_with_debug(input))
diff --git a/example/optimized_erb.rb b/example/optimized_erb.rb
new file mode 100644
index 0000000..228619e
--- /dev/null
+++ b/example/optimized_erb.rb
@@ -0,0 +1,42 @@
+# Please also look at the more naive 'erb.rb'. This shows how to optimize an
+# ERB like parser using parslet.
+
+$:.unshift File.join(File.dirname(__FILE__), "/../lib")
+
+require 'parslet'
+require './qed/applique/gobbleup'
+require 'parslet/accelerator'
+
+class ErbParser < Parslet::Parser
+ rule(:ruby) { (str('%>').absent? >> any).repeat.as(:ruby) }
+
+ rule(:expression) { (str('=') >> ruby).as(:expression) }
+ rule(:comment) { (str('#') >> ruby).as(:comment) }
+ rule(:code) { ruby.as(:code) }
+ rule(:erb) { expression | comment | code }
+
+ rule(:erb_with_tags) { str('<%') >> erb >> str('%>') }
+ rule(:text) { (str('<%').absent? >> any).repeat(1) }
+
+ rule(:text_with_ruby) { (text.as(:text) | erb_with_tags).repeat.as(:text) }
+ root(:text_with_ruby)
+end
+
+parser = ErbParser.new
+
+A = Parslet::Accelerator
+optimized = A.apply(parser,
+ A.rule((A.str(:x).absent? >> A.any).repeat(1)) { GobbleUp.new(x, 1) },
+ A.rule((A.str(:x).absent? >> A.any).repeat(0)) { GobbleUp.new(x, 0) })
+
+input = File.read(File.dirname(__FILE__) + "/big.erb")
+
+# Remove the comment marks here to see what difference the optimisation makes.
+# Commented out for the acceptance tests to run.
+#
+# require 'benchmark'
+# Benchmark.bm(7) do |bm|
+# bm.report('original') { parser.parse(input) }
+# bm.report('gobble') { optimized.parse(input) }
+# end
+p optimized.parse(input)
\ No newline at end of file
diff --git a/example/output/optimized_erb.out b/example/output/optimized_erb.out
new file mode 100644
index 0000000..4aaf516
--- /dev/null
+++ b/example/output/optimized_erb.out
@@ -0,0 +1 @@
+{:text=>[{:text=>"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod\ntempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,\nquis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo\nconsequat. Duis aute irure dolor in reprehenderit in voluptate velit esse\ncillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non\nproident, sunt in culpa qui officia deserunt mollit anim id est laborum.\nLorem ipsum dolor sit [...]
diff --git a/example/output/prec_calc.out b/example/output/prec_calc.out
new file mode 100644
index 0000000..02c8403
--- /dev/null
+++ b/example/output/prec_calc.out
@@ -0,0 +1,5 @@
+a = 1
+b = 2
+c = 3 * 25
+d = 100 + 3*4
+{:a=>1, :b=>2, :c=>75, :d=>112}
diff --git a/example/prec_calc.rb b/example/prec_calc.rb
new file mode 100644
index 0000000..889958f
--- /dev/null
+++ b/example/prec_calc.rb
@@ -0,0 +1,71 @@
+
+# A demonstration of the new precedence climbing infix expression parser.
+
+$:.unshift File.dirname(__FILE__) + "/../lib"
+
+require 'pp'
+require 'rspec'
+require 'parslet'
+require 'parslet/rig/rspec'
+require 'parslet/convenience'
+
+class InfixExpressionParser < Parslet::Parser
+ root :variable_assignment_list
+
+ rule(:space) { match[' '] }
+
+ def cts atom
+ atom >> space.repeat
+ end
+ def infix *args
+ Infix.new(*args)
+ end
+
+ # This is the heart of the infix expression parser: real simple definitions
+ # for all the pieces we need.
+ rule(:mul_op) { cts match['*/'] }
+ rule(:add_op) { cts match['+-'] }
+ rule(:digit) { match['0-9'] }
+ rule(:integer) { cts digit.repeat(1).as(:int) }
+
+ rule(:expression) { infix_expression(integer,
+ [mul_op, 2, :left],
+ [add_op, 1, :right]) }
+
+ # And now adding variable assignments to that, just to a) demonstrate this
+ # embedded in a bigger parser, and b) make the example interesting.
+ rule(:variable_assignment_list) {
+ variable_assignment.repeat(1) }
+ rule(:variable_assignment) {
+ identifier.as(:ident) >> equal_sign >> expression.as(:exp) >> eol }
+ rule(:identifier) {
+ cts (match['a-z'] >> match['a-zA-Z0-9'].repeat) }
+ rule(:equal_sign) {
+ cts str('=') }
+ rule(:eol) {
+ cts(str("\n")) | any.absent? }
+end
+
+class InfixInterpreter < Parslet::Transform
+ rule(int: simple(:int)) { Integer(int) }
+ rule(ident: simple(:ident), exp: simple(:result)) { |d|
+ d[:doc][d[:ident].to_s.strip.to_sym] = d[:result] }
+
+ rule(l: simple(:l), o: /^\*/, r: simple(:r)) { l * r }
+ rule(l: simple(:l), o: /^\+/, r: simple(:r)) { l + r }
+end
+
+input = <<ASSIGNMENTS
+a = 1
+b = 2
+c = 3 * 25
+d = 100 + 3*4
+ASSIGNMENTS
+
+puts input
+
+int_tree = InfixExpressionParser.new.parse_with_debug(input)
+bindings = {}
+result = InfixInterpreter.new.apply(int_tree, doc: bindings)
+
+pp bindings
diff --git a/lib/parslet.rb b/lib/parslet.rb
index cebd3c5..2fef5ce 100644
--- a/lib/parslet.rb
+++ b/lib/parslet.rb
@@ -199,6 +199,38 @@ module Parslet
Parslet::Atoms::Dynamic.new(block)
end
module_function :dynamic
+
+ # Returns a parslet atom that parses infix expressions. Operations are
+ # specified as a list of <atom, precedence, associativity> tuples, where
+ # atom is simply the parslet atom that matches an operator, precedence is
+ # a number and associativity is either :left or :right.
+ #
+ # Higher precedence indicates that the operation should bind tighter than
+ # other operations with lower precedence. In common algebra, '+' has
+ # lower precedence than '*'. So you would have a precedence of 1 for '+' and
+ # a precedence of 2 for '*'. Only the order relation between these two
+ # counts, so any number would work.
+ #
+ # Associativity is what decides what interpretation to take for strings that
+ # are ambiguous like '1 + 2 + 3'. If '+' is specified as left associative,
+ # the expression would be interpreted as '(1 + 2) + 3'. If right
+ # associativity is chosen, it would be interpreted as '1 + (2 + 3)'. Note
+ # that the hash trees output reflect that choice as well.
+ #
+ # Example:
+ # infix_expression(integer, [add_op, 1, :left])
+ # # would parse things like '1 + 2'
+ #
+ # @param element [Parslet::Atoms::Base] elements that take the NUMBER position
+ # in the expression
+ # @param operations [Array<(Parslet::Atoms::Base, Integer, {:left, :right})>]
+ #
+ # @see Parslet::Atoms::Infix
+ #
+ def infix_expression(element, *operations)
+ Parslet::Atoms::Infix.new(element, operations)
+ end
+ module_function :infix_expression
# A special kind of atom that allows embedding whole treetop expressions
# into parslet construction.
@@ -254,7 +286,7 @@ module Parslet
Pattern::SubtreeBind.new(symbol)
end
module_function :subtree
-
+
autoload :Expression, 'parslet/expression'
end
diff --git a/lib/parslet/accelerator.rb b/lib/parslet/accelerator.rb
new file mode 100644
index 0000000..79987d3
--- /dev/null
+++ b/lib/parslet/accelerator.rb
@@ -0,0 +1,161 @@
+
+
+# Optimizes the parsers by pattern matching on the parser atoms and replacing
+# matches with better versions. See the file qed/accelerators.md for a more
+# in-depth description.
+#
+# Example:
+# quote = str('"')
+# parser = quote >> (quote.absent? >> any).repeat >> quote
+#
+# A = Accelerator # for making what follows a bit shorter
+# optimized_parser = A.apply(parser,
+# A.rule( (A.str(:x).absent? >> A.any).repeat ) { GobbleUp.new(x) })
+#
+# optimized_parser.parse('"Parsing is now fully optimized! (tm)"')
+#
+module Parslet::Accelerator
+
+ # An expression to match against a tree of parser atoms. Normally, an
+ # expression is produced by Parslet::Accelerator.any,
+ # Parslet::Accelerator.str or Parslet::Accelerator.re.
+ #
+ # Expressions can be chained much like parslet atoms can be:
+ #
+ # expr.repeat(1) # matching repetition
+ # expr.absent? # matching absent?
+ # expr.present? # matching present?
+ # expr1 >> expr2 # matching a sequence
+ # expr1 | expr2 # matching an alternation
+ #
+ # @see Parslet::Accelerator.str
+ # @see Parslet::Accelerator.re
+ # @see Parslet::Accelerator.any
+ #
+ # @see Parslet::Accelerator
+ #
+ class Expression
+ attr_reader :type
+ attr_reader :args
+
+ def initialize(type, *args)
+ @type = type
+ @args = args
+ end
+
+ # @return [Expression]
+ def >> other_expr
+ join_or_new :seq, other_expr
+ end
+
+ # @return [Expression]
+ def | other_expr
+ join_or_new :alt, other_expr
+ end
+
+ # @return [Expression]
+ def absent?
+ Expression.new(:absent, self)
+ end
+ # @return [Expression]
+ def present?
+ Expression.new(:present, self)
+ end
+
+ # @return [Expression]
+ def repeat min=0, max=nil
+ Expression.new(:rep, min, max, self)
+ end
+
+ # @return [Expression]
+ def as name
+ Expression.new(:as, name)
+ end
+
+ # @api private
+ # @return [Expression]
+ def join_or_new tag, other_expr
+ if type == tag
+ @args << other_expr
+ else
+ Expression.new(tag, self, other_expr)
+ end
+ end
+ end
+
+module_function
+ # Returns a match expression that will match `str` parslet atoms.
+ #
+ # @return [Parslet::Accelerator::Expression]
+ #
+ def str variable, *constraints
+ Expression.new(:str, variable, *constraints)
+ end
+
+ # Returns a match expression that will match `match` parslet atoms.
+ #
+ # @return [Parslet::Accelerator::Expression]
+ #
+ def re variable, *constraints
+ Expression.new(:re, variable, *constraints)
+ end
+
+ # Returns a match expression that will match `any` parslet atoms.
+ #
+ # @return [Parslet::Accelerator::Expression]
+ #
+ def any
+ Expression.new(:re, ".")
+ end
+
+ # Given a parslet atom and an expression, will determine if the expression
+ # matches the atom. If successful, returns the bindings into the pattern
+ # that were made. If no bindings had to be made to make the match successful,
+ # the empty hash is returned.
+ #
+ # @param atom [Parslet::Atoms::Base] parslet atom to match against
+ # @param expr [Parslet::Accelerator::Expression] expression to match
+ # @return [nil, Hash] bindings for the match, nil on failure
+ #
+ def match atom, expr
+ engine = Engine.new
+
+ return engine.bindings if engine.match(atom, expr)
+ end
+
+ # Constructs an accelerator rule. A rule is a matching expression and the
+ # code that should be executed once the expression could be bound to a
+ # parser.
+ #
+ # Example:
+ # Accelerator.rule(Accelerator.any) { Parslet.match('.') }
+ #
+ def rule expression, &action
+ [expression, action]
+ end
+
+ # Given a parslet atom and a set of rules, tries to match the rules
+ # recursively through the parslet atom. Once a rule could be matched,
+ # its action block will be called.
+ #
+ # Example:
+ # quote = str('"')
+ # parser = quote >> (quote.absent? >> any).repeat >> quote
+ #
+ # A = Accelerator # for making what follows a bit shorter
+ # optimized_parser = A.apply(parser,
+ # A.rule( (A.str(:x).absent? >> A.any).repeat ) { GobbleUp.new(x) })
+ #
+ # optimized_parser.parse('"Parsing is now fully optimized! (tm)"')
+ #
+ # @param atom [Parslet::Atoms::Base] a parser to optimize
+ # @param *rules [Parslet::Accelerator::Rule] rules produced by .rule
+ # @return [Parslet::Atoms::Base] optimized parser
+ #
+ def apply atom, *rules
+ Application.new(atom, rules).call
+ end
+end
+
+require 'parslet/accelerator/engine'
+require 'parslet/accelerator/application'
\ No newline at end of file
diff --git a/lib/parslet/accelerator/application.rb b/lib/parslet/accelerator/application.rb
new file mode 100644
index 0000000..8015cc8
--- /dev/null
+++ b/lib/parslet/accelerator/application.rb
@@ -0,0 +1,62 @@
+
+# @api private
+module Parslet::Accelerator
+ class Application
+ def initialize atom, rules
+ @atom = atom
+ @rules = rules
+ end
+
+ def call
+ @atom.accept(self)
+ end
+
+ def visit_parser(root)
+ transform root.accept(self)
+ end
+ def visit_entity(name, block)
+ transform Parslet::Atoms::Entity.new(name) { block.call.accept(self) }
+ end
+ def visit_named(name, atom)
+ transform Parslet::Atoms::Named.new(atom.accept(self), name)
+ end
+ def visit_repetition(tag, min, max, atom)
+ transform Parslet::Atoms::Repetition.new(atom.accept(self), min, max, tag)
+ end
+ def visit_alternative(alternatives)
+ transform Parslet::Atoms::Alternative.new(
+ *alternatives.map { |atom| atom.accept(self) })
+ end
+ def visit_sequence(sequence)
+ transform Parslet::Atoms::Sequence.new(
+ *sequence.map { |atom| atom.accept(self) })
+ end
+ def visit_lookahead(positive, atom)
+ transform Parslet::Atoms::Lookahead.new(atom, positive)
+ end
+ def visit_re(regexp)
+ transform Parslet::Atoms::Re.new(regexp)
+ end
+ def visit_str(str)
+ transform Parslet::Atoms::Str.new(str)
+ end
+
+ def transform atom
+ @rules.each do |expr, action|
+ # Try and match each rule in turn
+ binding = Parslet::Accelerator.match(atom, expr)
+ if binding
+ # On a successful match, allow the rule action to transform the
+ # parslet into something new.
+ ctx = Parslet::Context.new(binding)
+ return ctx.instance_eval(&action)
+ end
+ end # rules.each
+
+ # If no rule matches, this is the fallback - a clean new parslet atom.
+ return atom
+ end
+ end
+end
+
+require 'parslet/context'
\ No newline at end of file
diff --git a/lib/parslet/accelerator/engine.rb b/lib/parslet/accelerator/engine.rb
new file mode 100644
index 0000000..1f08181
--- /dev/null
+++ b/lib/parslet/accelerator/engine.rb
@@ -0,0 +1,112 @@
+
+require 'parslet/atoms/visitor'
+
+module Parslet::Accelerator
+ # @api private
+ class Apply
+ def initialize(engine, expr)
+ @engine = engine
+ @expr = expr
+ end
+
+ def visit_parser(root)
+ false
+ end
+ def visit_entity(name, block)
+ false
+ end
+ def visit_named(name, atom)
+ match(:as) do |key|
+ @engine.try_bind(key, name)
+ end
+ end
+ def visit_repetition(tag, min, max, atom)
+ match(:rep) do |e_min, e_max, expr|
+ e_min == min && e_max == max && @engine.match(atom, expr)
+ end
+ end
+ def visit_alternative(alternatives)
+ match(:alt) do |*expressions|
+ return false if alternatives.size != expressions.size
+
+ alternatives.zip(expressions).all? do |atom, expr|
+ @engine.match(atom, expr)
+ end
+ end
+ end
+ def visit_sequence(sequence)
+ match(:seq) do |*expressions|
+ return false if sequence.size != expressions.size
+
+ sequence.zip(expressions).all? do |atom, expr|
+ @engine.match(atom, expr)
+ end
+ end
+ end
+ def visit_lookahead(positive, atom)
+ match(:absent) do |expr|
+ return positive == false && @engine.match(atom, expr)
+ end
+ match(:present) do |expr|
+ return positive == true && @engine.match(atom, expr)
+ end
+ end
+ def visit_re(regexp)
+ match(:re) do |*bind_conditions|
+ bind_conditions.all? { |bind_cond|
+ @engine.try_bind(bind_cond, regexp) }
+ end
+ end
+ def visit_str(str)
+ match(:str) do |*bind_conditions|
+ bind_conditions.all? { |bind_cond|
+ @engine.try_bind(bind_cond, str) }
+ end
+ end
+
+ def match(type_tag)
+ expr_tag = @expr.type
+ if expr_tag == type_tag
+ yield *@expr.args
+ end
+ end
+ end
+
+ # @api private
+ class Engine
+ attr_reader :bindings
+
+ def initialize
+ @bindings = {}
+ end
+
+ def match(atom, expr)
+ atom.accept(
+ Apply.new(self, expr))
+ end
+
+ def try_bind(variable, value)
+ if bound? variable
+ return value == lookup(variable)
+ else
+ case variable
+ when Symbol
+ bind(variable, value)
+ else
+ # This does not look like a variable - let's try matching it against
+ # the value:
+ variable === value
+ end
+ end
+ end
+ def bound? var
+ @bindings.has_key? var
+ end
+ def lookup var
+ @bindings[var]
+ end
+ def bind var, val
+ @bindings[var] = val
+ end
+ end
+end
\ No newline at end of file
diff --git a/lib/parslet/atoms.rb b/lib/parslet/atoms.rb
index 91e52e3..5a4222c 100644
--- a/lib/parslet/atoms.rb
+++ b/lib/parslet/atoms.rb
@@ -30,5 +30,6 @@ module Parslet::Atoms
require 'parslet/atoms/capture'
require 'parslet/atoms/dynamic'
require 'parslet/atoms/scope'
+ require 'parslet/atoms/infix'
end
diff --git a/lib/parslet/atoms/base.rb b/lib/parslet/atoms/base.rb
index 4687905..e145ecf 100644
--- a/lib/parslet/atoms/base.rb
+++ b/lib/parslet/atoms/base.rb
@@ -36,7 +36,7 @@ class Parslet::Atoms::Base
# Cheating has not paid off. Now pay the cost: Rerun the parse,
# gathering error information in the process.
reporter = options[:reporter] || Parslet::ErrorReporter::Tree.new
- source.pos = 0
+ source.bytepos = 0
success, value = setup_and_apply(source, reporter, !options[:prefix])
fail "Assertion failed: success was true when parsing with reporter" \
@@ -78,7 +78,7 @@ class Parslet::Atoms::Base
# @param consume_all [Boolean] true if the current parse must consume
# all input by itself.
def apply(source, context, consume_all=false)
- old_pos = source.pos
+ old_pos = source.bytepos
success, value = result = context.try_with_cache(self, source, consume_all)
@@ -91,7 +91,7 @@ class Parslet::Atoms::Base
offending_input = source.consume(10)
# Rewind input (as happens always in error case)
- source.pos = old_pos
+ source.bytepos = old_pos
return context.err_at(
self,
@@ -106,7 +106,7 @@ class Parslet::Atoms::Base
end
# We only reach this point if the parse has failed. Rewind the input.
- source.pos = old_pos
+ source.bytepos = old_pos
return result
end
diff --git a/lib/parslet/atoms/context.rb b/lib/parslet/atoms/context.rb
index 231f5e0..7167a74 100644
--- a/lib/parslet/atoms/context.rb
+++ b/lib/parslet/atoms/context.rb
@@ -24,14 +24,14 @@ module Parslet::Atoms
# advance the input pos by the same amount of bytes.
#
def try_with_cache(obj, source, consume_all)
- beg = source.pos
+ beg = source.bytepos
# Not in cache yet? Return early.
unless entry = lookup(obj, beg)
result = obj.try(source, self, consume_all)
if obj.cached?
- set obj, beg, [result, source.pos-beg]
+ set obj, beg, [result, source.bytepos-beg]
end
return result
@@ -43,7 +43,7 @@ module Parslet::Atoms
# The data we're skipping here has been read before. (since it is in
# the cache) PLUS the actual contents are not interesting anymore since
# we know obj matches at beg. So skip reading.
- source.pos = beg + advance
+ source.bytepos = beg + advance
return result
end
@@ -81,11 +81,15 @@ module Parslet::Atoms
end
private
+ # NOTE These methods use #object_id directly, since that seems to bring the
+ # most performance benefit. This is a hot spot; going through
+ # Atoms::Base#hash doesn't yield as much.
+ #
def lookup(obj, pos)
- @cache[pos][obj]
+ @cache[pos][obj.object_id]
end
def set(obj, pos, val)
- @cache[pos][obj] = val
+ @cache[pos][obj.object_id] = val
end
end
end
\ No newline at end of file
diff --git a/lib/parslet/atoms/infix.rb b/lib/parslet/atoms/infix.rb
new file mode 100644
index 0000000..6a08ca2
--- /dev/null
+++ b/lib/parslet/atoms/infix.rb
@@ -0,0 +1,121 @@
+class Parslet::Atoms::Infix < Parslet::Atoms::Base
+ attr_reader :element, :operations
+
+ def initialize(element, operations)
+ super()
+
+ @element = element
+ @operations = operations
+ end
+
+ def try(source, context, consume_all)
+ return catch_error {
+ return succ(
+ produce_tree(
+ precedence_climb(source, context, consume_all)))
+ }
+ end
+
+ # Turns an array of the form ['1', '+', ['2', '*', '3']] into a hash that
+ # reflects the same structure.
+ #
+ def produce_tree(ary)
+ return ary unless ary.kind_of? Array
+
+ left = ary.shift
+
+ until ary.empty?
+ op, right = ary.shift(2)
+
+ # p [left, op, right]
+
+ if right.kind_of? Array
+ # Subexpression -> Subhash
+ left = {l: left, o: op, r: produce_tree(right)}
+ else
+ left = {l: left, o: op, r: right}
+ end
+ end
+
+ left
+ end
+
+ # A precedence climbing algorithm married to parslet, as described here
+ # http://eli.thegreenplace.net/2012/08/02/parsing-expressions-by-precedence-climbing/
+ #
+ # @note Error handling in this routine is done by throwing :error and
+ # as a value the error to return to parslet. This avoids cluttering
+ # the recursion logic here with parslet error handling.
+ #
+ def precedence_climb(source, context, consume_all, current_prec=1, needs_element=false)
+ result = []
+
+ # To even begin parsing an arithmetic expression, there needs to be
+ # at least one @element.
+ success, value = @element.apply(source, context, false)
+
+ unless success
+ abort context.err(self, source, "#{@element.inspect} was expected", [value])
+ end
+
+ result << flatten(value, true)
+
+ # Loop until we fail on operator matching or until input runs out.
+ loop do
+ op_pos = source.bytepos
+ op_match, prec, assoc = match_operation(source, context, false)
+
+ # If no operator could be matched here, one of several cases
+ # applies:
+ #
+ # - end of file
+ # - end of expression
+ # - syntax error
+ #
+ # We abort matching the expression here.
+ break unless op_match
+
+ if prec >= current_prec
+ next_prec = (assoc == :left) ? prec+1 : prec
+
+ result << op_match
+ result << precedence_climb(
+ source, context, consume_all, next_prec, true)
+ else
+ source.bytepos = op_pos
+ return unwrap(result)
+ end
+ end
+
+ return unwrap(result)
+ end
+
+ def unwrap expr
+ expr.size == 1 ? expr.first : expr
+ end
+
+ def match_operation(source, context, consume_all)
+ errors = []
+ @operations.each do |op_atom, prec, assoc|
+ success, value = op_atom.apply(source, context, consume_all)
+ return flatten(value, true), prec, assoc if success
+
+ # assert: this was in fact an error, accumulate
+ errors << value
+ end
+
+ return nil
+ end
+
+ def abort(error)
+ throw :error, error
+ end
+ def catch_error
+ catch(:error) { yield }
+ end
+
+ def to_s_inner(prec)
+ ops = @operations.map { |o, _, _| o.inspect }.join(', ')
+ "infix_expression(#{@element.inspect}, [#{ops}])"
+ end
+end
\ No newline at end of file
diff --git a/lib/parslet/atoms/lookahead.rb b/lib/parslet/atoms/lookahead.rb
index d50a1b9..db44f7a 100644
--- a/lib/parslet/atoms/lookahead.rb
+++ b/lib/parslet/atoms/lookahead.rb
@@ -22,22 +22,23 @@ class Parslet::Atoms::Lookahead < Parslet::Atoms::Base
end
def try(source, context, consume_all)
- pos = source.pos
+ rewind_pos = source.bytepos
+ error_pos = source.pos
success, value = bound_parslet.apply(source, context, consume_all)
if positive
return succ(nil) if success
- return context.err_at(self, source, @error_msgs[:positive], pos)
+ return context.err_at(self, source, @error_msgs[:positive], error_pos)
else
return succ(nil) unless success
- return context.err_at(self, source, @error_msgs[:negative], pos)
+ return context.err_at(self, source, @error_msgs[:negative], error_pos)
end
# This is probably the only parslet that rewinds its input in #try.
# Lookaheads NEVER consume their input, even on success, that's why.
ensure
- source.pos = pos
+ source.bytepos = rewind_pos
end
precedence LOOKAHEAD
diff --git a/lib/parslet/atoms/re.rb b/lib/parslet/atoms/re.rb
index b6c2ea9..75b0ac1 100644
--- a/lib/parslet/atoms/re.rb
+++ b/lib/parslet/atoms/re.rb
@@ -21,7 +21,7 @@ class Parslet::Atoms::Re < Parslet::Atoms::Base
end
def try(source, context, consume_all)
- return succ(source.consume(1)) if source.matches?(re)
+ return succ(source.consume(1)) if source.matches?(@re)
# No string could be read
return context.err(self, source, @error_msgs[:premature]) \
diff --git a/lib/parslet/atoms/repetition.rb b/lib/parslet/atoms/repetition.rb
index b45822f..dde129f 100644
--- a/lib/parslet/atoms/repetition.rb
+++ b/lib/parslet/atoms/repetition.rb
@@ -11,6 +11,11 @@ class Parslet::Atoms::Repetition < Parslet::Atoms::Base
def initialize(parslet, min, max, tag=:repetition)
super()
+ raise ArgumentError,
+ "Asking for zero repetitions of a parslet. (#{parslet.inspect} repeating #{min},#{max})" \
+ if max == 0
+
+
@parslet = parslet
@min, @max = min, max
@tag = tag
@@ -42,7 +47,7 @@ class Parslet::Atoms::Repetition < Parslet::Atoms::Base
# Last attempt to match parslet was a failure, failure reason in break_on.
# Greedy matcher has produced a failure. Check if occ (which will
- # contain the number of sucesses) is >= min.
+ # contain the number of successes) is >= min.
return context.err_at(
self,
source,
diff --git a/lib/parslet/atoms/str.rb b/lib/parslet/atoms/str.rb
index ade63dc..a55061f 100644
--- a/lib/parslet/atoms/str.rb
+++ b/lib/parslet/atoms/str.rb
@@ -10,6 +10,7 @@ class Parslet::Atoms::Str < Parslet::Atoms::Base
super()
@str = str.to_s
+ @pat = Regexp.new(Regexp.escape(str))
@len = str.size
@error_msgs = {
:premature => "Premature end of input",
@@ -18,7 +19,7 @@ class Parslet::Atoms::Str < Parslet::Atoms::Base
end
def try(source, context, consume_all)
- return succ(source.consume(@len)) if source.matches?(str)
+ return succ(source.consume(@len)) if source.matches?(@pat)
# Input ending early:
return context.err(self, source, @error_msgs[:premature]) \
diff --git a/lib/parslet/transform/context.rb b/lib/parslet/context.rb
similarity index 90%
rename from lib/parslet/transform/context.rb
rename to lib/parslet/context.rb
index 3a974b6..a1d4a47 100644
--- a/lib/parslet/transform/context.rb
+++ b/lib/parslet/context.rb
@@ -10,12 +10,15 @@ require 'blankslate'
# a # => :b
# end
#
-class Parslet::Transform::Context < BlankSlate
+# @api private
+class Parslet::Context < BlankSlate
reveal :methods
reveal :respond_to?
reveal :inspect
reveal :to_s
reveal :instance_variable_set
+
+ include Parslet
def meta_def(name, &body)
metaclass = class <<self; self; end
diff --git a/lib/parslet/graphviz.rb b/lib/parslet/graphviz.rb
new file mode 100644
index 0000000..6d6cd75
--- /dev/null
+++ b/lib/parslet/graphviz.rb
@@ -0,0 +1,97 @@
+
+# Paints a graphviz graph of your parser.
+
+begin
+ require 'ruby-graphviz'
+rescue LoadError
+ puts "Please install the 'ruby-graphviz' gem first."
+ fail
+end
+
+require 'set'
+require 'parslet/atoms/visitor'
+
+module Parslet
+ class GraphvizVisitor
+ def initialize g
+ @graph = g
+ @known_links = Set.new
+ @visited = Set.new
+ end
+
+ attr_reader :parent
+
+ def visit_parser(root)
+ recurse root, node('parser')
+ end
+ def visit_entity(name, block)
+ s = node(name)
+
+ downwards s
+
+ return if @visited.include?(name)
+ @visited << name
+
+ recurse block.call, s
+ end
+ def visit_named(name, atom)
+ recurse atom, parent
+ end
+ def visit_repetition(tag, min, max, atom)
+ recurse atom, parent
+ end
+ def visit_alternative(alternatives)
+ p = parent
+ alternatives.each do |atom|
+ recurse atom, p
+ end
+ end
+ def visit_sequence(sequence)
+ p = parent
+ sequence.each do |atom|
+ recurse atom, p
+ end
+ end
+ def visit_lookahead(positive, atom)
+ recurse atom, parent
+ end
+ def visit_re(regexp)
+ # downwards node(regexp.object_id, label: escape("re(#{regexp.inspect})"))
+ end
+ def visit_str(str)
+ # downwards node(str.object_id, label: escape("#{str.inspect}"))
+ end
+
+ def escape str
+ str.gsub('"', "'")
+ end
+ def node name, opts={}
+ @graph.add_nodes name.to_s, opts
+ end
+ def downwards child
+ if @parent && !@known_links.include?([@parent, child])
+ @graph.add_edges(@parent, child)
+ @known_links << [@parent, child]
+ end
+ end
+ def recurse node, current
+ @parent = current
+ node.accept(self)
+ end
+ end
+
+ module Graphable
+ def graph opts
+ g = GraphViz.new(:G, type: :digraph)
+ visitor = GraphvizVisitor.new(g)
+
+ new.accept(visitor)
+
+ g.output opts
+ end
+ end
+
+ class Parser # reopen for introducing the .graph method
+ extend Graphable
+ end
+end
\ No newline at end of file
diff --git a/lib/parslet/pattern.rb b/lib/parslet/pattern.rb
index 476d902..6efc5d0 100644
--- a/lib/parslet/pattern.rb
+++ b/lib/parslet/pattern.rb
@@ -55,7 +55,7 @@ class Parslet::Pattern
return element_match_ary_single(tree, exp, bindings)
else
# If elements match exactly, then that is good enough in all cases
- return true if tree == exp
+ return true if exp === tree
# If exp is a bind variable: Check if the binding matches
if exp.respond_to?(:can_bind?) && exp.can_bind?(tree)
diff --git a/lib/parslet/position.rb b/lib/parslet/position.rb
new file mode 100644
index 0000000..3c93812
--- /dev/null
+++ b/lib/parslet/position.rb
@@ -0,0 +1,21 @@
+
+# Encapsules the concept of a position inside a string.
+#
+class Parslet::Position
+ attr_reader :bytepos
+
+ include Comparable
+
+ def initialize string, bytepos
+ @string = string
+ @bytepos = bytepos
+ end
+
+ def charpos
+ @string.byteslice(0, @bytepos).size
+ end
+
+ def <=> b
+ self.bytepos <=> b.bytepos
+ end
+end
\ No newline at end of file
diff --git a/lib/parslet/rig/rspec.rb b/lib/parslet/rig/rspec.rb
index 0ce2175..71f1aa7 100644
--- a/lib/parslet/rig/rspec.rb
+++ b/lib/parslet/rig/rspec.rb
@@ -1,6 +1,14 @@
RSpec::Matchers.define(:parse) do |input, opts|
as = block = nil
result = trace = nil
+
+ unless self.respond_to? :failure_message # if RSpec 2.x
+ class << self
+ alias_method :failure_message, :failure_message_for_should
+ alias_method :failure_message_when_negated, :failure_message_for_should_not
+ end
+ end
+
match do |parser|
begin
result = parser.parse(input)
@@ -13,7 +21,7 @@ RSpec::Matchers.define(:parse) do |input, opts|
end
end
- failure_message_for_should do |is|
+ failure_message do |is|
if block
"expected output of parsing #{input.inspect}" <<
" with #{is.inspect} to meet block conditions, but it didn't"
@@ -29,7 +37,7 @@ RSpec::Matchers.define(:parse) do |input, opts|
end
end
- failure_message_for_should_not do |is|
+ failure_message_when_negated do |is|
if block
"expected output of parsing #{input.inspect} with #{is.inspect} not to meet block conditions, but it did"
else
diff --git a/lib/parslet/slice.rb b/lib/parslet/slice.rb
index 3e2dd46..f8ad518 100644
--- a/lib/parslet/slice.rb
+++ b/lib/parslet/slice.rb
@@ -23,17 +23,23 @@
# delegation, we opt for a partial emulation that gets the job done.
#
class Parslet::Slice
- attr_reader :str, :offset
+ attr_reader :str
+ attr_reader :position
attr_reader :line_cache
# Construct a slice using a string, an offset and an optional line cache.
# The line cache should be able to answer to the #line_and_column message.
#
- def initialize(string, offset, line_cache=nil)
- @str, @offset = string, offset
+ def initialize(position, string, line_cache=nil)
+ @position = position
+ @str = string
@line_cache = line_cache
end
+ def offset
+ @position.charpos
+ end
+
# Compares slices to other slices or strings.
#
def == other
@@ -57,7 +63,7 @@ class Parslet::Slice
# as the one of this slice.
#
def +(other)
- self.class.new(str + other.to_s, offset, line_cache)
+ self.class.new(@position, str + other.to_s, line_cache)
end
# Returns a <line, column> tuple referring to the original input.
@@ -66,7 +72,7 @@ class Parslet::Slice
raise ArgumentError, "No line cache was given, cannot infer line and column." \
unless line_cache
- line_cache.line_and_column(self.offset)
+ line_cache.line_and_column(@position.bytepos)
end
@@ -96,6 +102,7 @@ class Parslet::Slice
# Prints the slice as <code>"string"@offset</code>.
def inspect
+
str.inspect << "@#{offset}"
end
end
\ No newline at end of file
diff --git a/lib/parslet/source.rb b/lib/parslet/source.rb
index 3ec14a9..0f5a20e 100644
--- a/lib/parslet/source.rb
+++ b/lib/parslet/source.rb
@@ -1,6 +1,8 @@
require 'stringio'
+require 'strscan'
+require 'parslet/position'
require 'parslet/source/line_cache'
module Parslet
@@ -8,22 +10,28 @@ module Parslet
#
class Source
def initialize(str)
- raise ArgumentError unless str.respond_to?(:to_str)
-
- @pos = 0
- @str = str
-
+ raise(
+ ArgumentError,
+ "Must construct Source with a string like object."
+ ) unless str.respond_to?(:to_str)
+
+ @str = StringScanner.new(str)
+
+ # maps 1 => /./m, 2 => /../m, etc...
+ @re_cache = Hash.new { |h,k|
+ h[k] = /(.|$){#{k}}/m }
+
@line_cache = LineCache.new
- @line_cache.scan_for_line_endings(0, @str)
+ @line_cache.scan_for_line_endings(0, str)
end
# Checks if the given pattern matches at the current input position.
#
- # @param pattern [Regexp, String] pattern to check for
+ # @param pattern [Regexp] pattern to check for
# @return [Boolean] true if the pattern matches at #pos
#
def matches?(pattern)
- @str.index(pattern, @pos) == @pos
+ @str.match?(pattern)
end
alias match matches?
@@ -31,32 +39,58 @@ module Parslet
# input.
#
def consume(n)
- slice_str = @str.slice(@pos, n)
+ position = self.pos
+ slice_str = @str.scan(@re_cache[n])
slice = Parslet::Slice.new(
- slice_str,
- pos,
+ position,
+ slice_str,
@line_cache)
-
- @pos += slice_str.size
+
return slice
end
# Returns how many chars remain in the input.
#
def chars_left
- @str.size - @pos
+ @str.rest_size
+ end
+
+ # Returns how many chars there are between current position and the
+ # string given. If the string given doesn't occur in the source, then
+ # the remaining chars (#chars_left) are returned.
+ #
+ # @return [Fixnum] count of chars until str or #chars_left
+ #
+ def chars_until str
+ slice_str = @str.check_until(Regexp.new(Regexp.escape(str)))
+ return chars_left unless slice_str
+ return slice_str.size - str.size
end
# Position of the parse as a character offset into the original string.
- # @note: Encodings...
- attr_accessor :pos
+ #
+ # @note Please be aware of encodings at this point.
+ #
+ def pos
+ Position.new(@str.string, @str.pos)
+ end
+ def bytepos
+ @str.pos
+ end
+
+ # @note Please be aware of encodings at this point.
+ #
+ def bytepos=(n)
+ @str.pos = n
+ rescue RangeError
+ end
# Returns a <line, column> tuple for the given position. If no position is
# given, line/column information is returned for the current position
# given by #pos.
#
def line_and_column(position=nil)
- @line_cache.line_and_column(position || self.pos)
+ @line_cache.line_and_column(position || self.bytepos)
end
end
-end
\ No newline at end of file
+end
diff --git a/lib/parslet/source/line_cache.rb b/lib/parslet/source/line_cache.rb
index 4699162..e8c59b9 100644
--- a/lib/parslet/source/line_cache.rb
+++ b/lib/parslet/source/line_cache.rb
@@ -12,9 +12,11 @@ class Parslet::Source
@line_ends.extend RangeSearch
end
- # Returns a <line, column> tuple for the given input position.
- #
+ # Returns a <line, column> tuple for the given input position. Input
+ # position must be given as byte offset into original string.
+ #
def line_and_column(pos)
+ pos = pos.bytepos if pos.respond_to? :bytepos
eol_idx = @line_ends.lbound(pos)
if eol_idx
@@ -32,22 +34,23 @@ class Parslet::Source
def scan_for_line_endings(start_pos, buf)
return unless buf
- return unless buf.index("\n")
- cur = -1
- # If we have already read part or all of buf, we already know about
- # line ends in that portion. remove it and correct cur (search index)
+ buf = StringScanner.new(buf)
+ return unless buf.exist?(/\n/)
+
+ ## If we have already read part or all of buf, we already know about
+ ## line ends in that portion. remove it and correct cur (search index)
if @last_line_end && start_pos < @last_line_end
# Let's not search the range from start_pos to last_line_end again.
- cur = @last_line_end - start_pos -1
+ buf.pos = @last_line_end - start_pos
end
- # Scan the string for line endings; store the positions of all endings
- # in @line_ends.
- while buf && cur = buf.index("\n", cur+1)
- @last_line_end = (start_pos + cur+1)
+ ## Scan the string for line endings; store the positions of all endings
+ ## in @line_ends.
+ while buf.skip_until(/\n/)
+ @last_line_end = start_pos + buf.pos
@line_ends << @last_line_end
- end
+ end
end
end
diff --git a/lib/parslet/transform.rb b/lib/parslet/transform.rb
index e3ee4ac..b04fa29 100644
--- a/lib/parslet/transform.rb
+++ b/lib/parslet/transform.rb
@@ -116,8 +116,6 @@ class Parslet::Transform
# context?
include Parslet
- autoload :Context, 'parslet/transform/context'
-
class << self
# FIXME: Only do this for subclasses?
include Parslet
@@ -233,4 +231,6 @@ class Parslet::Transform
def recurse_array(ary, ctx)
ary.map { |elt| apply(elt, ctx) }
end
-end
\ No newline at end of file
+end
+
+require 'parslet/context'
\ No newline at end of file
diff --git a/metadata.yml b/metadata.yml
index 7c20da6..c761d5a 100644
--- a/metadata.yml
+++ b/metadata.yml
@@ -1,160 +1,29 @@
--- !ruby/object:Gem::Specification
name: parslet
version: !ruby/object:Gem::Version
- version: 1.5.0
- prerelease:
+ version: 1.6.1
platform: ruby
authors:
- Kaspar Schiess
autorequire:
bindir: bin
cert_chain: []
-date: 2012-12-27 00:00:00.000000000 Z
+date: 2014-05-22 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
name: blankslate
- prerelease: false
requirement: !ruby/object:Gem::Requirement
requirements:
- - - ~>
+ - - "~>"
- !ruby/object:Gem::Version
version: '2.0'
- none: false
type: :runtime
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ~>
- - !ruby/object:Gem::Version
- version: '2.0'
- none: false
-- !ruby/object:Gem::Dependency
- name: rspec
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: flexmock
prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
version_requirements: !ruby/object:Gem::Requirement
requirements:
- - - ! '>='
+ - - "~>"
- !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: rdoc
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: sdoc
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: guard
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: guard-rspec
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: rb-fsevent
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
-- !ruby/object:Gem::Dependency
- name: growl
- prerelease: false
- requirement: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
- type: :development
- version_requirements: !ruby/object:Gem::Requirement
- requirements:
- - - ! '>='
- - !ruby/object:Gem::Version
- version: '0'
- none: false
+ version: '2.0'
description:
email: kaspar.schiess at absurd.li
executables: []
@@ -164,44 +33,9 @@ extra_rdoc_files:
files:
- HISTORY.txt
- LICENSE
-- Rakefile
- README
-- lib/parslet/atoms/alternative.rb
-- lib/parslet/atoms/base.rb
-- lib/parslet/atoms/can_flatten.rb
-- lib/parslet/atoms/capture.rb
-- lib/parslet/atoms/context.rb
-- lib/parslet/atoms/dsl.rb
-- lib/parslet/atoms/dynamic.rb
-- lib/parslet/atoms/entity.rb
-- lib/parslet/atoms/lookahead.rb
-- lib/parslet/atoms/named.rb
-- lib/parslet/atoms/re.rb
-- lib/parslet/atoms/repetition.rb
-- lib/parslet/atoms/scope.rb
-- lib/parslet/atoms/sequence.rb
-- lib/parslet/atoms/str.rb
-- lib/parslet/atoms/visitor.rb
-- lib/parslet/atoms.rb
-- lib/parslet/cause.rb
-- lib/parslet/convenience.rb
-- lib/parslet/error_reporter/deepest.rb
-- lib/parslet/error_reporter/tree.rb
-- lib/parslet/error_reporter.rb
-- lib/parslet/export.rb
-- lib/parslet/expression/treetop.rb
-- lib/parslet/expression.rb
-- lib/parslet/parser.rb
-- lib/parslet/pattern/binding.rb
-- lib/parslet/pattern.rb
-- lib/parslet/rig/rspec.rb
-- lib/parslet/scope.rb
-- lib/parslet/slice.rb
-- lib/parslet/source/line_cache.rb
-- lib/parslet/source.rb
-- lib/parslet/transform/context.rb
-- lib/parslet/transform.rb
-- lib/parslet.rb
+- Rakefile
+- example/big.erb
- example/boolean_algebra.rb
- example/calc.rb
- example/capture.rb
@@ -219,6 +53,7 @@ files:
- example/minilisp.rb
- example/modularity.rb
- example/nested_errors.rb
+- example/optimized_erb.rb
- example/output/boolean_algebra.out
- example/output/calc.out
- example/output/capture.out
@@ -238,7 +73,9 @@ files:
- example/output/minilisp.out
- example/output/modularity.out
- example/output/nested_errors.out
+- example/output/optimized_erb.out
- example/output/parens.out
+- example/output/prec_calc.out
- example/output/readme.out
- example/output/scopes.out
- example/output/seasons.out
@@ -246,6 +83,7 @@ files:
- example/output/simple_xml.out
- example/output/string_parser.out
- example/parens.rb
+- example/prec_calc.rb
- example/readme.rb
- example/scopes.rb
- example/seasons.rb
@@ -254,31 +92,73 @@ files:
- example/simple_xml.rb
- example/string_parser.rb
- example/test.lit
+- lib/parslet.rb
+- lib/parslet/accelerator.rb
+- lib/parslet/accelerator/application.rb
+- lib/parslet/accelerator/engine.rb
+- lib/parslet/atoms.rb
+- lib/parslet/atoms/alternative.rb
+- lib/parslet/atoms/base.rb
+- lib/parslet/atoms/can_flatten.rb
+- lib/parslet/atoms/capture.rb
+- lib/parslet/atoms/context.rb
+- lib/parslet/atoms/dsl.rb
+- lib/parslet/atoms/dynamic.rb
+- lib/parslet/atoms/entity.rb
+- lib/parslet/atoms/infix.rb
+- lib/parslet/atoms/lookahead.rb
+- lib/parslet/atoms/named.rb
+- lib/parslet/atoms/re.rb
+- lib/parslet/atoms/repetition.rb
+- lib/parslet/atoms/scope.rb
+- lib/parslet/atoms/sequence.rb
+- lib/parslet/atoms/str.rb
+- lib/parslet/atoms/visitor.rb
+- lib/parslet/cause.rb
+- lib/parslet/context.rb
+- lib/parslet/convenience.rb
+- lib/parslet/error_reporter.rb
+- lib/parslet/error_reporter/deepest.rb
+- lib/parslet/error_reporter/tree.rb
+- lib/parslet/export.rb
+- lib/parslet/expression.rb
+- lib/parslet/expression/treetop.rb
+- lib/parslet/graphviz.rb
+- lib/parslet/parser.rb
+- lib/parslet/pattern.rb
+- lib/parslet/pattern/binding.rb
+- lib/parslet/position.rb
+- lib/parslet/rig/rspec.rb
+- lib/parslet/scope.rb
+- lib/parslet/slice.rb
+- lib/parslet/source.rb
+- lib/parslet/source/line_cache.rb
+- lib/parslet/transform.rb
homepage: http://kschiess.github.com/parslet
-licenses: []
+licenses:
+- MIT
+metadata: {}
post_install_message:
rdoc_options:
-- --main
+- "--main"
- README
require_paths:
- lib
required_ruby_version: !ruby/object:Gem::Requirement
requirements:
- - - ! '>='
+ - - ">="
- !ruby/object:Gem::Version
version: '0'
- none: false
required_rubygems_version: !ruby/object:Gem::Requirement
requirements:
- - - ! '>='
+ - - ">="
- !ruby/object:Gem::Version
version: '0'
- none: false
requirements: []
rubyforge_project:
-rubygems_version: 1.8.24
+rubygems_version: 2.2.2
signing_key:
-specification_version: 3
+specification_version: 4
summary: Parser construction library with great error reporting in Ruby.
test_files: []
has_rdoc:
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-ruby-extras/ruby-parslet.git
More information about the Pkg-ruby-extras-commits
mailing list