[DRE-commits] [librarian-puppet] 03/153: Issue #210 Use forgeapi.puppetlabs.com and API v3
Stig Sandbeck Mathisen
ssm at debian.org
Wed Jun 1 20:30:32 UTC 2016
This is an automated email from the git hooks/post-receive script.
ssm pushed a commit to branch master
in repository librarian-puppet.
commit f7b59976e25055e14603d4db5624416a99d289b4
Author: Carlos Sanchez <csanchez at maestrodev.com>
Date: Mon Jun 9 11:22:51 2014 +0200
Issue #210 Use forgeapi.puppetlabs.com and API v3
puppet_forge requires ruby 1.9 so we do too
Use API v3 only for official puppetlabs repo
Puppet 2 can't handle the 302 returned by the https url, so stick to http
---
.travis.yml | 1 -
Changelog.md | 4 +-
features/install/forge.feature | 8 +-
lib/librarian/puppet/source/forge.rb | 19 ++++-
lib/librarian/puppet/source/forge/repo.rb | 107 +++++++--------------------
lib/librarian/puppet/source/forge/repo_v1.rb | 102 +++++++++++++++++++++++++
lib/librarian/puppet/source/forge/repo_v3.rb | 53 +++++++++++++
lib/librarian/puppet/version.rb | 2 +-
librarian-puppet.gemspec | 1 +
9 files changed, 207 insertions(+), 90 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 1686d50..5bab58f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,7 +2,6 @@ rvm:
- 2.1.0
- 2.0.0
- 1.9.3
- - 1.8.7
notifications:
email:
- carlos at apache.org
diff --git a/Changelog.md b/Changelog.md
index 4b1ee9f..b5cede7 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,7 +1,9 @@
## Changelog
-### 1.0.4
+### 1.1.0
+ * [Issue #210](https://github.com/rodjek/librarian-puppet/issues/210) Use forgeapi.puppetlabs.com and API v3
+ * Accesing the v3 API requires Ruby 1.9 due to the puppet_forge library used
### 1.0.3
diff --git a/features/install/forge.feature b/features/install/forge.feature
index 614eb2a..2802adc 100644
--- a/features/install/forge.feature
+++ b/features/install/forge.feature
@@ -4,13 +4,13 @@ Feature: cli/install/forge
Scenario: Installing a module and its dependencies
Given a file named "Puppetfile" with:
"""
- forge "http://forge.puppetlabs.com"
+ forge "https://forgeapi.puppetlabs.com"
- mod 'puppetlabs/ntp', '3.0.3'
+ mod 'puppetlabs/ntp'
"""
When I run `librarian-puppet install`
Then the exit status should be 0
- And the file "modules/ntp/Modulefile" should match /name *'puppetlabs-ntp'/
+ And the file "modules/ntp/metadata.json" should match /"name": "puppetlabs-ntp"/
And the file "modules/stdlib/Modulefile" should match /name *'puppetlabs-stdlib'/
Scenario: Installing an exact version of a module
@@ -117,7 +117,7 @@ Feature: cli/install/forge
"""
When I run `librarian-puppet install`
Then the exit status should be 1
- And the output should contain "Unable to find module 'puppetlabs/xxxxx' on http://forge.puppetlabs.com"
+ And the output should contain "Unable to find module 'puppetlabs/xxxxx' on https://forgeapi.puppetlabs.com"
Scenario: Install a module with conflicts
Given a file named "Puppetfile" with:
diff --git a/lib/librarian/puppet/source/forge.rb b/lib/librarian/puppet/source/forge.rb
index d9bb765..b432700 100644
--- a/lib/librarian/puppet/source/forge.rb
+++ b/lib/librarian/puppet/source/forge.rb
@@ -1,6 +1,7 @@
require 'uri'
require 'librarian/puppet/util'
-require 'librarian/puppet/source/forge/repo'
+require 'librarian/puppet/source/forge/repo_v1'
+require 'librarian/puppet/source/forge/repo_v3'
module Librarian
module Puppet
@@ -51,6 +52,12 @@ module Librarian
def initialize(environment, uri, options = {})
self.environment = environment
+
+ if uri =~ %r{^http(s)?://forge\.puppetlabs\.com}
+ uri = "https://forgeapi.puppetlabs.com"
+ debug { "Replacing Puppet Forge API URL to use v3 #{uri}. You should update your Puppetfile" }
+ end
+
@uri = URI::parse(uri)
@cache_path = nil
end
@@ -141,7 +148,15 @@ module Librarian
def repo(name)
@repo ||= {}
- @repo[name] ||= Repo.new(self, name)
+ unless @repo[name]
+ # if we are using the official Forge then use API v3, otherwise stick to v1 for now
+ if uri.hostname =~ /\.puppetlabs\.com$/
+ @repo[name] = RepoV3.new(self, name)
+ else
+ @repo[name] = RepoV1.new(self, name)
+ end
+ end
+ @repo[name]
end
end
end
diff --git a/lib/librarian/puppet/source/forge/repo.rb b/lib/librarian/puppet/source/forge/repo.rb
index c400b78..7365e2d 100644
--- a/lib/librarian/puppet/source/forge/repo.rb
+++ b/lib/librarian/puppet/source/forge/repo.rb
@@ -10,19 +10,9 @@ module Librarian
class Repo < Librarian::Puppet::Source::Repo
include Librarian::Puppet::Util
- def initialize(source, name)
- super(source, name)
- # API returned data for this module including all versions and dependencies, indexed by module name
- # from http://forge.puppetlabs.com/api/v1/releases.json?module=#{name}
- @api_data = nil
- # API returned data for this module and a specific version, indexed by version
- # from http://forge.puppetlabs.com/api/v1/releases.json?module=#{name}&version=#{version}
- @api_version_data = {}
- end
-
def versions
return @versions if @versions
- @versions = api_data(name).map { |r| r['version'] }.reverse
+ @versions = get_versions
if @versions.empty?
info { "No versions found for module #{name}" }
else
@@ -31,8 +21,21 @@ module Librarian
@versions
end
+ # fetch list of versions ordered for newer to older
+ def get_versions
+ # implement in subclasses
+ end
+
+ # return map with dependencies in the form {module_name => version,...}
+ # version: Librarian::Manifest::Version
def dependencies(version)
- api_version_data(name, version)['dependencies']
+ # implement in subclasses
+ end
+
+ # return the url for a specific version tarball
+ # version: Librarian::Manifest::Version
+ def url(name, version)
+ # implement in subclasses
end
def manifests
@@ -83,10 +86,17 @@ module Librarian
target = vendored?(name, version) ? vendored_path(name, version).to_s : name
- # TODO can't pass the default forge url (http://forge.puppetlabs.com) to clients that use the v3 API (https://forgeapi.puppetlabs.com)
+ # TODO can't pass the default v3 forge url (http://forgeapi.puppetlabs.com) to clients that use the v1 API (https://forge.puppetlabs.com)
module_repository = source.to_s
- if Forge.client_api_version() > 1 and module_repository =~ %r{^http(s)?://forge\.puppetlabs\.com}
- module_repository = "https://forgeapi.puppetlabs.com"
+ m = module_repository.match(%r{^http(s)?://forgeapi\.puppetlabs\.com})
+ if Forge.client_api_version() == 1 and m
+ ssl = m[1]
+ # Puppet 2.7 can't handle the 302 returned by the https url, so stick to http
+ if ssl and Librarian::Puppet::puppet_gem_version < Gem::Version.create('3.0.0')
+ warn { "Using plain http as your version of Puppet #{Librarian::Puppet::puppet_gem_version} can't download from forge.puppetlabs.com using https" }
+ ssl = nil
+ end
+ module_repository = "http#{ssl}://forge.puppetlabs.com"
end
command = %W{puppet module install --version #{version} --target-dir}
@@ -122,8 +132,7 @@ module Librarian
end
def vendor_cache(name, version)
- info = api_version_data(name, version)
- url = "#{source}#{info[name].first['file']}"
+ url = url(name, version)
path = vendored_path(name, version).to_s
debug { "Downloading #{url} into #{path}"}
environment.vendor!
@@ -134,70 +143,6 @@ module Librarian
end
end
- private
-
- # Issue #223 dependencies may be duplicated
- def clear_duplicated_dependencies(data)
- return nil if data.nil?
- data.each do |m,versions|
- versions.each do |v|
- if v["dependencies"] and !v["dependencies"].empty?
- dependency_names = v["dependencies"].map {|d| d[0]}
- duplicated = dependency_names.select{ |e| dependency_names.count(e) > 1 }
- unless duplicated.empty?
- duplicated.uniq.each do |module_duplicated|
- to_remove = []
- v["dependencies"].each_index{|i| to_remove << i if module_duplicated == v["dependencies"][i][0]}
- warn { "Module #{m}@#{v["version"]} contains duplicated dependencies for #{module_duplicated}, ignoring all but the first of #{to_remove.map {|i| v["dependencies"][i]}}" }
- to_remove.slice(1..-1).reverse.each {|i| v["dependencies"].delete_at(i) }
- v["dependencies"] = v["dependencies"] - to_remove.slice(1..-1)
- end
- end
- end
- end
- end
- data
- end
-
- # get and cache the API data for a specific module with all its versions and dependencies
- def api_data(module_name)
- return @api_data[module_name] if @api_data
- # call API and cache data
- @api_data = clear_duplicated_dependencies(api_call(module_name))
- if @api_data.nil?
- raise Error, "Unable to find module '#{name}' on #{source}"
- end
- @api_data[module_name]
- end
-
- # get and cache the API data for a specific module and version
- def api_version_data(module_name, version)
- # if we already got all the versions, find in cached data
- return @api_data[module_name].detect{|x| x['version'] == version.to_s} if @api_data
- # otherwise call the api for this version if not cached already
- @api_version_data[version] = clear_duplicated_dependencies(api_call(name, version)) if @api_version_data[version].nil?
- @api_version_data[version]
- end
-
- def api_call(module_name, version=nil)
- url = source.uri.clone
- url.path += "#{'/' if url.path.empty? or url.path[-1] != '/'}api/v1/releases.json"
- url.query = "module=#{module_name}"
- url.query += "&version=#{version}" unless version.nil?
- debug { "Querying Forge API for module #{name}#{" and version #{version}" unless version.nil?}: #{url}" }
-
- begin
- data = open(url) {|f| f.read}
- JSON.parse(data)
- rescue OpenURI::HTTPError => e
- case e.io.status[0].to_i
- when 404,410
- nil
- else
- raise e, "Error requesting #{url}: #{e.to_s}"
- end
- end
- end
end
end
end
diff --git a/lib/librarian/puppet/source/forge/repo_v1.rb b/lib/librarian/puppet/source/forge/repo_v1.rb
new file mode 100644
index 0000000..26867e8
--- /dev/null
+++ b/lib/librarian/puppet/source/forge/repo_v1.rb
@@ -0,0 +1,102 @@
+require 'json'
+require 'open-uri'
+require 'librarian/puppet/source/forge/repo'
+
+module Librarian
+ module Puppet
+ module Source
+ class Forge
+ class RepoV1 < Librarian::Puppet::Source::Forge::Repo
+
+ def initialize(source, name)
+ super(source, name)
+ # API returned data for this module including all versions and dependencies, indexed by module name
+ # from http://forge.puppetlabs.com/api/v1/releases.json?module=#{name}
+ @api_data = nil
+ # API returned data for this module and a specific version, indexed by version
+ # from http://forge.puppetlabs.com/api/v1/releases.json?module=#{name}&version=#{version}
+ @api_version_data = {}
+ end
+
+ def get_versions
+ api_data(name).map { |r| r['version'] }.reverse
+ end
+
+ def dependencies(version)
+ api_version_data(name, version)['dependencies']
+ end
+
+ def url(name, version)
+ info = api_version_data(name, version)
+ "#{source}#{info[name].first['file']}"
+ end
+
+ private
+
+ # Issue #223 dependencies may be duplicated
+ def clear_duplicated_dependencies(data)
+ return nil if data.nil?
+ data.each do |m,versions|
+ versions.each do |v|
+ if v["dependencies"] and !v["dependencies"].empty?
+ dependency_names = v["dependencies"].map {|d| d[0]}
+ duplicated = dependency_names.select{ |e| dependency_names.count(e) > 1 }
+ unless duplicated.empty?
+ duplicated.uniq.each do |module_duplicated|
+ to_remove = []
+ v["dependencies"].each_index{|i| to_remove << i if module_duplicated == v["dependencies"][i][0]}
+ warn { "Module #{m}@#{v["version"]} contains duplicated dependencies for #{module_duplicated}, ignoring all but the first of #{to_remove.map {|i| v["dependencies"][i]}}" }
+ to_remove.slice(1..-1).reverse.each {|i| v["dependencies"].delete_at(i) }
+ v["dependencies"] = v["dependencies"] - to_remove.slice(1..-1)
+ end
+ end
+ end
+ end
+ end
+ data
+ end
+
+ # get and cache the API data for a specific module with all its versions and dependencies
+ def api_data(module_name)
+ return @api_data[module_name] if @api_data
+ # call API and cache data
+ @api_data = clear_duplicated_dependencies(api_call(module_name))
+ if @api_data.nil?
+ raise Error, "Unable to find module '#{name}' on #{source}"
+ end
+ @api_data[module_name]
+ end
+
+ # get and cache the API data for a specific module and version
+ def api_version_data(module_name, version)
+ # if we already got all the versions, find in cached data
+ return @api_data[module_name].detect{|x| x['version'] == version.to_s} if @api_data
+ # otherwise call the api for this version if not cached already
+ @api_version_data[version] = clear_duplicated_dependencies(api_call(name, version)) if @api_version_data[version].nil?
+ @api_version_data[version]
+ end
+
+ def api_call(module_name, version=nil)
+ url = source.uri.clone
+ url.path += "#{'/' if url.path.empty? or url.path[-1] != '/'}api/v1/releases.json"
+ url.query = "module=#{module_name}"
+ url.query += "&version=#{version}" unless version.nil?
+ debug { "Querying Forge API for module #{name}#{" and version #{version}" unless version.nil?}: #{url}" }
+
+ begin
+ data = open(url) {|f| f.read}
+ JSON.parse(data)
+ rescue OpenURI::HTTPError => e
+ case e.io.status[0].to_i
+ when 404,410
+ nil
+ else
+ raise e, "Error requesting #{url}: #{e.to_s}"
+ end
+ end
+ end
+ end
+ end
+ end
+ end
+end
diff --git a/lib/librarian/puppet/source/forge/repo_v3.rb b/lib/librarian/puppet/source/forge/repo_v3.rb
new file mode 100644
index 0000000..a1784df
--- /dev/null
+++ b/lib/librarian/puppet/source/forge/repo_v3.rb
@@ -0,0 +1,53 @@
+require 'librarian/puppet/source/forge/repo'
+require 'puppet_forge'
+
+PuppetForge.user_agent = "librarian-puppet/#{Librarian::Puppet::VERSION}"
+
+module Librarian
+ module Puppet
+ module Source
+ class Forge
+ class RepoV3 < Librarian::Puppet::Source::Forge::Repo
+
+ def get_versions
+ get_module.releases.map{|r| r.version}
+ end
+
+ def dependencies(version)
+ array = get_release(version).metadata[:dependencies].map{|d| [d['name'], d['version_requirement']]}
+ Hash[*array.flatten(1)]
+ end
+
+ def url(name, version)
+ if name == "#{get_module().owner.username}/#{get_module().name}"
+ release = get_release(version)
+ else
+ # should never get here as we use one repo object for each module (to be changed in the future)
+ debug { "Looking up url for #{name}@#{version}" }
+ release = PuppetForge::Release.find("#{name.sub('/','-')}-#{version}")
+ end
+ "#{source}#{release.file_uri}"
+ end
+
+ private
+
+ def get_module
+ @module ||= PuppetForge::Module.find(name.sub('/','-'))
+ raise(Error, "Unable to find module '#{name}' on #{source}") unless @module
+ @module
+ end
+
+ def get_release(version)
+ release = get_module.releases.find{|r| r.version == version.to_s}
+ if release.nil?
+ versions = get_module.releases.map{|r| r.version}
+ raise Error, "Unable to find version '#{version}' for module '#{name}' on #{source} amongst #{versions}"
+ end
+ release
+ end
+
+ end
+ end
+ end
+ end
+end
diff --git a/lib/librarian/puppet/version.rb b/lib/librarian/puppet/version.rb
index eb5f1d6..130f363 100644
--- a/lib/librarian/puppet/version.rb
+++ b/lib/librarian/puppet/version.rb
@@ -1,5 +1,5 @@
module Librarian
module Puppet
- VERSION = "1.0.4"
+ VERSION = "1.1.0"
end
end
diff --git a/librarian-puppet.gemspec b/librarian-puppet.gemspec
index 89ea4ba..a1b1295 100644
--- a/librarian-puppet.gemspec
+++ b/librarian-puppet.gemspec
@@ -24,6 +24,7 @@ Gem::Specification.new do |s|
s.executables = ['librarian-puppet']
s.add_dependency "librarian", ">=0.1.2"
+ s.add_dependency "puppet_forge"
# only needed for ruby 1.8
s.add_dependency "json"
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-ruby-extras/librarian-puppet.git
More information about the Pkg-ruby-extras-commits
mailing list