[Pkg-jed-devel] byte compiling sl files

G. Milde g.milde@web.de
Tue, 26 Apr 2005 10:43:45 +0200


--OXfL5xGRrasGEqWY
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On 25.04.05, Jörg Sommer wrote:
> G. Milde schrieb am Mon 25. Apr, 12:57 (+0200) :
> > > 1) Each package needing byte-compiling
> > 
> >      * can do this in the postinst script with:
> >      
> >          cd <dir where the file is installed, e.g. Jed_Site_Library>
> >          jed --batch -f 'byte_compile_file("<myfile.sl>", 0)'
> 
> and didn't get updated if a new jed package comes.

It gets updated by 3), if it is in Jed_Site_Library
 
> >        (this is the fast way) or 
> >          
> >          jed --batch -l make_ini.sl
> 
> Which doesn't work, if jed-extra isn't installed. 

I forgot to say explicitely, that in order to make my scheme work,
make_ini.sl and home-lib.sl should be made part of jed-common.

This would mean an addition to the standard jed package, however I
consider it to be non-invaive as it doesnot change the behaviour of jed.
At least, it in not more invasive than the add-on of the jed-init.d
directory already in place.

With home-lib, jed-common would also provide (and initialize)
Jed_Site_Library for other packages (jed-extra or any other). So packages
that want to provide a mode for jed do not depend on jed-extra in order
to have a a space to put their files in. (/etc/jed-init.d should be
reserved for as few as possible and as small as possible initialization
files, to ensure a fast startup of jed.)
 
> > > 3) The jed-common package will run 
> > 
> >      cd <Jed_Site_Library>
> >      jed --batch -l make_ini.sl
> 
> Which fails.

This is bad.  Of course, in order to be in jed-common, make_ini.sl needs
to be stable and tested.

Why does it fail? Could you try the attached version?


Günter

-- 
G.Milde web.de

--OXfL5xGRrasGEqWY
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: attachment; filename="make_ini.sl"
Content-Transfer-Encoding: 8bit

% A tool to set up extensions (modes and tools)for jed.
%
% Creates a file ini.sl that declares all (public) functions
% in the current directory. Also bytecompiles the files, if set to do so.
%
% Copyright (c) 2003 Günter Milde
% Released under the terms of the GNU General Public License (ver. 2 or later)
%
% USAGE:
%    M-x make_ini   to get a buffer with the autoload commands (and helpfull
%                   comments) for viewing/editing and subsequent saving
%    M-x update_ini to update the ini.sl file in the current buffers working
%                   dir without user interaction
%
%    jed -batch -l make_ini.sl   to update the ini.sl file in the current
%                                directory in a batch process
%                                (using update_ini)
%
% TODO:  * Consider preprocessor options (How?)
%
% Versions:  0.9 initial release     Guenter Milde <g.milde web.de>
%            1.0 non-interactive functions and support for batch use
%            1.1 08/07/03 made compatible to txtutils 2.2 (change in get_word)
%            1.2 only write _autoload statements, do not add_completion
%            1.3 * make_ini_look_for_functions made public:
%                  add autoloads for specified file (request P. Boekholt)
%                * parse the files in a special buffer (avoiding mode-sets
%                  and problems with currently open files)
% 2004-03-08 2.0 * added support for documentation extraction with tm.sl
%                * make_ini no longer bytecompiles (use update_ini for that)
% 2004-06-10 2.1 * fixed a bug that prevented bytecompiling
%                * added tm documentation for public functions and variables
%                * provisions for a section with custom code in ini.sl
% 2004-11-26 2.2 * code cleanup (hopefully without introducing bugs)
% 	         * look for a <INITIALIZATION> </INITIALIZATION> block
% 	           Use this for custom initialization code (menu addings, etc)
% 	           If such a block is found, no search for functions is done!
% 2005-03-18 2.3 * bugfixes: list_slang_files() did only work, if 
%                            dir == current working dir (report Dino Sangoi)
%                            "%" in vinsert needs to be doubled (Dino Sangoi)
%                            documentation comments did not work
%                  removed the need for a chdir() alltogether
% 2005-03-31 2.4 * windows bugfix: quote backslashes in add_autoload_fun() (Dino)
% 	         * bugfix in list_slang_files(): skip jed lock files
% 2005-04-01 2.5 * cleanup in make_ini_look_for_functions(): do not insert
% 	     	   library path in autoload commands
% 2005-04-07 2.6 * provide add_completion commands (with Make_ini_Add_Completions)
% 2005-04-25 2.7 * bugfix: list_slang_files() failed for directories without 
% 	     	   	   slang files or nonexisting directories.
% 	     	   	   make_libfun_doc() failed if there was no documentation
%                * Test for existing tm.sl with #ifexists:
%                  the function tm_parse() must be defined/declared at 
%                  evaluation/preparse time of make_ini.sl in order to enable 
%                  the documentation extract feature.
%                  

% _debug_info=1;

autoload("get_word", "txtutils");
autoload("push_array", "sl_utils");

% --- Settings -----------------------------------------------------------

%!%+
%\variable{Make_ini_Scope}
%\synopsis{Scope for automatic generation of autoload commands}
%\usage{Int_Type Make_ini_Scope = 1}
%\description
%  Set the scope for automatic generation of autoload commands
%   0 no declarations
%   1 only explicitly public definitions
%   2 public definitions and (if no namespace is declared) simple definitions
%\seealso{make_ini, make_ini_look_for_functions}
%!%-
custom_variable("Make_ini_Scope", 1);

%!%+
%\variable{Make_ini_Verbose}
%\synopsis{Comment the ini-file}
%\usage{Int_Type Make_ini_Verbose = 2}
%\description
% Comment the ini.sl file generated by make_ini().
%  n == 0: no comments
%  n >  0: n lines of global comments + list of custom variables
%\seealso{make_ini}
%!%-
custom_variable("Make_ini_Verbose", 2);

%!%+
%\variable{Make_ini_Bytecompile}
%\synopsis{Bytecompile the files with update_ini()}
%\usage{Int_Type Make_ini_Bytecompile = 1}
%\description
% Let update_ini, update_home_lib, and update_site_lib bytecompile 
% the files as well.
% This gives considerable evalutation speedup but might introduce problems.
%\notes 
%  Attention: byte-compiling can give problems
%     * with constructs like #ifdef XWINDOWS
%     	when xjed and jed are used in parallel
%     * with constructs like
%       #ifndefined my_new_function()
%       define my_new_function()
%       {  ... }
%       #endif
%\seealso{update_ini, update_home_lib, update_site_lib}
%!%-
custom_variable("Make_ini_Bytecompile", 1);

%!%+
%\variable{Make_ini_Exclusion_List}
%\synopsis{Array of files to exclude from make_ini()}
%\usage{String_Type[] Make_ini_Exclusion_List = ["ini.sl"]}
%\description
% Exclusion list: do not add autoloads for these files.
% Also exlude these files from bytecompiling with update_ini().
%\seealso{make_ini, update_ini, Make_ini_Bytecompile}
%!%-
custom_variable("Make_ini_Exclusion_List", ["ini.sl"]);

%!%+
%\variable{Make_ini_Extract_Documentation}
%\synopsis{Extract documentation with update_ini}
%\usage{Int_Type Make_ini_Extract_Documentation = 1}
%\description
% Let update_ini() also extract documentation and save in libfuns.txt file.
%\notes 
% Documentation extraction requires tm_parse() from the tm.sl mode.
% If tm_parse() is not defined at evaluation (or byt-compile) time of
% make_ini.sl, Make_ini_Extract_Documentation will be set to 0.
%\seealso{update_ini, tm_parse, tm_view}
%!%-
custom_variable("Make_ini_Extract_Documentation", 1);

#ifnexists tm_parse
  Make_ini_Extract_Documentation = 0;
#endif

%!%+
%\variable{Make_ini_Add_Completions}
%\synopsis{Insert add_completion commands into ini.sl}
%\usage{Int_Type Make_ini_Add_Completions = 0}
%\description
%  Should make_ini() insert add_completion commands into ini.sl?
%\seealso{Make_ini_Bytecompile, Make_ini_Extract_Documentation, Make_ini_Verbose}
%!%-
custom_variable("Make_ini_Add_Completions", 0);

% valid chars in function and variable definitions
static variable Slang_word = "A-Za-z0-9_";
static variable Ini_File = "ini.sl";
static variable Parsing_Buffer = "*make_ini tmp*";
static variable Tm_Doc_File = "libfuns.txt";

% --- functions ---------------------------------------------------

static define _get_function_name()
{
   variable str, fun;
   !if (ffind("define"))
     return "";
   skip_word();
   return get_word(Slang_word, 1);  % (Slang_word, skip=True)
}

%!%+
%\function{make_ini_look_for_functions}
%\synopsis{Insert initialisation code for a slang file}
%\usage{Str = make_ini_look_for_functions(file)}
%\description
%  Browse file and insert 
%    either an explicit <INITIALIZATION> </INITIALIZATION> block 
%    or autoload commands for function definitions
%  in the current buffer. The variable \var{Make_ini_Scope} can be used to
%  control whether only explicitly public definitions are tracked
%\notes
%  The slang documentation file preprocess.txt list the preprocessor 
%  directives
%#v+
%    #<TAG>  % - start embedded text region
%      ...  embedded text region ...
%    #</TAG> % - end embedded text region
%#v-
%  but they did not work with my S-Lang Version: 1.4.9
%  
%  My workaround is a construct of type
%#v+
%  #iffalse %<INITIALIZATION>
%     ... initialisation block ...
%  #endif %</INITIALIZATION>
%#v-
%
%  If there is an INITIALIZATION block, no automatic search for 
%  function definitions is done. You need to explicitely list the required 
%  autoload commands! (This way, people not using make_ini can copy the 
%  INITIALIZATION block directly to their .jedrc file.
%  
%\seealso{make_ini, update_ini, Make_ini_Scope,}
%!%-
define make_ini_look_for_functions(file)
{
   variable str = "",
     funs = "", no_of_funs, funs_n_files,
     named_namespace = 0,
     currbuf = whatbuf();

   % show("processing", whatbuf(), file);

   % Parse the file in a special buffer, without setting the mode
   % (saves time and does not interfere with open files)
   sw2buf(Parsing_Buffer);
   erase_buffer(); % paranoia
   () = insert_file(file);
   set_buffer_modified_flag(0);
   
   % if `file' is in the jed-library-path, remove the library-path from it
   variable dir, libdirs = strchop(get_jed_library_path, ',' , 0);
   libdirs = libdirs[where(libdirs != ".")]; % filter the current dir
   foreach (libdirs)
     {
	dir = ();
	if (is_substr(file, dir) == 1)
	  file = file[[strlen(dir)+1:]];
     }

   % global comment
   bob();
   loop(Make_ini_Verbose)  % max as many lines as Make_ini_Verbose indicates
     {
	!if (looking_at("%"))
	  break;
	push_mark();
	go_down_1();
	str += bufsubstr();
     }
   % list custom variables (assuming the definition starts at bol)
   if (Make_ini_Verbose)
     while (bol_fsearch("custom_variable"))
       {
	  push_mark_eol();
	  str += "% " + bufsubstr() + "\n";
       }
   % find out if the mode defines/uses a namespace
   bob();
   if (andelse{bol_fsearch("implements")} {bol_fsearch("use_namespace")})
     {
	named_namespace = 1;
        str += "% private namespace: " + line_as_string + "\n";
     }

   % Look for an <INITIALIZATION> </INITIALIZATION> block
   bob();
   if (fsearch("<INITIALIZATION>"), bol() and looking_at_char('#'))
     {
	go_down_1(); bol();
	push_mark();
	if (fsearch("</INITIALIZATION>"))
	  {
	     bol();
	     str += bufsubstr();
	  }
	else
	  {
	     str += "no </INITIALIZATION> end tag found";
	     pop_mark(0);
	  }
     }
   else 
     {
	% Search function definitions
	% 1. explicitly public definitions
	if (Make_ini_Scope)
	  while (bol_fsearch("public define "))
	    funs += _get_function_name() + "\n";
	bob;
	% 2. "normal" (i.e. unspecified) definitions
	if (Make_ini_Scope - named_namespace > 1)
	  while (bol_fsearch("define "))
	    funs += _get_function_name() + "\n";
	% convert to Array 
	% (strchop would append an empty element because of the trailing \n)
	funs = strtok(funs, "\n");
	no_of_funs = length(funs); 
	if (no_of_funs)
	  {
	     funs = "\"" + funs + "\"";
	     funs_n_files = funs + sprintf(", \"%s\";", 
					   str_quote_string(file, "", '\\'));
	     % autoloads
	     str += strjoin(funs_n_files, "\n") 
	       + sprintf("\n_autoload(%d);\n", no_of_funs);
	     % add_completions
	     if (Make_ini_Add_Completions)
	     str += "\n" + strjoin(funs, ";\n") 
		 + sprintf(";\n_add_completion(%d);\n", no_of_funs);
	  }
     }
   % cleanup
   delbuf(Parsing_Buffer);
   sw2buf(currbuf);
   insert(str);
}

static define list_slang_files(dir)
{
   variable exclusion_file, files = listdir(dir);
   if (files == NULL or length(files) == 0)
     return String_Type[0];
   % Skip files that are  no slang-source (test for extension ".sl")
   files = files[where(array_map(String_Type, &path_extname, files) == ".sl")];
   !if (length(files))
     return String_Type[0];
   % Skip jed lock files
   files = files[where(array_map(Integer_Type, &is_substr, files, ".#") != 1)];
   % Skip files from the exclusion list
   foreach (Make_ini_Exclusion_List)
     {
	exclusion_file = ();
	files = files[where(files != exclusion_file)];
     }
   !if (length(files))
     return String_Type[0];
   % Prepend the directory to the path
   files = array_map(String_Type, &path_concat, dir, files);
   % Sort alphabetically and return
   return files[array_sort(files)];
}

%!%+
%\function{make_ini}
%\synopsis{}
%\usage{ make_ini([dir])}
%\description
%   Scan all slang files in \var{dir} for function definitions and
%   place autoload commands in a buffer ini.sl.
%   After customizing, it can be saved and serve as an initialization
%   for a slang-library. The home-lib mode at jedmodes.sf.net will
%   automatically evaluate this ini.sl files at startup, making the 
%   installation of additional modes easy.
%\seealso{update_ini, Make_ini_Scope, Make_ini_Exclusion_List, Make_ini_Verbose}
%!%-
public define make_ini() % ([dir])
{
   % get optional argument
   variable dir;
   if (_NARGS)
     dir = ();
   else 
     dir = read_file_from_mini("Make ini.sl for:");

   variable files = list_slang_files(dir), file = "";
   % find old ini file 
   () = find_file(path_concat(dir, Ini_File));
   slang_mode();
   bob();
   % skip customized part, write header
   !if (bol_fsearch("% [code below will be replaced"))
     vinsert("%% ini.sl: initialization file for the library dir %s\n"
	+ "%% automatically generated by make_ini()\n\n"
	+ "%% --- Place customized code above this line ---\n"
	+ "%% [code below will be replaced by the next run of make_ini()]\n"
	, dir);
   % blank rest of buffer
   go_down_1; bol();
   push_mark_eob();
   del_region();
   
   foreach (files)
     {
	file = ();
	% show(file, file_type(file), file_status(file));
	insert("\n% " + path_basename(file) + "\n");
	make_ini_look_for_functions(file);
     }
}

%!%+
%\function{make_libfun_doc}
%\synopsis{Write tm documentation in dir to "libfuns.txt"}
%\usage{ make_libfun_doc([dir])}
%\description
%  Extract tm documentation for all Slang files in \var{dir}, 
%  convert to ascii format and write to file "libfuns.txt".
%\notes
%  requires tm.sl (jedmodes.sf.net/mode/tm/) 
%\seealso{update_ini, Make_ini_Extract_Documentation}
%!%-
public define make_libfun_doc() % ([dir])
{
#ifnexists tm_parse
     error("make_libfun_doc needs a current version of tm.sl");
#else
   % get optional argument
   variable dir;
   if (_NARGS)
     dir = ();
   else
     {
	(, dir, , ) = getbuf_info (); % default
	dir = read_file_from_mini("Extract tm Documentation from dir:");
     }
   
   % extract tm documentation blocks
   variable docstrings, str, files=list_slang_files(dir);
   !if (length(files))
     return vmessage("no slang files in %s", dir);
   docstrings = array_map(String_Type, &tm_parse, files);
   str = strjoin(docstrings, "");
   if (str == "")
     return vmessage("no tm documentation in %s", dir);
   
   () = write_string_to_file (str, path_concat(dir, Tm_Doc_File));
#endif
}

%!%+
%\function{update_ini}
%\synopsis{Update the ini.sl initialization file}
%\usage{update_ini(directory=buffer_dir())}
%\description
%  Update the ini.sl initialization file with autoload commands
%  functions in all slang files in \var(dir).
%  Depending on the Make_ini_* custom variables this also bytecompiles
%  the files and extracts tm documentation.
%\seealso{make_ini, update_home_lib, update_site_lib, Make_ini_Bytecompile, Make_ini_Extract_Documentation}
%!%-
public define update_ini() % (directory=buffer_dir())
{

   variable file, dir, name, flags, buf = whatbuf();
   if (_NARGS)
     dir = ();
   else
	(, dir, , ) = getbuf_info (); % default

   make_ini(dir);
   save_buffer();
   delbuf(whatbuf);
   % bytecompile (the ini-file as well)
   if(Make_ini_Bytecompile)
     {
	flush("byte compiling files");
	foreach (list_slang_files(dir))
	  {
	     file = ();
	     byte_compile_file(file, 0);
	  }
	byte_compile_file(path_concat(dir, Ini_File), 0);
     }
   % extract the documentation and put in file libfuns.txt
   if(Make_ini_Extract_Documentation)
	make_libfun_doc(dir);
   sw2buf(buf);
   message("update_ini completed");
}

% run update_ini, if called as a batch process
if (BATCH)
  update_ini();

#ifexists Jed_Home_Library
%!%+
%\function{update_home_lib}
%\synopsis{update Jed_Home_Library/ini.sl}
%\usage{ update_home_lib()}
%\description
%   Run update_ini for the Jed_Home_Library
%\seealso{update_ini, make_ini, Jed_Home_Library}
%!%-
public define update_home_lib()
{
   update_ini(Jed_Home_Library);
}
#endif

#ifexists Jed_Site_Library
%!%+
%\function{update_site_lib}
%\synopsis{update Jed_Site_Library/ini.sl}
%\usage{ update_site_lib()}
%\description
%   Run update_ini for the Jed_Site_Library
%\seealso{update_ini, make_ini, Jed_Site_Library}
%!%-
public define update_site_lib()
{
   update_ini(Jed_Site_Library);
}
#endif


--OXfL5xGRrasGEqWY--