To: [email protected]

Subject: Patch 7.2.060

Fcc: outbox

From: Bram Moolenaar <[email protected]>

Mime-Version: 1.0

Content-Type: text/plain; charset=ISO-8859-1

Content-Transfer-Encoding: 8bit

------------

Patch 7.2.060

Problem:    When a spell files has many compound rules it may take a very long

	    time making the list of suggestions.  Displaying also can be slow

	    when there are misspelled words.

	    Can't parse some Hunspell .aff files.

Solution:   Check if a compounding can possibly work before trying a

	    combination, if the compound rules don't contain wildcards.

	    Implement using CHECKCOMPOUNDPATTERN.

	    Ignore COMPOUNDRULES.  Ignore a comment after most items.

	    Accept ONLYINCOMPOUND as an alias for NEEDCOMPOUND.

	    Accept FORBIDDENWORD as an alias for BAD.

Files:	    runtime/doc/spell.txt, src/spell.c

*** ../vim-7.2.059/runtime/doc/spell.txt	Sat Aug  9 19:36:52 2008

--- runtime/doc/spell.txt	Sun Nov 30 16:30:02 2008

***************

*** 1,4 ****

! *spell.txt*	For Vim version 7.2.  Last change: 2008 Jun 21

  		  VIM REFERENCE MANUAL	  by Bram Moolenaar

--- 1,4 ----

! *spell.txt*	For Vim version 7.2.  Last change: 2008 Nov 30

  		  VIM REFERENCE MANUAL	  by Bram Moolenaar

***************

*** 831,838 ****

  	# comment line ~

! With some items it's also possible to put a comment after it, but this isn't

! supported in general.

  ENCODING							*spell-SET*

--- 831,841 ----

  	# comment line ~

! Items with a fixed number of arguments can be followed by a comment.  But only

! if none of the arguments can contain white space.  The comment must start with

! a "#" character.  Example:

! 

! 	KEEPCASE =  # fix case for words with this flag ~

  ENCODING							*spell-SET*

***************

*** 965,970 ****

--- 968,976 ----

  Note: When using utf-8 only characters up to 65000 may be used for flags.

+ Note: even when using "num" or "long" the number of flags available to

+ compounding and prefixes is limited to about 250.

+ 

  AFFIXES

  					    *spell-PFX* *spell-SFX*

***************

*** 1178,1183 ****

--- 1185,1193 ----

  The flag also applies to the word with affixes, thus this can be used to mark

  a whole bunch of related words as bad.

+ 							*spell-FORBIDDENWORD*

+ FORBIDDENWORD can be used just like BAD.  For compatibility with Hunspell.

+ 

  							*spell-NEEDAFFIX*

  The NEEDAFFIX flag is used to require that a word is used with an affix.  The

  word itself is not a good word (unless there is an empty affix).  Example:

***************

*** 1268,1273 ****

--- 1278,1287 ----

  	NEEDCOMPOUND & ~

+ 							*spell-ONLYINCOMPOUND*

+ The ONLYINCOMPOUND does exactly the same as NEEDCOMPOUND.  Supported for

+ compatiblity with Hunspell.

+ 

  							*spell-COMPOUNDMIN*

  The minimal character length of a word used for compounding is specified with

  COMPOUNDMIN.  Example:

***************

*** 1328,1333 ****

--- 1342,1361 ----

  rules.  Can also be used for an affix to count the affix as a compounding

  word.

+ 						*spell-CHECKCOMPOUNDPATTERN*

+ CHECKCOMPOUNDPATTERN is used to define patterns that, when matching at the

+ position where two words are compounded together forbids the compound.

+ For example:

+ 	CHECKCOMPOUNDPATTERN o e ~

+ 

+ This forbids compounding if the first word ends in "o" and the second word

+ starts with "e".

+ 

+ The arguments must be plain text, no patterns are actually supported, despite

+ the item name.  Case is always ignored.

+ 

+ The Hunspell feature to use three arguments and flags is not supported.

+ 

  							*spell-SYLLABLE*

  The SYLLABLE item defines characters or character sequences that are used to

  count the number of syllables in a word.  Example:

***************

*** 1496,1501 ****

--- 1524,1533 ----

  ACCENT		(Hunspell)				*spell-ACCENT*

  		Use MAP instead. |spell-MAP|

+ BREAK		(Hunspell)				*spell-BREAK*

+ 		Define break points.  Unclear how it works exactly.

+ 		Not supported.

+ 

  CHECKCOMPOUNDCASE  (Hunspell)			*spell-CHECKCOMPOUNDCASE*

  		Disallow uppercase letters at compound word boundaries.

  		Not supported.

***************

*** 1512,1520 ****

  		Forbid three identical characters when compounding.  Not

  		supported.

- CHECKCOMPOUNDPATTERN  (Hunspell)		*spell-CHECKCOMPOUNDPATTERN*

- 		Forbid compounding when patterns match.  Not supported.

- 

  COMPLEXPREFIXES  (Hunspell)				*spell-COMPLEXPREFIXES*

  		Enables using two prefixes.  Not supported.

--- 1544,1549 ----

***************

*** 1536,1548 ****

  COMPOUNDMIDDLE	(Hunspell)				*spell-COMPOUNDMIDDLE*

  		Use COMPOUNDRULE instead. |spell-COMPOUNDRULE|

  COMPOUNDSYLLABLE  (Hunspell)			*spell-COMPOUNDSYLLABLE*

  		Use SYLLABLE and COMPOUNDSYLMAX instead. |spell-SYLLABLE|

  		|spell-COMPOUNDSYLMAX|

! FORBIDDENWORD	(Hunspell)				*spell-FORBIDDENWORD*

! 		Use BAD instead. |spell-BAD|

! 

  LANG		(Hunspell)				*spell-LANG*

  		This specifies language-specific behavior.  This actually

  		moves part of the language knowledge into the program,

--- 1565,1582 ----

  COMPOUNDMIDDLE	(Hunspell)				*spell-COMPOUNDMIDDLE*

  		Use COMPOUNDRULE instead. |spell-COMPOUNDRULE|

+ COMPOUNDRULES	(Hunspell)				*spell-COMPOUNDRULES*

+ 		Number of COMPOUNDRULE lines following.  Ignored, but the

+ 		argument must be a number.

+ 

  COMPOUNDSYLLABLE  (Hunspell)			*spell-COMPOUNDSYLLABLE*

  		Use SYLLABLE and COMPOUNDSYLMAX instead. |spell-SYLLABLE|

  		|spell-COMPOUNDSYLMAX|

! KEY		(Hunspell)				*spell-KEY*

! 		Define characters that are close together on the keyboard.

! 		Used to give better suggestions.  Not supported.

! 		

  LANG		(Hunspell)				*spell-LANG*

  		This specifies language-specific behavior.  This actually

  		moves part of the language knowledge into the program,

***************

*** 1553,1562 ****

  		Only needed for morphological analysis.

  MAXNGRAMSUGS	(Hunspell)				*spell-MAXNGRAMSUGS*

! 		Not supported.

! 

! ONLYINCOMPOUND	(Hunspell)				*spell-ONLYINCOMPOUND*

! 		Use NEEDCOMPOUND instead. |spell-NEEDCOMPOUND|

  PSEUDOROOT	(Hunspell)				*spell-PSEUDOROOT*

  		Use NEEDAFFIX instead. |spell-NEEDAFFIX|

--- 1587,1593 ----

  		Only needed for morphological analysis.

  MAXNGRAMSUGS	(Hunspell)				*spell-MAXNGRAMSUGS*

! 		Set number of n-gram suggestions.  Not supported.

  PSEUDOROOT	(Hunspell)				*spell-PSEUDOROOT*

  		Use NEEDAFFIX instead. |spell-NEEDAFFIX|

*** ../vim-7.2.059/src/spell.c	Sat Nov 29 20:18:44 2008

--- src/spell.c	Sun Nov 30 20:59:13 2008

***************

*** 469,474 ****

--- 469,475 ----

      garray_T	sl_comppat;	/* CHECKCOMPOUNDPATTERN items */

      regprog_T	*sl_compprog;	/* COMPOUNDRULE turned into a regexp progrm

  				 * (NULL when no compounding) */

+     char_u	*sl_comprules;	/* all COMPOUNDRULE concatenated (or NULL) */

      char_u	*sl_compstartflags; /* flags for first compound word */

      char_u	*sl_compallflags; /* all flags for compound words */

      char_u	sl_nobreak;	/* When TRUE: no spaces between words */

***************

*** 839,845 ****

--- 840,849 ----

  static void slang_clear __ARGS((slang_T *lp));

  static void slang_clear_sug __ARGS((slang_T *lp));

  static void find_word __ARGS((matchinf_T *mip, int mode));

+ static int match_checkcompoundpattern __ARGS((char_u *ptr, int wlen, garray_T *gap));

  static int can_compound __ARGS((slang_T *slang, char_u *word, char_u *flags));

+ static int can_be_compound __ARGS((trystate_T *sp, slang_T *slang, char_u *compflags, int flag));

+ static int match_compoundrule __ARGS((slang_T *slang, char_u *compflags));

  static int valid_word_prefix __ARGS((int totprefcnt, int arridx, int flags, char_u *word, slang_T *slang, int cond_req));

  static void find_prefix __ARGS((matchinf_T *mip, int mode));

  static int fold_more __ARGS((matchinf_T *mip));

***************

*** 1519,1524 ****

--- 1523,1533 ----

  					    ((unsigned)flags >> 24)))

  		    continue;

+ 		/* If there is a match with a CHECKCOMPOUNDPATTERN rule

+ 		 * discard the compound word. */

+ 		if (match_checkcompoundpattern(ptr, wlen, &slang->sl_comppat))

+ 		    continue;

+ 

  		if (mode == FIND_COMPOUND)

  		{

  		    int	    capflags;

***************

*** 1577,1582 ****

--- 1586,1596 ----

  		    if (!can_compound(slang, fword, mip->mi_compflags))

  			continue;

  		}

+ 		else if (slang->sl_comprules != NULL

+ 			     && !match_compoundrule(slang, mip->mi_compflags))

+ 		    /* The compound flags collected so far do not match any

+ 		     * COMPOUNDRULE, discard the compounded word. */

+ 		    continue;

  	    }

  	    /* Check NEEDCOMPOUND: can't use word without compounding. */

***************

*** 1727,1732 ****

--- 1741,1779 ----

  }

  /*

+  * Return TRUE if there is a match between the word ptr[wlen] and

+  * CHECKCOMPOUNDPATTERN rules, assuming that we will concatenate with another

+  * word.

+  * A match means that the first part of CHECKCOMPOUNDPATTERN matches at the

+  * end of ptr[wlen] and the second part matches after it.

+  */

+     static int

+ match_checkcompoundpattern(ptr, wlen, gap)

+     char_u	*ptr;

+     int		wlen;

+     garray_T	*gap;  /* &sl_comppat */

+ {

+     int		i;

+     char_u	*p;

+     int		len;

+ 

+     for (i = 0; i + 1 < gap->ga_len; i += 2)

+     {

+ 	p = ((char_u **)gap->ga_data)[i + 1];

+ 	if (STRNCMP(ptr + wlen, p, STRLEN(p)) == 0)

+ 	{

+ 	    /* Second part matches at start of following compound word, now

+ 	     * check if first part matches at end of previous word. */

+ 	    p = ((char_u **)gap->ga_data)[i];

+ 	    len = STRLEN(p);

+ 	    if (len <= wlen && STRNCMP(ptr + wlen - len, p, len) == 0)

+ 		return TRUE;

+ 	}

+     }

+     return FALSE;

+ }

+ 

+ /*

   * Return TRUE if "flags" is a valid sequence of compound flags and "word"

   * does not have too many syllables.

   */

***************

*** 1773,1778 ****

--- 1820,1917 ----

  }

  /*

+  * Return TRUE when the sequence of flags in "compflags" plus "flag" can

+  * possibly form a valid compounded word.  This also checks the COMPOUNDRULE

+  * lines if they don't contain wildcards.

+  */

+     static int

+ can_be_compound(sp, slang, compflags, flag)

+     trystate_T	*sp;

+     slang_T	*slang;

+     char_u	*compflags;

+     int		flag;

+ {

+     /* If the flag doesn't appear in sl_compstartflags or sl_compallflags

+      * then it can't possibly compound. */

+     if (!byte_in_str(sp->ts_complen == sp->ts_compsplit

+ 		? slang->sl_compstartflags : slang->sl_compallflags, flag))

+ 	return FALSE;

+ 

+     /* If there are no wildcards, we can check if the flags collected so far

+      * possibly can form a match with COMPOUNDRULE patterns.  This only

+      * makes sense when we have two or more words. */

+     if (slang->sl_comprules != NULL && sp->ts_complen > sp->ts_compsplit)

+     {

+ 	int v;

+ 

+ 	compflags[sp->ts_complen] = flag;

+ 	compflags[sp->ts_complen + 1] = NUL;

+ 	v = match_compoundrule(slang, compflags + sp->ts_compsplit);

+ 	compflags[sp->ts_complen] = NUL;

+ 	return v;

+     }

+ 

+     return TRUE;

+ }

+ 

+ 

+ /*

+  * Return TRUE if the compound flags in compflags[] match the start of any

+  * compound rule.  This is used to stop trying a compound if the flags

+  * collected so far can't possibly match any compound rule.

+  * Caller must check that slang->sl_comprules is not NULL.

+  */

+     static int

+ match_compoundrule(slang, compflags)

+     slang_T	*slang;

+     char_u	*compflags;

+ {

+     char_u	*p;

+     int		i;

+     int		c;

+ 

+     /* loop over all the COMPOUNDRULE entries */

+     for (p = slang->sl_comprules; *p != NUL; ++p)

+     {

+ 	/* loop over the flags in the compound word we have made, match

+ 	 * them against the current rule entry */

+ 	for (i = 0; ; ++i)

+ 	{

+ 	    c = compflags[i];

+ 	    if (c == NUL)

+ 		/* found a rule that matches for the flags we have so far */

+ 		return TRUE;

+ 	    if (*p == '/' || *p == NUL)

+ 		break;  /* end of rule, it's too short */

+ 	    if (*p == '[')

+ 	    {

+ 		int match = FALSE;

+ 

+ 		/* compare against all the flags in [] */

+ 		++p;

+ 		while (*p != ']' && *p != NUL)

+ 		    if (*p++ == c)

+ 			match = TRUE;

+ 		if (!match)

+ 		    break;  /* none matches */

+ 	    }

+ 	    else if (*p != c)

+ 		break;  /* flag of word doesn't match flag in pattern */

+ 	    ++p;

+ 	}

+ 

+ 	/* Skip to the next "/", where the next pattern starts. */

+ 	p = vim_strchr(p, '/');

+ 	if (p == NULL)

+ 	    break;

+     }

+ 

+     /* Checked all the rules and none of them match the flags, so there

+      * can't possibly be a compound starting with these flags. */

+     return FALSE;

+ }

+ 

+ /*

   * Return non-zero if the prefix indicated by "arridx" matches with the prefix

   * ID in "flags" for the word "word".

   * The WF_RAREPFX flag is included in the return value for a rare prefix.

***************

*** 2513,2521 ****

--- 2652,2662 ----

      lp->sl_midword = NULL;

      vim_free(lp->sl_compprog);

+     vim_free(lp->sl_comprules);

      vim_free(lp->sl_compstartflags);

      vim_free(lp->sl_compallflags);

      lp->sl_compprog = NULL;

+     lp->sl_comprules = NULL;

      lp->sl_compstartflags = NULL;

      lp->sl_compallflags = NULL;

***************

*** 3460,3465 ****

--- 3601,3607 ----

      char_u	*pp;

      char_u	*cp;

      char_u	*ap;

+     char_u	*crp;

      int		cnt;

      garray_T	*gap;

***************

*** 3545,3550 ****

--- 3687,3698 ----

      slang->sl_compallflags = ap;

      *ap = NUL;

+     /* And a list of all patterns in their original form, for checking whether

+      * compounding may work in match_compoundrule().  This is freed when we

+      * encounter a wildcard, the check doesn't work then. */

+     crp = alloc(todo + 1);

+     slang->sl_comprules = crp;

+ 

      pp = pat;

      *pp++ = '^';

      *pp++ = '\\';

***************

*** 3587,3592 ****

--- 3735,3754 ----

  		    atstart = 0;

  	    }

  	}

+ 

+ 	/* Copy flag to "sl_comprules", unless we run into a wildcard. */

+ 	if (crp != NULL)

+ 	{

+ 	    if (c == '+' || c == '*')

+ 	    {

+ 		vim_free(slang->sl_comprules);

+ 		slang->sl_comprules = NULL;

+ 		crp = NULL;

+ 	    }

+ 	    else

+ 		*crp++ = c;

+ 	}

+ 

  	if (c == '/')	    /* slash separates two items */

  	{

  	    *pp++ = '\\';

***************

*** 3611,3616 ****

--- 3773,3781 ----

      *pp++ = '$';

      *pp = NUL;

+     if (crp != NULL)

+ 	*crp = NUL;

+ 

      slang->sl_compprog = vim_regcomp(pat, RE_MAGIC + RE_STRING + RE_STRICT);

      vim_free(pat);

      if (slang->sl_compprog == NULL)

***************

*** 4915,4920 ****

--- 5080,5086 ----

  } spellinfo_T;

  static afffile_T *spell_read_aff __ARGS((spellinfo_T *spin, char_u *fname));

+ static int is_aff_rule __ARGS((char_u **items, int itemcnt, char *rulename, int	 mincount));

  static void aff_process_flags __ARGS((afffile_T *affile, affentry_T *entry));

  static int spell_info_item __ARGS((char_u *s));

  static unsigned affitem2flag __ARGS((int flagtype, char_u *item, char_u	*fname, int lnum));

***************

*** 5223,5230 ****

  	/* Handle non-empty lines. */

  	if (itemcnt > 0)

  	{

! 	    if (STRCMP(items[0], "SET") == 0 && itemcnt == 2

! 						       && aff->af_enc == NULL)

  	    {

  #ifdef FEAT_MBYTE

  		/* Setup for conversion from "ENC" to 'encoding'. */

--- 5389,5395 ----

  	/* Handle non-empty lines. */

  	if (itemcnt > 0)

  	{

! 	    if (is_aff_rule(items, itemcnt, "SET", 2) && aff->af_enc == NULL)

  	    {

  #ifdef FEAT_MBYTE

  		/* Setup for conversion from "ENC" to 'encoding'. */

***************

*** 5239,5245 ****

  		    smsg((char_u *)_("Conversion in %s not supported"), fname);

  #endif

  	    }

! 	    else if (STRCMP(items[0], "FLAG") == 0 && itemcnt == 2

  					      && aff->af_flagtype == AFT_CHAR)

  	    {

  		if (STRCMP(items[1], "long") == 0)

--- 5404,5410 ----

  		    smsg((char_u *)_("Conversion in %s not supported"), fname);

  #endif

  	    }

! 	    else if (is_aff_rule(items, itemcnt, "FLAG", 2)

  					      && aff->af_flagtype == AFT_CHAR)

  	    {

  		if (STRCMP(items[1], "long") == 0)

***************

*** 5284,5352 ****

  			spin->si_info = p;

  		    }

  	    }

! 	    else if (STRCMP(items[0], "MIDWORD") == 0 && itemcnt == 2

  							   && midword == NULL)

  	    {

  		midword = getroom_save(spin, items[1]);

  	    }

! 	    else if (STRCMP(items[0], "TRY") == 0 && itemcnt == 2)

  	    {

  		/* ignored, we look in the tree for what chars may appear */

  	    }

  	    /* TODO: remove "RAR" later */

! 	    else if ((STRCMP(items[0], "RAR") == 0

! 			|| STRCMP(items[0], "RARE") == 0) && itemcnt == 2