3.5 Implementing Text Indentation

At one point during the development of FunnelWeb, text indentation was fully implemented. However, it was subsequently removed because it was considered a dangerous feature. This section records the way in which text indentation was implemented so that if the feature ever has to be put back, this technique can be used again.

1. Create a new field in the sc_t record call sc_postn.

      char *sc_postn; /* Pointer in the range [sc_first,sc_last+1].           */
                      /* It is the minimum possible value of sc_postn for     */
                      /* which EOL does not appear in *sc_postn..*sc_last.    */
                      /* i.e. Points to the byte following the first EOL in   */
                      /* the scrap or sc_first if EOL does not appear.        */

2. Modify the scanner so that it generates this field. Sendtext should be modified so that it accepts an argument for the p_postn value.

LOCAL void sendtext P_((p_ps_t,char *,char *,char *,bool));
LOCAL void sendtext(p_tkps,p_first,p_last,p_postn,is_white)
/* Appends a text token to the end of the token list.                         */
/* IN: p_ps is a pointer to a position structure giving the position of the   */
/*     first character of the token.                                          */
/* IN: p_first and p_last point to the first and last byte of the text scrap. */
/* IN: p_postn has the same definition as sc_postn (see fwdata.h).            */
/* IN: is_white should be set to TRUE iff scrap is entirely whitespace.       */
p_ps_t p_tkps;
char  *p_first;
char  *p_last;
char  *p_postn;
bool   is_white;
{
 tk_t token;

 /* Empty text scraps should never be generated. */
 assert(p_first<=p_last,"sendtext: Text scrap bounds are bad.");

 /* If ch=EOL then we should be scanning more text, not shipping it! */
 assert(ch!=EOL,"senttext: Shipping text while still more to scan.");

 /* Check that p_postn is in range. See definition in fwdata.h. */
 assert(p_first<=p_postn && p_postn<=p_last+1,
        "sendtext: p_postn is out of range.");

 /* Debug: Check the p_postn field using a brute force check. */
 {
  char *i,*j;
  j=p_first;
  for (i=p_first;i<=p_last;i++)
     if (*i==EOL)
        j=i+1;
  assert(j==p_postn,"sendtext: sc_postn field is incorrect.");
 }

 /* Load the text token. */
 token.tk_kind        = TK_TEXT;
 ASSIGN(token.tk_ps,*p_tkps);
 token.tk_sc.sc_first = p_first;
 token.tk_sc.sc_last  = p_last;
 token.tk_sc.sc_postn = p_postn;
 token.tk_white       = is_white;
 token.tk_parno       = 0;
 ls_add(token_list,PV &token);
}

Then all the calls to sendtext have to be changed:

/* @ instructs FunnelWeb to replace the special construct with the */
/* special character. Luckily one appears just before the @ !!     */
/* Note: FALSE is OK because space is not a legal specialch.       */
/* Note: Setting parameter p_postn to p_ch-1 is OK as EOL is not a */
/*       legal specialch.                                          */
sendtext(ps_spec,p_ch-1,p_ch-1,p_ch-1,FALSE);
break;

/* + instructs FunnelWeb to insert an EOL. We can't look to the end of */
/* the previous line to find an EOL as this might be the first line.   */
/* Running ahead to the end of the line is expensive, and having the   */
/* liner mini-package maintain a variable for it would be extra        */
/* housekeeping. Instead of all this, we just point to a static.       */
{CONST static char stateol = EOL;
 sendtext(&ps_spec,&stateol,&stateol,(&stateol)+1,TRUE);}
break;

/* If we hit something that ends a text token */
/* then we can transmit a white text token.   */
if (ch==specialch || ch==EOFCH)
   {sendtext(&ps_start,p_first,p_ch-1,MAX(p_sol,p_first),TRUE); return;}

/* Otherwise we have some more (non-white) text to scan. */
/* We can then send a non-white text token.              */
while (ch!=specialch && ch!=EOFCH)
   NEXTCH;
sendtext(&ps_start,p_first,p_ch-1,MAX(p_sol,p_first),FALSE);

        wf_str(p_wf,"\"");
assert(token->tk_sc.sc_first !=NULL,"dm_tkls: NULL ptr1.");
assert(token->tk_sc.sc_last  !=NULL,"dm_tkls: NULL ptr2.");
for (i=token->tk_sc.sc_first; i<=token->tk_sc.sc_last; i++)
  {
   if (i==token->tk_sc.sc_postn)
      wf_str(p_wf,"<postn>");
   if (*i=='\n')
      wf_wl(p_wf,"");
   else
      dm_byte(p_wf,*((ubyte_ *) i));
  }
if (i==token->tk_sc.sc_postn)
   wf_str(p_wf,"<postn>");
wf_str(p_wf,"\"");
}

3. Over in the Tangle module, create a massive array of pointers to scraps to be used as a stack. Maintain pointers into the stack called current and base  (similar to the blank indentation variables). Implement the following:

• To write out a scrap, scan it byte by byte. Output each byte. When you hit an EOL, pop the stack back to base. Then write out an EOL followed by the stack contents but writing each scrap only from postn to end end of each scrap. When you have finished the new scrap, push it on the stack.
• When you hit a new macro to expand, save base. Restore it later.

The postn field solves the big problem of how to cope with something like this:

The rain in Spain
falls mainly @<on the plain@>

The trouble is that we want to text indent the lines in @<on the plain@> with just "falls mainly ". However, this string is only part of a scrap. The solution is to get the scanner to record, in the postn field of each scrap, the position of the first byte with a EOL-free run to the end of the scrap.

This scheme is very efficient because all we are doing is pushing and popping pointers to scraps on a stack array. The main disadvantage is that the array must necessarily be finite and would impose a limit on the depth of indentation nesting.