package com.x5.template;

import java.util.Hashtable;
import java.util.Vector;
import java.util.Enumeration;

// Project Title: Chunk
// Description: Template Util
// Copyright: Copyright (c) 2007
// Author: Tom McClure

/**
 * <P>
 * Chunk is part Hashtable, part StringBuffer, part find-and-replace.
 *
 * <P>
 * Assign an initial template (you can stitch on bits of additional template<BR>
 * content as you go) with placeholder tags -- eg {~myTag} -- and then<BR>
 * set up replacement rules for those tags like so:
 *
 * <PRE>
 *    TemplateSet templates = getTemplates(); // defined elsewhere
 *    Chunk myChunk = templates.makeChunk("my_template");
 *    myChunk.set("myTag","hello tag");
 *    System.out.print( myChunk.toString() );
 * </PRE>
 *
 * <P>
 * <B>NB: Template {~tags} are bounded by curly brackets, not (parentheses).<BR>
 * And don't forget the ~squiggle (aka tilde, pronounced "TILL-duh").</B>  Also,<BR>
 * be careful to always close off {#sub_templates}bla bla bla{#} with a single<BR>
 * hash mark surrounded by curly brackets.
 *
 * <P>
 * TemplateSet is handy if you have a folder with lots of html templates.<BR>
 * Here's an even simpler example, where the template string is supplied<BR>
 * without using TemplateSet:
 *
 * <PRE>
 *    String templateBody = "Hello {~name}!  Your balance is ${~balance}."
 *         + "Pleasure serving you, {~name}!";
 *    Chunk myChunk = new Chunk();
 *
 *    // .add() and .set() may be called in any order
 *    // because tag replacement is delayed until the .toString() call.
 *    myChunk.add( templateBody );
 *    myChunk.set("name", user.getName());
 *    myChunk.set("balance", user.getBalance());
 *    System.out.println( myChunk.toString() );
 *
 *    // reset values and re-use -- original templates are not modified
 *    // when .toString() output is generated.
 *    myChunk.set("name", user2.getName());
 *    myChunk.set("balance", user2.getBalance());
 *    System.out.println( myChunk.toString() );
 *
 * </PRE>
 *
 * <P>
 * The .toString() method transparently invokes the find and replace<BR>
 * functionality.
 *
 * <P>
 * FREQUENTLY ASKED QUESTIONS
 *
 * <P>
 * <B>Q: If I name things just right, will subtemplates get automatically<BR>
 * connected to like-named tags?</B>  ie, if I write a template file like this:
 *
 * <PRE>
 * bla bla bla {~myTemplate} foo foo foo
 * {#myTemplate}Hello {~name}!{#}
 * </PRE>
 *
 * <P>
 * A: No*.  To keep things simple and reduce potential for confusion, Chunk<BR>
 * does not auto-magically fill any tags based on naming conventions or<BR>
 * in-template directives.  You must explicitly define this rule via set:<BR>
 *   set("myTemplate", templates.get("file.myTemplate"));
 *
 * <P>* Actually, this documentation is outdated, and several extensions to the<BR>
 * original template syntax are now available:
 *
 * <P>There is now a powerful new tag modifier syntax which supports:<BR>
 *           * providing default values for tags in-template<BR>
 *           * automating template placement with "includes"<BR>
 *           * in-template text filters including perl-style regex and sprintf<BR>
 *           * macro-style templating<BR>
 *           * extending the system to access alternate template repositories<BR>
 *
 * <P>Complete details are here: <a href="http://www.dagblastit.com/src/template/howto.html">http://www.dagblastit.com/src/template/howto.html</a>
 *
 * <P>
 * <B>Q: My final output says "infinite recursion detected."  What gives?</B>
 *
 * <P>
 * A: You did some variation of this:
 * <PRE>
 *   TEMPLATE:
 *     bla bla bla {~name}
 *     {#name_info}My name is {~name}{#}
 *
 *   CODE:
 *     ...set("name", templates.get("file.name_info"));
 *     ...toString();
 * </PRE>
 *
 * <P>
 * The outer template gets its {~name} tag replaced with "My name is {~name}" --<BR>
 * then, that replacement value is scanned for any tags that might need to be<BR>
 * swapped out for their values.  It finds {~name} and, using the rule you gave,<BR>
 * replaces it with "My name is {~name}" so we now have My name is My name is ...<BR>
 * ad infinitum.
 *
 * <P>
 * This situation is detected by assuming recursion depth will not normally go<BR>
 * deeper than 7.  If you legitimately need to nest templates that deep, you<BR>
 * can flatten out the recursion by doing a .toString() expansion partway<BR>
 * through the nest OR you can tweak the depth limit value in the Chunk.java<BR>
 * source code.
 *
 * <P>
 * <B>Q: Where did my subtemplates go?</B>
 *
 * <P>
 * A: TemplateSet parses out subtemplates and leaves no trace of them in the<BR>
 * outer template where they were defined.  Some people are surprised when<BR>
 * no placeholder tag is automagically generated and left in place of the<BR>
 * subtemplate definition -- sorry, this is not the convention.  For optional<BR>
 * elements the template/code usually looks like this:
 *
 * <PRE>
 *   TEMPLATE:
 *     bla bla bla
 *     {~memberMenu}
 *     {#member_menu}this that etc{#}
 *     foo foo foo
 *
 *   CODE:
 *     if (isLoggedIn) {
 *          myChunk.set("memberMenu", templates.get("file.member_menu"));
 *     } else {
 *          myChunk.set("memberMenu", "");
 *     }
 * </PRE>
 *
 * <P>
 * The subtemplate does not need to be defined right next to the tag where<BR>
 * it will be used (although that practice does promote readability).
 *
 * <P>
 * <B>Q: Are tag names and subtemplate names case sensitive?</B>
 *
 * <P>
 * A: Yes. I prefer to use mixed case in {~tagNames} with first letter<BR>
 * lowercase.  In my experience this aids readability since tags are similar<BR>
 * to java variables in concept and that is the java case convention for<BR>
 * variables.   Similarly, I prefer lowercase with underscores for all<BR>
 * {#sub_template_names}{#} since templates tend to be defined within html<BR>
 * files which are typically named in all lowercase.
 *
 * <P>
 * <B>Q: I defined a whole lot of subtemplates in one file.  Do I have to<BR>
 * specify the filename stub over and over every time I fetch a subtemplate?</B>
 *
 * <P>
 * A: Nope!  You can now make a TemplateSet out of a single template file.<BR>
 * For example, these two pieces of code are equivalent:<BR>
 *
 * <PRE>
 *    // old way
 *    TemplateSet html = getTemplates();
 *    Chunk myTable = html.makeChunk("my_file.table");
 *    Chunk myRow   = html.makeChunk("my_file.row");
 *    Chunk myCell  = html.makeChunk("my_file.cell");
 *
 *    // new way, less repetitious
 *    TemplateSet html = getTemplates();
 *    TemplateSet myHtml = html.getSubset("my_file");
 *    Chunk myTable = myHtml.makeChunk("table");
 *    Chunk myRow   = myHtml.makeChunk("row");
 *    Chunk myCell  = myHtml.makeChunk("cell");
 * </PRE>
 *
 * <P>
 * ADVANCED USES
 *
 * <P>
 * Chunk works great for the simple examples above, but Chunk can do<BR>
 * so much more... the find-and-replace alg is recursive, which is<BR>
 * extremely handy once you get the hang of it.
 *
 * <P>
 * Internally it resists creating an expensive Hashtable until certain<BR>
 * threshhold conditions are reached.
 *
 * <P>
 * Output can be constructed on-the-fly with .append() -- say<BR>
 * you're building an HTML table but don't know ahead of time how<BR>
 * many rows and columns it will contain, just loop through all<BR>
 * your data with calls to .append() after preparing each cell and row:
 *
 * <PRE>
 *    Chunk table = templates.makeChunk("my_table");
 *    Chunk rows = templates.makeChunk();
 *    Chunk row = templates.makeChunk("my_table.my_row");
 *    Chunk cell = templates.makeChunk("my_table.my_cell");
 *
 *    rows.set("backgroundColor", getRowColor() );
 *
 *    while (dataSet.hasMoreData()) {
 *        DataObj data = dataSet.nextDataObj();
 *        String[] attributes = data.getAttributes();
 *
 *        StringBuffer cells = new StringBuffer();
 *        for (int i=0; i &lt; attributes.length; i++) {
 *            cell.set("cellContent", attributes[i]);
 *            cells.append( cell.toString() );
 *        }
 *
 *        row.set("name", data.getName());
 *        row.set("id", data.getID());
 *        row.set("cells",cells);
 *
 *        rows.append( row.toString() );
 *    }
 *
 *    table.set("tableRows",rows);
 *    System.out.println( table.toString() );
 * </PRE>
 *
 * <PRE>
 * Possible contents of my_table.html:
 *
 * &lt;TABLE&gt;
 * {~tableRows}
 * &lt;/TABLE&gt;
 *
 * {#my_row}
 * &lt;TR bgcolor="{~backgroundColor}"&gt;
 *  &lt;TD&gt;{~id} - {~name}&lt;/TD&gt;
 *  {~cells}
 * &lt;/TR&gt;
 * {#}
 *
 * {#my_cell}
 * &lt;TD&gt;{~cellContent}&lt;/TD&gt;
 * {#}
 * </PRE>
 *
 * Copyright: Copyright (c) 2003<BR>
 * Company: <A href="http://www.x5software.com/">X5 Software</A><BR>
 * Updates: <A href="http://www.dagblastit.com/">www.dagblastit.com</A><BR>
 *
 * @author Tom McClure
 * @version 3.0
 */

public class Chunk
{
    public static final int HASH_THRESH = 25;
    public static final int DEPTH_LIMIT = 9;

    protected Object templateRoot = null;
    private String[] firstTags = new String[HASH_THRESH];
    private Object[] firstValues = new Object[HASH_THRESH];
    private int tagCount = 0;
    protected Vector template = null;
    private Hashtable tags = null;
    protected String tagStart = TemplateSet.DEFAULT_TAG_START;
    protected String tagEnd = TemplateSet.DEFAULT_TAG_END;

    private String delayedFilter = null;

    private TemplateSet macroLibrary = null;

    public void setTagBoundaries(String tagStart, String tagEnd)
    {
        this.tagStart = tagStart;
        this.tagEnd = tagEnd;
    }

    public void setMacroLibrary(TemplateSet repository)
    {
        this.macroLibrary = repository;
        if (altSources != null) {
            addProtocol(repository);
        }
    }

    /**
     * Add a String on to the end a Chunk's template.
     */
    public void append(String toAdd)
    {
        // don't bother with overhead of vector until necessary
        if (templateRoot == null && template == null) {
            templateRoot = toAdd;
        } else {
            if (template == null) {
                template = new Vector();
                template.addElement(templateRoot);
            }
            template.addElement(toAdd);
        }
    }

    /**
     * Add a Chunk on to the end of a Chunk's "template" -- this "child"
     * Chunk won't get it's .toString() invoked until the parent Chunk's
     * tags are replaced, ie when the parent Chunk's .toString() method
     * is invoked.  The child does not get cloned in this call so be
     * careful that you know what you're doing if you alter the child's
     * template or rules after attaching it to the parent in this way.
     */
    public void append(Chunk toAdd)
    {
        // if we're adding a chunk we'll almost definitely add more than one.
        // switch to vector
        if (template == null) {
            template = new Vector();
            if (templateRoot != null) template.addElement(templateRoot);
        }
        template.addElement(toAdd);
    }

    /**
     * Creates a find-and-replace rule for tag replacement.  Overwrites any
     * previous rules for this tagName.  Do not include the tag boundary
     * markers in the tagName, ie
     * GOOD: set("this","that")
     * BAD: set("{~this}","that")
     *
     * @param tagName will be ignored if null.
     * @param tagValue will be translated to the empty String if null.
     */
    public void set(String tagName, String tagValue)
    {
        set(tagName, tagValue, "");
    }

    /**
     * Creates a find-and-replace rule for tag replacement.  See remarks
     * at append(Chunk c).
     * @param tagName will be ignored if null.
     * @param tagValue will be translated to the empty String if null.
     * @see append(Chunk c)
     */
    public void set(String tagName, Chunk tagValue)
    {
        set(tagName, tagValue, "");
    }

    /**
     * Create a tag replacement rule, supplying a default value in case
     * the value passed is null.  If both the tagValue and the fallback
     * are null, the rule created will resolve all instances of the tag
     * to the string "NULL"
     * @param tagName tag to replace
     * @param tagValue replacement value -- no-op unless this is of type String or Chunk.
     * @param ifNull fallback replacement value in case tagValue is null
     */
    public void set(String tagName, Object tagValue, String ifNull)
    {
        // all "set" methods eventually chain to here
        if (tagName == null) return;
        // ensure that tagValue is either a String or a Chunk
        if (tagValue != null && !(tagValue instanceof String || tagValue instanceof Chunk)) {
            // bail...
            return;
        }
        if (tagValue == null) {
            tagValue = (ifNull == null) ? "NULL" : ifNull;
        }
        if (tags != null) {
            tags.put(tagName,tagValue);
        } else {
            // to consider: sorted insertion so we can do
            // binary search here instead of sequential scan?
            for (int i=0; i<tagCount; i++) {
                if (firstTags[i].equals(tagName)) {
                    // supplying new value for existing tag
                    firstValues[i] = tagValue;
                    return;
                }
            }
            if (tagCount >= HASH_THRESH) {
                // threshhold reached, upgrade to hashtable
                tags = new Hashtable(HASH_THRESH * 2);
                copyToHashtable();
                tags.put(tagName,tagValue);
            } else {
                firstTags[tagCount] = tagName;
                firstValues[tagCount] = tagValue;
                tagCount++;
            }
        }
    }

    /**
     * Create a tag replacement rule, supplying a default value in case
     * the value passed is null.  If both the tagValue and the fallback
     * are null, a translate-to-empty-string rule is created.
     * @param tagName tag to replace
     * @param tagValue replacement value -- no-op unless this is of type String or Chunk.
     * @param ifNull fallback replacement value in case tagValue is null
     */
    public void set(String tagName, Object tagValue, Chunk ifNull)
    {
        if (tagName == null) return;
        if (tagValue == null) {
            if (ifNull == null) {
                tagValue = "";
            } else {
                tagValue = ifNull;
            }
        }
        if (tagValue instanceof Chunk || tagValue instanceof String) {
            set(tagName, tagValue, "");
        }
    }

    /**
     * For convenience, auto-converts int to String and creates
     * tag replacement rule.  Overwrites any existing rule with this tagName.
     */
    public void set(String tagName, int tagValue)
    {
        set(tagName, Integer.toString(tagValue));
    }

    /**
     * For convenience, auto-converts char to String and creates
     * tag replacement rule.  Overwrites any existing rule with this tagName.
     */
    public void set(String tagName, char tagValue)
    {
        set(tagName, Character.toString(tagValue));
    }

    /**
     * For convenience, auto-converts long to String and creates
     * tag replacement rule.  Overwrites any existing rule with this tagName.
     */
    public void set(String tagName, long tagValue)
    {
        set(tagName, Long.toString(tagValue));
    }

    /**
     * For convenience, auto-converts StringBuffer to String and creates
     * tag replacement rule.  Overwrites any existing rule with this tagName.
     */
    public void set(String tagName, StringBuffer tagValue)
    {
        if (tagValue != null) set(tagName, tagValue.toString());
    }

    /**
     * @return true if a rule exists for this tagName, otherwise false.
     * Returns false if tagName is null.
     */
    public boolean hasValue(String tagName)
    {
        if (tagName == null) return false;
        if (tags != null) {
            return tags.containsKey(tagName);
        } else {
            for (int i=0; i<tagCount; i++) {
                if (firstTags[i].equals(tagName)) return true;
            }
            return false;
        }
    }


    /**
     * @return true if a rule does not yet exist for this tagName, otherwise
     * false. Returns false if tagName is null.
     */
    public boolean stillNeeds(String tagName)
    {
        if (tagName == null) return false;
        return !hasValue(tagName);
    }

    /**
     * Apply all tag replacement rules recursively and return template
     * contents with translated tags.  Rules and original template pieces
     * remain intact, so toString can be called several times, modifying
     * rules between each invocation to produce a slightly different output
     * each time.
     * @return A String with all template pieces assembled and all known tags recursively resolved.
     */
    public String toString()
    {
        return explodeForParent(null);
    }

    private String explodeForParent(Vector ancestors)
    {
        if (template == null && templateRoot == null) return "";
        StringBuffer buf = new StringBuffer();
        if (template == null) {
            explodeAndAppend(templateRoot, buf, ancestors, 1);
        } else {
            for (int i=0; i < template.size(); i++) {
                Object obj = template.elementAt(i);
                explodeAndAppend(obj, buf, ancestors, 1);
            }
        }
        //        return TextFilter.applyTextFilter(delayedFilter, buf.toString());

        // not sure if this is right, but it had the desired effect...
        if (delayedFilter == null) {
            return buf.toString();
        } else {
            // ick, the post-filter output may contain explodable tags
            String postFilter = TextFilter.applyTextFilter(delayedFilter, buf.toString());
            StringBuffer buf2 = new StringBuffer();
            // re-process (hopefully this won't have any weird side-effects)
            explodeAndAppend(postFilter, buf2, ancestors, 1);
            return buf2.toString();
        }
    }

    private void explodeAndAppend(Object obj, StringBuffer buf, Vector ancestors, int depth)
    {
        if (depth >= DEPTH_LIMIT) {
            buf.append("[**ERR** max template recursions: "+DEPTH_LIMIT+"]");
        } else if (obj instanceof String) {
            buf.append(explodeString((String)obj, ancestors, depth));
        } else if (obj instanceof Chunk) {
            if (ancestors == null) ancestors = new Vector();
            ancestors.addElement(this);
            Chunk c = (Chunk) obj;
            buf.append(c.explodeForParent(ancestors));
        }
    }

    /**
     * Retrieves a tag replacement rule.  getTag responds outside the context
     * of recursive tag replacement, so the return value may include unresolved
     * tags.
     * @return The String or Chunk that this tag will resolve to, or null
     * if no rule yet exists.
     */
    public Object getTag(String tagName)
    {
        if (tags != null) {
            return tags.get(tagName);
        } else {
            for (int i=0; i<tagCount; i++) {
                if (firstTags[i].equals(tagName)) {
                    return firstValues[i];
                }
            }
        }
        return null;
    }

    private Hashtable altSources = null;
    private static final java.util.regex.Pattern includeIfPattern =
        java.util.regex.Pattern.compile("^\\.include(If|\\.\\()");

    public void addProtocol(ContentSource src)
    {
        if (altSources == null) {
            altSources = new Hashtable();
            // delayed adding macro library for memory efficiency
            // (avoid overhead of hashtable whenever possible)
            if (macroLibrary != null) {
                altSources.put(macroLibrary.getProtocol(), macroLibrary);
            }
        }
        String protocol = src.getProtocol();
        altSources.put(protocol,src);
    }

    private String altFetch(String tagName, Vector ancestors)
    {
        String tagValue = null;

        if (altSources == null && macroLibrary == null && ancestors == null) {
            // it ain't there to fetch
            return null;
        }

        // the includeIfPattern (defined above)
        // matches ".includeIf" and ".include.(" <-- ie from +(cond) expansion
        if (TextFilter.matches(tagName,includeIfPattern)) {
            // this is either lame or very sneaky
            return TextFilter.translateIncludeIf(tagName,tagStart,tagEnd);
        }

        // parse content source "protocol"
        int delimPos = tagName.indexOf(".",1);
        String srcName = tagName.substring(1,delimPos);
        String itemName = tagName.substring(delimPos+1);

        // for this to work, caller must have already provided an object which
        // implements com.x5.template.ContentSource
        //  -- then templates can delegate to this source using the syntax
        // {~.protocol.itemName}   eg {~.wiki.About_Us}  or {~.include.some_template}

        if (altSources != null) {
            ContentSource fetcher = (ContentSource)altSources.get(srcName);
            // when altSources exists, it handles includes too
            if (fetcher != null) tagValue = fetcher.fetch(itemName);
        } else {
            // if the only alt source is the macro library for includes,
            // no hashtable is made (for memory efficiency)
            if (macroLibrary != null && srcName.equals(macroLibrary.getProtocol())) {
                // include's are special, handle via macroLibrary TemplateSet
                tagValue = macroLibrary.fetch(itemName);
            }
        }

        if (tagValue == null && ancestors != null) {
            // still null? maybe an ancestor knows how to grok
            for (int i=ancestors.size()-1; i>=0 && tagValue == null; i--) {
                Chunk ancestor = (Chunk)ancestors.elementAt(i);
                // lazy... should repeat if/else above to avoid re-parsing the tag
                tagValue = ancestor.altFetch(tagName, null);
            }
        }

        return tagValue;
    }

    // resolveTagValue responds in the context of an explosion tree.
    // ie, if the tag has not been set in this chunk, it goes up the
    // chain of parent chunks for the first one able to resolve the
    // tag into a value.  For example, several row chunks might share
    // a whole-table parent chunk.  Some of the tags in the row are
    // set differently in each row but some will always resolve the
    // same throughout the whole table -- rather than set it over and
    // over the same in each row, the tag is given a value once at the
    // table level.
    protected Object resolveTagValue(String tagName)
    {
        return resolveTagValue(tagName, null);
    }

    protected Object resolveTagValue(String tagName, Vector ancestors)
    {
        //strip off the default if provided eg {~tagName:333} means use 333
        // if no specific value is provided.
        String lookupName = tagName;

        int colonPos = tagName.indexOf(':');
        int pipePos = tagName.indexOf('|');
        if (colonPos > 0 || pipePos > 0) {
            int firstMod = (colonPos > 0) ? colonPos : pipePos;
            if (pipePos > 0 && pipePos < colonPos) firstMod = pipePos;
            lookupName = tagName.substring(0,firstMod);
        }

        Object tagValue = null;

        if (lookupName.charAt(0) == '.') {
            // if the tag starts with a period, we need to delegate
            tagValue = altFetch(lookupName,ancestors);
        } else if (hasValue(lookupName)) {
            // first look in this chunk's own tags
            tagValue = getTag(lookupName);
        } else if (ancestors != null) {
            // now look in ancestors (iteration, not recursion, so sue me)
            for (int i=ancestors.size()-1; i>=0 && tagValue == null; i--) {
                Chunk ancestor = (Chunk)ancestors.elementAt(i);
                tagValue = ancestor.resolveTagValue(lookupName, null);
            }
        }

        // apply filter if provided
        if (tagValue != null) {
            if (pipePos > 0 && tagValue instanceof String) {
                /*
                String filters = parseTagTokens(tagName, pipePos, colonPos)[0];
                // ack! should do this post-expansion
                return TextFilter.applyTextFilter(filters, (String)tagValue);
                */

                // tagValue could be some complex entity --
                // delay filter application until it has been expanded into a string
                Chunk filterMeLater = (macroLibrary == null) ? new Chunk() : macroLibrary.makeChunk();

                // set up filters to be applied from the inside out
                String filter = parseTagTokens(tagName, pipePos, colonPos)[0];
                String[] filters = TextFilter.splitFilters(filter);
                // innermost first...
                // 3rd arg to set is ignored if 2nd arg is non-null,
                // so I'm just passing the filters string to hit the right method
                filterMeLater.set("oneTag",tagValue,filter);
                filterMeLater.append("{~oneTag}");
                filterMeLater.delayedFilter = filters[0];

                // then subsequent filters each wrap a new layer
                for (int i=1; i<filters.length; i++) {
                    Chunk wrapper = (macroLibrary == null) ? new Chunk() : macroLibrary.makeChunk();
                    wrapper.set("oneTag",filterMeLater,filter);
                    wrapper.append("{~oneTag}");
                    wrapper.delayedFilter = filters[i];
                    filterMeLater = wrapper;
                }

                return filterMeLater;
            } else {
                // no filter, no need to subchunk
                return tagValue;
            }
        }

        // reached here? no value supplied.  template might contain a default...
        if (colonPos > 0) {
            String defValue = null;
            String filter = null;
            String order = TextFilter.FILTER_LAST;
            if (pipePos > 0) {
                // apply filter if provided
                String[] tokens = parseTagTokens(tagName, pipePos, colonPos);
                filter   = tokens[0];
                defValue = tokens[1];
                order    = tokens[2];
            } else {
                // everything after the colon is a default value
                defValue = tagName.substring(colonPos+1);
            }

            if (defValue != null && defValue.length() > 0) {
                // now allowing tag/include syntax in the default value
                //
                // eg: {~my_unsupplied_tag:~some_other_tag} morph into another tag
                //     {~my_unsupplied_tag:~.include.some.template} replace w/template
                //     {~my_unsupplied_tag:+some.template} same as above but discouraged (cryptic)
                //
                // or, handled below, nothing fancy
                //     {~my_unsupplied_tag:simple default} default to "simple default"
                //     {~my_unsupplied_tag:} default to empty string
                //
                // but nested tags in the default area are still not allowed:
                //     {~my_unsupplied_tag:not {~other_tag}} NOT VALID
                //     {~my_unsupplied_tag:{~other_tag}} NOT VALID
		char firstChar = defValue.charAt(0);
                if (firstChar == '~' || firstChar == '+' || firstChar == '^') {
                    if (filter == null) {
                        return '{'+defValue+'}';
                    } else if (order.equals(TextFilter.FILTER_FIRST)) {
                        String filtered = TextFilter.applyTextFilter(filter, null);
                        if (filtered != null) {
                            return filtered;
                        } else {
                            return '{'+defValue+'}';
                        }
                    } else {
                        return '{'+defValue+'|'+filter+'}';
                    }
                }
            }
            // reached here?  simple case: no funny chained replacement business
            if (filter != null) {
                if (order.equals(TextFilter.FILTER_FIRST)) {
                    String filtered = TextFilter.applyTextFilter(filter, null);
                    return (filtered != null) ? filtered : defValue;
                } else {
                    return TextFilter.applyTextFilter(filter, defValue);
                }
            } else {
                return defValue;
            }
        } else {
            if (pipePos > 0) {
                // apply filter if provided
                String filter = tagName.substring(pipePos+1);
                return TextFilter.applyTextFilter(filter, null);
            } else {
                return null;
            }
        }
    }

    // pipe denotes a request to apply a filter
    // colon denotes a default value
    // they may come in either order {~tag_name:hello there|url} or {~tag_name|url:hello there}
    //
    // In retrospect, I probably should have considered a nice legible syntax like
    // {~tag_name default="hello there" filter="url"}
    private String[] parseTagTokens(String tagName, int pipePos, int colonPos)
    {
        String filter = null;
        String defValue = null;

        String order = TextFilter.FILTER_LAST;

        if (colonPos < 0) {
            // no colon token, just pipe
            filter = tagName.substring(pipePos+1);
        } else if (pipePos < colonPos) {
            // both tokens, pipe before colon
            //
            // ok, so colon CAN appear inside regex or onmatch() etc
            // these need to be IGNORED!!
            //
            // pipe may NOT appear in default value, so at least we can limit our scan to the final filter
            int finalPipe = TextFilter.grokFinalFilterPipe(tagName,pipePos);
            int nextColon = tagName.indexOf(":",finalPipe+1);
            if (nextColon < 0) {
                // lucked out, colon was fake-out, embedded in earlier filter
                filter = tagName.substring(pipePos+1);
            } else {
                int startScan = TextFilter.grokValidColonScanPoint(tagName,finalPipe+1);
                nextColon = tagName.indexOf(":",startScan);
                if (nextColon < 0) {
                    // colon was fake-out
                    filter = tagName.substring(pipePos+1);
                } else {
                    filter = tagName.substring(pipePos+1,nextColon);
                    defValue = tagName.substring(nextColon+1);
                    order = TextFilter.FILTER_FIRST;
                }
            }
        } else {
            // both tokens, colon before pipe
            filter = tagName.substring(pipePos+1);
            defValue = tagName.substring(colonPos+1,pipePos);
        }

        return new String[]{ filter, defValue, order };
    }

    private String expandIncludes(String template)
    {
        // presto change-o {+template} into {~.include.template}
	// also handles {^protocol.arg} into {~.protocol.arg}
	//
        // is supporting this worth the cost of another TWO passes through the string?
	//
        String p1 = findAndReplace(template, TemplateSet.INCLUDE_SHORTHAND, "{~.include.");
	String p2 = findAndReplace(p1, TemplateSet.PROTOCOL_SHORTHAND, "{~.");
	return p2;
    }

    private String expandMacros(String template)
    {
        template = expandIncludes(template);

        int macroTagBegin = template.indexOf(TemplateSet.MACRO_START);
        if (macroTagBegin < 0) return template;

        // found a macro.  expand!
        StringBuffer expanded = new StringBuffer();
        expanded.append(template.substring(0,macroTagBegin));

        macroTagBegin += TemplateSet.MACRO_START.length();
        int macroTagEnd = template.indexOf(TemplateSet.MACRO_NAME_END,macroTagBegin);
        int macroSectionEnd = template.indexOf(TemplateSet.MACRO_END,macroTagEnd+1);

        if (macroSectionEnd < macroTagEnd+1) {
            return "[template syntax error -- missing macro-end tag? please close off all macros with {__}]";
        }

        String templateRef = template.substring(macroTagBegin,macroTagEnd);
        while (templateRef.endsWith("_")) {
            // {__box} and {__box__} are both allowed until I decide which is prettier/easier
            templateRef = templateRef.substring(0,templateRef.length()-1);
        }

        String macroVars = template.substring(macroTagEnd+1, macroSectionEnd);
        int letBegin = macroVars.indexOf(TemplateSet.MACRO_LET);
        if (letBegin < 0) {
            // no assignments
            String theTemplate = altFetch(".include."+templateRef,null);
            expanded.append( expandMacros(theTemplate) );
        } else {
            // make chunk from templateRef and delegate assignments
            if (letBegin > 0) macroVars = macroVars.substring(letBegin);
            expanded.append( expandMacros( expandMacros2(templateRef, macroVars) ) );
        }

        int allTheRest = macroSectionEnd + TemplateSet.MACRO_END.length();
        expanded.append( expandMacros(template.substring(allTheRest)) );

        return expanded.toString();
    }

    /* if the macro includes variable assignments a la {=var=}value{=}, expandMacros2
     * parses them out and handles the expansion */
    private String expandMacros2(String templateRef, String macroVars)
    {
        if (this.macroLibrary == null) return "";

        Chunk macro = macroLibrary.makeChunk(templateRef);

        int marker = 0;
        int delimPos;
        String varName;
        String varValue;
        while (marker < macroVars.length()) {
            marker += TemplateSet.MACRO_LET.length();
            delimPos = macroVars.indexOf(TemplateSet.MACRO_LET_END,marker);
            varName = macroVars.substring(marker,delimPos);
            while (varName.endsWith("=")) {
                // {=var} and {=var=} are both allowed
                varName = varName.substring(0,varName.length()-1);
            }
            delimPos += TemplateSet.MACRO_LET_END.length();
            marker = macroVars.indexOf(TemplateSet.MACRO_LET,delimPos);
            if (marker < delimPos) marker = macroVars.length();
            varValue = macroVars.substring(delimPos,marker);
            macro.set(varName,varValue);
        }

        // this will trigger a standard expansion which will
        // properly reference all ancestor tag values.
        // otherwise defaults will be used incorrectly when there
        // are actually expansion values available up/down the tree.
        String macroTag = getNextMacroTag();
        this.set(macroTag, macro);
        return this.tagStart + macroTag + this.tagEnd;
    }

    private int macroCounter = 0;

    private String getNextMacroTag()
    {
        // these tags are unlikely to collide with application-supplied tags
        String macroTag = "CHUNK_-_MACRO_-_"+macroCounter;
        macroCounter++;
        return macroTag;
    }

    private int findMatchingEndBrace(String template, int searchFrom)
    {
        int endPos = template.indexOf(tagEnd,searchFrom);
        if (endPos < 0) return endPos;

        // tricky business: ignore all tag markers inside a regex

        // search backwards for regex start
        int x = endPos;
        int regexPos = 0;
        boolean isMatchOnly = false;
        while (regexPos == 0 && x > searchFrom+1) {
            char c = template.charAt(x);
            if (c == '/') {
                char preC = template.charAt(x-1);
                if (preC == 's' && template.charAt(x-2) == '|') {
                    regexPos = x-1;
                } else {
                    if (preC == 'm') preC = template.charAt(x-2); // skip over optional m
                    if (preC == ',' || preC == '(') {
                        regexPos = x-1;
                        isMatchOnly = true;
                    }
                }
            }
            x--;
        }
        // no regex? valid endPos
        if (regexPos == 0) return endPos;

        // found regex? find end, make sure this end brace is not inside the regex.
        int regexMid = TextFilter.nextRegexDelim(template,regexPos+2);
        int regexEnd = (isMatchOnly) ? regexMid : TextFilter.nextRegexDelim(template,regexMid+1);

        if (endPos > regexEnd) {
            // brace is outside the regex, valid endpoint
            return endPos;
        } else {
            // invalid brace, is inside regex.  recurse from here.
            return findMatchingEndBrace(template, regexEnd+1);
        }
    }

    // the core search-and-replace routine
    private String explodeString(String template, Vector ancestors, int depth)
    {
        template = expandMacros(template);

        StringBuffer buf = new StringBuffer();

        int begin, end;
        int tagStartLen = tagStart.length();
        int tagEndLen = tagEnd.length();

        int marker = 0;
        while ((begin = template.indexOf(tagStart,marker)) > -1) {
            // found a tag.  everything up to here has no more tags,
            // so... put in the can!
            if (begin > marker) buf.append(template.substring(marker, begin));

            begin += tagStartLen;
            // find end of tag
            if ((end = findMatchingEndBrace(template,begin)) > -1) {
                String tagName = template.substring(begin,end);
                Object tagValue = resolveTagValue(tagName, ancestors);
                // unresolved tags get put back the way we found
                // them in case the final String which explode() returns
                // is then fed into another Chunk which *does* have
                // a value.
                if (tagValue == null) {
                    buf.append(tagStart);
                    buf.append(tagName);
                    buf.append(tagEnd);
                } else {
                    explodeAndAppend(tagValue, buf, ancestors, depth+1);
                }
                marker = end + tagEndLen;
            } else {
                // somebody didn't end a tag...
                // leave broken tagstart and move along
                buf.append(tagStart);
                marker = begin;
            }
        }
        if (marker == 0) {
            // no tags found
            return template;
        } else {
            buf.append(template.substring(marker));
            return buf.toString();
        }
    }


    /**
     * Clears all tag replacement rules.
     */
    public void resetTags()
    {
        if (tags != null) {
            tags.clear();
        } else {
            tagCount = 0;
        }
    }

    /**
     * Adds multiple find-and-replace rules using all entries in the
     * Hashtable.  Replaces an existing rule if tagNames collide.
     * Assumes keys are strings and values are either type String
     * or type Chunk.
     */
    public void setMultiple(Hashtable rules)
    {
        if (rules == null || rules.size() <= 0) return;
        Enumeration e = rules.keys();
        while (e.hasMoreElements()) {
            String tagName = (String) e.nextElement();
            set(tagName,rules.get(tagName),"");
        }
    }

    /**
     * Adds multiple find-and-replace rules using all rules from the passed
     * Chunk.  Replaces any existing rules with the same tagName.
     */
    public void setMultiple(Chunk copyFrom)
    {
        if (copyFrom != null) {
            Hashtable h = copyFrom.getTagsTable();
            setMultiple(h);
        }
    }

    /**
     * Retrieve all find-and-replace rules.  Alterations to the returned
     * Hashtable WILL AFFECT the tag replacement rules of the Chunk directly.
     * Does not return a clone.
     * @return a Hashtable containing the Chunk's find-and-replace rules.
     */
    public Hashtable getTagsTable()
    {
        if (tags != null) {
            return tags;
        } else {
            if (tagCount <= 0) {
                return null;
            } else {
                copyToHashtable();
                return tags;
            }
        }
    }

    private void copyToHashtable()
    {
        if (tags == null) tags = new Hashtable(tagCount);
        for (int i=0; i<tagCount; i++) {
            tags.put(firstTags[i],firstValues[i]);
        }
    }

    /**
     * Useful utility function.  An efficient find-and-replace-all algorithm
     * for simple cases when regexp would be overkill.  IMO they should have
     * included this with String.
     * @param x text body.
     * @param find text to search for in x.
     * @param replace text to insert in place of "find" -- defaults to empty String if null is passed.
     * @return a new String based on x with all instances of "find" replaced with "replace"
     */
    public static String findAndReplace(String toSearch, String find, String replace)
    {
        if (find == null || toSearch == null || toSearch.indexOf(find) == -1) return toSearch;
        if (replace == null) replace = "";
        int marker=0, findPos, findLen = find.length();
        StringBuffer sb = new StringBuffer();
        while ((findPos = toSearch.indexOf(find,marker)) > -1) {
            sb.append(toSearch.substring(marker,findPos));
            sb.append(replace);
            marker = findPos+findLen;
        }
        sb.append(toSearch.substring(marker));
        return sb.toString();
    }

}