LWS: How Can I Display/Update My Dynamic Fields?


This is going to be a short one. I’m on the road and I feel compelled to just get one out. Kind of like a hairball only more useful.

Let’s explicitly state a few assumptions.

Solr supports the creation of dynamic fields.

LucidWorks Search 2.6.2 has the following entry for dynamic fields (with comments and other offending text removed):

<dynamicField name="*_i"  type="int"    indexed="true"  stored="true"/>
<dynamicField name="*_s"  type="string"  indexed="true"  stored="true"/>
<dynamicField name="*_l"  type="long"   indexed="true"  stored="true"/>
<dynamicField name="*_t"  type="text_en"    indexed="true"  stored="true"/>
<dynamicField name="*_b"  type="boolean" indexed="true"  stored="true"/>
<dynamicField name="*_f"  type="float"  indexed="true"  stored="true"/>
<dynamicField name="*_d"  type="double" indexed="true"  stored="true"/>
<dynamicField name="*_tiled"  type="double" indexed="true"  stored="false"/>
<dynamicField name="*_dt" type="date"    indexed="true"  stored="true"/>
<dynamicField name="*_p"  type="location" indexed="true" stored="true"/>
<dynamicField name="random_*" type="random" />

<dynamicField name="attr_*" type="string" indexed="true" stored="true" multiValued="true" />

<!-- THIS IS THE ONE I CARE ABOUT -->
<dynamicField indexed="true" multiValued="true" name="*" stored="true" type="text_en"/>

The above fields are easily understood as regular expressions for incoming field names. If the field totalAmount_s came in we know that, based on the list above, it would be field of type string that would both be indexed and stored. The same would occur if we had a field called attr_boy with one difference: it would be a field of type string and it would be multi-valued (my favorite).

Notice the last entry I highlighted. That’s the one I care about. It is the catch-all definition. If a field comes in and the system has never heard of it then Solr will create it as a field of type text_en, it would be indexed and stored and it would be multi-valued.

This is like encountering Alien for the first time. It is perfect. The field is of type text_en, which means that when it is indexed it is searchable with all the goodness that that means, it will be stored, and it can hold multiple values. It brings a tear to my eyes just thinking about it. A dynamic field that does not need to adhere to any naming convention, can hold multiple values, and it will be added without a second thought.

Perfect.

Just for yucks:

  1. Create a folder and place the alien-solr.xml file in it (the file’s contents can be seen below)
  2. Create a collection named alien-field-test. Create a data source of type Solr XML, call it alien-content, and point it at the folder where you placed alien-solr.xml
  3. Run the crawl

When you click on the Tool item in the menubar and execute an empty search it will return to you the one document you just crawled. Click on the rather interesting GUID the system created as the id and take a look at the fields that were created.

1-solrxml-result

Cool, right?

Not so cool is when you go to Indexing -> Fields and look for all the a- fields. Not a one to be found. What is the story with that?

The story is that this page is only meant to display the fields that have been statically defined in schema.xml.

Outrageous, you say? Indefensible, you say? Why did you wake me up, you say?

This is what I mean: when you go to the page at http://localhost:8989/admin/collections/alien-field-test/fields (your server name might be different), the a- fields are not present.

Go ahead and look. I can wait.

If instead you entered http://localhost:8989/admin/collections/alien-field-test/fields?include_dynamic=true (notice the bolded parameter include_dynamic) then you will see the new fields.

2-fields-w-dynamic-fields

Now it is cool again.

If you were to click on any of those fields and save them explicitly they would appear forever on the Field page without the use of the include_dynamic parameter (the others would continue to act like the ghosts they are). The act of saving them gives them form, substance, reality. Probably more than you were bargaining for, but you can now treat them like any other field without having to play tricks to get them to behave in new and novel ways.

And you don’t have to worry about them hunting down users and eating them (no matter what you might want).

The File

alien-solr.xml

<add>
  <doc>
    <field name="a-ripley">Sigourney Weaver</field>
    <field name="a-dallas">Tom Skeritt</field>
    <field name="a-lambert">Veronica Cartwright</field>
    <field name="a-brett">Harry Dean Stanton</field>
    <field name="a-kane">John Hurt</field>
    <field name="a-ash">Ian Holm</field>
    <field name="a-parker">Yaphet Kotto</field>
    <field name="a-mother">Helen Horton</field>
    <field name="a-alien">Eddie Powell</field>
  </doc>
</add>

Disclosures

Carlos Valcarcel is a full time employee of LucidWorks, but lives in New York as he prefers hurricanes to earthquakes. Having worked at IBM, Microsoft, and Fast Search and Transfer the only thing he is sure of is that the font editor he wrote on his Atari 800 was the coolest program he has ever written. While questions can be a drag he admits that answers will be harder to give without them.

The cat isn’t real, but then neither are you. Enjoy your search responsibly.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s