1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><title>Using the GEDCOM parser library</title>
4 <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"></head><body text="#000000" bgcolor="#ffffff" link="#000099" vlink="#990099" alink="#000099">
6 <h1 align="center">Using the GEDCOM parser library</h1>
12 <li><a href="#anchor">Overview</a></li>
13 <li><a href="#Error_handling">Error handling</a></li>
14 <li><a href="#Data_callback_mechanism">Data callback mechanism</a></li>
17 <li><a href="#Start_and_end_callbacks">Start and end callbacks</a></li>
18 <li><a href="#Default_callbacks">Default callbacks</a></li>
21 <li><a href="#Other_API_functions">Other API functions</a></li>
24 <li><a href="#Debugging">Debugging</a></li>
25 <li><a href="#Error_treatment">Error treatment</a></li>
26 <li><a href="#Compatibility_mode">Compatibility mode</a></li>
28 </ul><li><a href="#Converting_character_sets">Converting character sets</a></li><li><a href="#Support_for_configure.in">Support for configure.in</a><br>
32 <li><a href="interface.html">Interface details</a><br>
37 <hr width="100%" size="2">
38 <h2><a name="Overview"></a>Overview<br>
40 The GEDCOM parser library is built as a callback-based parser (comparable
41 to the SAX interface of XML). It comes with:<br>
44 <li>a library (<code>libgedcom.so</code>), to be linked in the application
46 <li>a header file (<code>gedcom.h</code>), to be used in the sources
47 of the application program</li>
48 <li>a header file (<code>gedcom-tags.h</code>) that is also installed,
49 but that is automatically included via <code>gedcom.h</code><br>
53 Next to these, there is also a data directory in <code>$PREFIX/share/gedcom-parse</code>
54 that contains some additional stuff, but which is not immediately
55 important at first. I'll leave the description of the data directory
58 The very simplest call of the gedcom parser is simply the following
59 piece of code (include of the gedcom header is assumed, as everywhere in
62 <blockquote><code>int result;<br>
64 result = <b>gedcom_parse_file</b>("myfamily.ged");<br>
66 Although this will not provide much information, one thing it does
67 is parse the entire file and return the result. The function returns
68 0 on success and 1 on failure. No other information is available using
69 this function only.<br>
71 The next sections will refine this to be able to have meaningful errors
72 and the actual data that is in the file.<br>
74 <hr width="100%" size="2">
75 <h2><a name="Error_handling"></a>Error handling</h2>
76 Since this is a relatively simple topic, it is discussed before the
77 actual callback mechanism, although it also uses a callback...<br>
79 The library can be used in several different circumstances, both terminal-based
80 as GUI-based. Therefore, it leaves the actual display of the error
81 message up to the application. For this, the application needs to register
82 a callback before parsing the GEDCOM file, which will be called by the library
83 on errors, warnings and messages.<br>
85 A typical piece of code would be:<br>
87 <blockquote><code>void <b>my_message_handler</b> (Gedcom_msg_type type,
93 <b>gedcom_set_message_handler</b>(my_message_handler);<br>
95 result = <b>gedcom_parse_file</b>("myfamily.ged");</code><br>
97 In the above piece of code, <code>my_message_handler</code> is the callback
98 that will be called for errors (<code>type=ERROR</code>), warnings (<code>type=WARNING</code>) and messages (<code>type=MESSAGE</code>). The
99 callback must have the signature as in the example. For errors, the
100 <code> msg</code> passed to the callback will have the format:<br>
102 <blockquote><code>Error on line</code> <i><lineno></i>: <i><actual_message></i><br>
104 Note that the entire string will be properly internationalized, and
105 encoded in UTF-8 (<a href="encoding.html">Why UTF-8?</a>). Also,
106 no newline is appended, so that the application program can use it in any
107 way it wants. Warnings are similar, but use "Warning" instead of
108 "Error". Messages are plain text, without any prefix.<br>
110 With this in place, the resulting code will already show errors and
111 warnings produced by the parser, e.g. on the terminal if a simple <code>
112 printf</code> is used in the message handler.<br>
114 <hr width="100%" size="2">
115 <h2><a name="Data_callback_mechanism"></a>Data callback mechanism</h2>
116 The most important use of the parser is of course to get the data out
117 of the GEDCOM file. As already mentioned, the parser uses a callback
118 mechanism for that. In fact, the mechanism involves two levels.<br>
120 The primary level is that each of the sections in a GEDCOM file is notified
121 to the application code via a "start element" callback and an "end element"
122 callback (much like in a SAX interface for XML), i.e. when a line containing
123 a certain tag is parsed, the "start element" callback is called for that
124 tag, and when all its subordinate lines with their tags have been processed,
125 the "end element" callback is called for the original tag. Since GEDCOM
126 is hierarchical, this results in properly nested calls to appropriate "start
127 element" and "end element" callbacks.<br>
129 However, it would be typical for a genealogy program to support only
130 a subset of the GEDCOM standard, certainly a program that is still under
131 development. Moreover, under GEDCOM it is allowed for an application
132 to define its own tags, which will typically not be supported by another
133 application. Still, in that case, data preservation is important;
134 it would hardly be accepted that information that is not understood by a
135 certain program is just removed.<br>
137 Therefore, the second level of callbacks involves a "default callback".
138 An application needs to subscribe to callbacks for tags it does support,
139 and need to provide a "default callback" which will be called for tags
140 it doesn't support. The application can then choose to just store
141 the information that comes via the default callback in plain textual format.<br>
143 After this introduction, let's see what the API looks like...<br>
146 <h3><a name="Start_and_end_callbacks"></a>Start and end callbacks</h3>
148 <h4><i>Callbacks for records</i> <br>
150 As a simple example, we will get some information from the header of
151 a GEDCOM file. First, have a look at the following piece of code:<br>
153 <blockquote><code>Gedcom_ctxt <b>my_header_start_cb</b> (int level,
155
156 Gedcom_val xref, <br>
157
158 char *tag, <br>
159
160 char *raw_value,<br>
161
162 int parsed_tag, <br>
163
164 Gedcom_val parsed_value)<br>
166 printf("The header starts\n");<br>
167 return (Gedcom_ctxt)1;<br>
170 void <b>my_header_end_cb</b> (Gedcom_ctxt self)<br>
172 printf("The header ends, context is %d\n", (int)self); /* context
173 will print as "1" */<br>
177 <b>gedcom_subscribe_to_record</b>(REC_HEAD, my_header_start_cb,
178 my_header_end_cb);<br>
180 result = <b>gedcom_parse_file</b>("myfamily.ged");</code><br>
182 Using the <code>gedcom_subscribe_to_record</code> function, the application
183 requests to use the specified callbacks as start and end callback. The end
184 callback is optional: you can pass <code>NULL</code> if you are not interested
185 in the end callback. The identifiers to use as first argument to
186 the function (here <code>REC_HEAD</code>) are described in the <a href="interface.html#Record_identifiers">
187 interface details</a>.<br>
189 From the name of the function it becomes clear that this function is
190 specific to complete records. For the separate elements in records
191 there is another function, which we'll see shortly. Again, the callbacks
192 need to have the signatures as shown in the example.<br>
194 The <code>Gedcom_ctxt</code> type that is used as a result of the start
195 callback and as an argument to the end callback is vital for passing context
196 necessary for the application. This type is meant to be opaque; in
197 fact, it's a void pointer, so you can pass anything via it. The important
198 thing to know is that the context that the application returns in the start
199 callback will be passed in the end callback as an argument, and as we will
200 see shortly, also to all the directly subordinate elements of the record.<br>
202 The <code>tag</code> is the GEDCOM tag in string format, the <code>parsed_tag</code>
203 is an integer, for which symbolic values are defined as <code>TAG_HEAD,</code>
204 <code>TAG_SOUR,</code> <code>TAG_DATA,</code> ... and <code>USERTAG </code><code></code>
205 for the application-specific tags. These values are defined in the
206 header <code>gedcom-tags.h</code> that is installed, and included via <code>
207 gedcom.h</code> (so no need to include <code>gedcom-tags.h</code> yourself).<br>
209 The example passes a simple integer as context, but an application could
210 e.g. pass a <code>struct</code> (or an object in a C++ application) that will contain the information for the
211 header. In the end callback, the application could then e.g. do some
212 finalizing operations on the <code>struct</code> to put it in its database.<br>
214 (Note that the <code>Gedcom_val</code> type for the <code>xref</code>
215 and <code>parsed_value</code> arguments was not discussed, see further
219 <h4><i>Callbacks for elements</i></h4>
220 We will now retrieve the SOUR field (the name of the program that wrote
221 the file) from the header:<br>
223 <blockquote><code>Gedcom_ctxt <b>my_header_source_start_cb</b>(Gedcom_ctxt
225
226 int
227 level,<br>
228
229 char*
230 tag,<br>
231
232 char*
233 raw_value,<br>
234
235 int
236 parsed_tag,<br>
237
238 Gedcom_val
239 parsed_value)<br>
241 char *source = GEDCOM_STRING(parsed_value);<br>
242 printf("This file was written by %s\n", source);<br>
243 return parent;<br>
246 void <b>my_header_source_end_cb</b>(Gedcom_ctxt parent,<br>
247
248 Gedcom_ctxt self,<br>
249
250 Gedcom_val parsed_value)<br>
252 printf("End of the source description\n");<br>
256 <b>gedcom_subscribe_to_element</b>(ELT_HEAD_SOUR,<br>
257
258 my_header_source_start_cb,<br>
259
260 my_header_source_end_cb);<br>
262 result = <b>gedcom_parse_file</b>("myfamily.ged");</code><br>
264 The subscription mechanism for elements is similar, only the signatures
265 of the callbacks differ. The signature for the start callback shows
266 that the context of the parent line (here e.g. the <code>struct</code> that
267 describes the header) is passed to this start callback. The callback
268 itself returns here in this example the same context, but this can be its own context object
269 of course. The end callback is called with both the context of the
270 parent and the context of itself, which in this example will be the same.
271 Again, the list of identifiers to use as a first argument for the
272 subscription function are detailed in the <a href="interface.html#Element_identifiers">
273 interface details</a> .<br>
275 If we look at the other arguments of the start callback, we see the
276 level number (the initial number of the line in the GEDCOM file), the tag
277 (e.g. "SOUR"), and then a raw value, a parsed tag and a parsed value. The
278 raw value is just the raw string that occurs as value on the line next to
279 the tag (in UTF-8 encoding). The parsed value is the meaningful value
280 that is parsed from that raw string. The parsed tag is described in
281 the section for record callbacks above.<br>
283 The <code>Gedcom_val</code> type is meant to be an opaque type. The
284 only thing that needs to be known about it is that it can contain specific
285 data types, which have to be retrieved from it using pre-defined macros.
286 These data types are described in the <a href="interface.html#Gedcom_val_types">
287 interface details</a>. <br>
289 Some extra notes:<br>
292 <li>The <code>Gedcom_val</code> argument of the end callback
293 is currently not used. It is there for future enhancements.</li>
294 <li>There are also two <code>Gedcom_val</code> arguments in
295 the start callback for records. The first one (<code>xref</code>) contains the <code>xref_value</code> corresponding to the cross-reference (or <code>NULL</code> if there isn't one), the second one (<code>parsed_value</code>) contains the value that is parsed from the <code>raw_value</code>. See the <a href="interface.html#Record_identifiers">interface details</a>.</li>
299 <h3><a name="Default_callbacks"></a>Default callbacks<br>
301 As described above, an application doesn't always implement the entire
302 GEDCOM spec, and application-specific tags may have been added by other applications.
303 To preserve this extra data anyway, a default callback can be registered
304 by the application, as in the following example:<br>
306 <blockquote><code>void <b>my_default_cb</b> (Gedcom_ctxt parent,
307 int level, char* tag, char* raw_value, int parsed_tag)<br>
313 <b>gedcom_set_default_callback</b>(my_default_cb);<br>
315 result = <b>gedcom_parse_file</b>("myfamily.ged");</code><br>
317 This callback has a similar signature as the previous ones,
318 but it doesn't contain a parsed value. However, it does contain the
319 parent context, that was returned by the application for the most specific
320 containing tag that the application supported.<br>
322 Suppose e.g. that this callback is called for some tags in the header
323 that are specific to some other application, then our application could make
324 sure that the parent context contains the struct or object that represents
325 the header, and use the default callback here to add the level, tag and
326 raw_value as plain text in a member of that struct or object, thus preserving
327 the information. The application can then write this out when the
328 data is saved again in a GEDCOM file. To make it more specific, consider
329 the following example:<br>
331 <blockquote><code>struct header {<br>
332 char* source;<br>
334 char* extra_text;<br>
337 Gedcom_ctxt my_header_start_cb(int level, Gedcom_val xref, char* tag,
339
340 int parsed_tag, Gedcom_val parsed_value)<br>
342 struct header head = my_make_header_struct();<br>
343 return (Gedcom_ctxt)head;<br>
346 void my_default_cb(Gedcom_ctxt parent, int level, char* tag, char* raw_value,
349 struct header head = (struct header)parent;<br>
350 my_header_add_to_extra_text(head, level, tag, raw_value);<br>
353 gedcom_set_default_callback(my_default_cb);<br>
354 gedcom_subscribe_to_record(REC_HEAD, my_header_start, NULL);<br>
356 result = gedcom_parse_file(filename);</code><br>
358 Note that the default callback will be called for any tag that isn't
359 specifically subscribed upon by the application, and can thus be called
360 in various contexts. For simplicity, the example above doesn't take
361 this into account (the <code>parent</code> could be of different
362 types, depending on the context).<br>
364 Note also that the default callback is not called when the parent context is <code>NULL</code><code></code>. This is e.g. the case if none of the "upper" tags has been subscribed upon.<br>
366 <hr width="100%" size="2">
368 <h2><a name="Other_API_functions"></a>Other API functions<br>
370 Although the above describes the basic interface of libgedcom, there
371 are some other functions that allow to customize the behaviour of the library.
372 These will be explained in the current section.<br>
374 <h3><a name="Debugging"></a>Debugging</h3>
375 The library can generate various debugging output, not only from itself,
376 but also the debugging output generated by the yacc parser. By default,
377 no debugging output is generated, but this can be customized using the
378 following function:<br>
380 <blockquote><code>void <b>gedcom_set_debug_level</b> (int level,
381 FILE* trace_output)</code><br>
383 The <code>level</code> can be one of the following values:<br>
386 <li>0: no debugging information (this is the
388 <li>1: only debugging information from libgedcom
390 <li>2: debugging information from libgedcom and
394 If the <code>trace_output</code> is <code>NULL</code>, debugging information
395 will be written to <code>stderr</code>, otherwise the given file handle
396 is used (which must be open).<br>
399 <h3><a name="Error_treatment"></a>Error treatment</h3>
400 One of the previous sections already described the callback to be registered
401 to get error messages. The library also allows to customize what
402 happens on an error, using the following function:<br>
404 <blockquote><code>void <b>gedcom_set_error_handling</b> (Gedcom_err_mech
405 mechanism)</code><br>
407 The <code>mechanism</code> can be one of:<br>
411 <li><code>IMMED_FAIL</code>: immediately fail the
412 parsing on an error (this is the default)</li>
413 <li><code>DEFER_FAIL</code>: continue parsing after
414 an error, but return a failure code eventually</li>
415 <li><code>IGNORE_ERRORS</code>: continue parsing
416 after an error, return success always</li>
420 This doesn't influence the generation of error or warning messages, only
421 the behaviour of the parser and its return code.<br>
425 <h3><a name="Compatibility_mode"></a>Compatibility mode<br>
427 Applications are not necessarily true to the GEDCOM spec (or use a different
428 version than 5.5). The intention is that the library is resilient
429 to this, and goes in compatibility mode for files written by specific programs
430 (detected via the HEAD.SOUR tag). This compatibility mode can be
431 enabled and disabled via the following function:<br>
434 <blockquote><code>void <b>gedcom_set_compat_handling</b>
435 (int enable_compat)</code><br>
437 The argument can be:<br>
441 <li>0: disable compatibility mode</li>
442 <li>1: allow compatibility mode (this is the default)<br>
447 Note that, currently, no actual compatibility code is present, but this
448 is on the to-do list.<br>
449 <hr width="100%" size="2">
450 <h2><a name="Converting_character_sets"></a>Converting character sets</h2>
451 All strings passed by the GEDCOM parser to the application are in UTF-8 encoding.
452 Typically, an application needs to convert this to something else to
453 be able to display it.<br>
455 The most common case is that the output character set is controlled by the <code>locale</code> mechanism (i.e. via the <code>LANG</code>, <code>LC_ALL</code> or <code>LC_CTYPE</code> environment variables), which also controls the <code>gettext</code>
456 mechanism in the application. <br>
463 The source distribution of <code>gedcom-parse</code> contains an example implementation (<code>utf8-locale.c</code> and <code>utf8-locale.h</code>
464 in the top directory). Feel free to use it in
465 your source code (it is not part of the library, and it isn't installed anywhere,
466 so you need to take over the source and header file in your application).
470 Its interface is:<br>
471 <blockquote><pre><code>char *<b>convert_utf8_to_locale</b> (char *input, int *conv_failures);<br>char *<b>convert_locale_to_utf8</b> (char *input);<br></code></pre></blockquote>
473 Both functions return a pointer to a static buffer that is overwritten on
474 each call. To function properly, the application must first set the
475 locale using the <code>setlocale</code> function (the second step detailed below).
476 All other steps given below, including setting up and closing down the conversion
477 handles, are transparantly handled by the two functions. <br>
479 If you pass a pointer to an integer to the first function, it will be set
480 to the number of conversion failures, i.e. characters that couldn't be converted;
481 you can also just pass <code>NULL</code> if you are not interested (note that usually, the interesting information is just whether there <i>were</i>
482 conversion failures or not, which is then given by the integer being bigger
483 than zero or not). The second function doesn't need this, because any
484 locale can be converted to UTF-8.<br>
487 You can change the "?" that is output for characters that can't be converted
488 to any string you want, using the following function before the conversion
490 <blockquote><pre><code>void <b>convert_set_unknown</b> (const char *unknown);</code></pre></blockquote>
492 If you want to have your own functions for it instead of this example implementation, the following steps need to
493 be taken by the application (more detailed info can be found in the info
494 file of the GNU libc library in the "Generic Charset Conversion" section
495 under "Character Set Handling" or online <a href="http://www.gnu.org/manual/glibc-2.2.3/html_chapter/libc_6.html#SEC99">here</a>):<br>
497 <li>inclusion of some headers:</li>
501 <pre><code>#include <locale.h> /* for setlocale */<br>#include <langinfo.h> /* for nl_langinfo */<br>#include <iconv.h> /* for iconv_* functions */<br></code></pre>
505 <li>set the program's current locale to what the user configured in the environment:</li>
509 <pre><code>setlocale(LC_ALL, "");</code><br></pre>
513 <li>open a conversion handle for conversion from UTF-8 to the character set of the current locale (once for the entire program):</li>
517 <pre><code>iconv_t iconv_handle;<br>...<br>iconv_handle = iconv_open(nl_langinfo(CODESET), "UTF-8");</code><br>if (iconv_handle == (iconv_t) -1)<br> /* signal an error */<br></pre>
521 <li>then, every string can be converted using the following:</li>
525 <pre><code>/* char* in_buf is the input buffer, size_t in_len is its length */<br>/* char* out_buf is the output buffer, size_t out_len is its length */<br><br>size_t nconv;<br>char *in_ptr = in_buf;<br>char *out_ptr = out_buf;<br>nconv = iconv(iconv_handle, &in_ptr, &in_len, &out_ptr, &out_len);</code></pre>
528 <blockquote>If the output buffer is not big enough, <code>iconv</code> will return -1 and set <code>errno</code> to <code>E2BIG</code>. Also, the <code>in_ptr</code> and <code>out_ptr</code> will point just after the last successfully converted character in the respective buffers, and the <code>in_len</code> and <code>out_len</code> will be updated to show the remaining lengths. There can be two strategies here:<br>
530 <li>Make sure from the beginning
531 that the output buffer is big enough. However, it's difficult to find
532 an absolute maximum length in advance, even given the length of the input
536 <li>Do the conversion in several steps, growing the output buffer each time to make more space, and calling <code>iconv</code>
537 consecutively until the conversion is complete. This is the preferred
538 way (a function could be written to encapsulate all this).</li>
540 Another error case is when the conversion was unsuccessful (if one of the
541 characters can't be represented in the target character set). The <code>iconv</code> function will then also return -1 and set <code>errno</code> to <code>EILSEQ</code>; the <code>in_ptr</code> will point to the character that couldn't be converted. In that case, again two strategies are possible:<br>
543 <li>Just fail the conversion, and show an error. This is not very user friendly, of course.<br>
546 <li>Skip over the character that can't be converted and append a "?" to the output buffer, then call <code>iconv</code> again. Skipping over a UTF-8 character is fairly simple, as follows from the <a href="http://www.cl.cam.ac.uk/%7Emgk25/unicode.html#utf-8">encoding rules</a>:</li>
550 <li>if the first byte is in binary 0xxxxxxx, then the character is only one byte long, just skip over that byte<br>
553 <li>if the first byte is in binary 11xxxxxx, then skip over that byte and all bytes 10xxxxxx that follow.<br>
559 <li>eventually, the conversion handle needs to be closed (when the program exits):<br>
564 <pre><code>iconv_close(iconv_handle);<br></code></pre>
567 The example implementation
568 mentioned above grows the output buffer dynamically and outputs "?" for characters
569 that can't be converted.<br>
571 <hr width="100%" size="2">
572 <h2><a name="Support_for_configure.in"></a>Support for configure.in</h2>
573 Programs using the GEDCOM parser library and using autoconf to configure
574 their sources can use the following statements in configure.in (the example
575 is checking for gedcom-parse, version 1.34):<br>
576 <blockquote><code>AC_CHECK_LIB(gedcom, gedcom_parse_file,,<br>
577 AC_MSG_ERROR(Cannot find libgedcom: Please install gedcom-parse))<br>
578 AC_MSG_CHECKING(for libgedcom version)<br>
580 #include <stdio.h><br>
581 #include <stdlib.h><br>
582 #include <gedcom.h><br>
586 if (GEDCOM_PARSE_VERSION >= 1034) exit(0);<br>
589 ac_gedcom_version_ok='yes',<br>
590 ac_gedcom_version_ok='no',<br>
591 ac_gedcom_version_ok='no')<br>
592 if test "$ac_gedcom_version_ok" = 'yes' ; then<br>
593 AC_MSG_RESULT(ok)<br>
595 AC_MSG_RESULT(not ok)<br>
596 AC_MSG_ERROR(You need at least version 1.34 of gedcom-parse)<br>
600 There are three preprocessor symbols defined for version checks in the header:<br>
602 <li><code>GEDCOM_PARSE_VERSION_MAJOR</code></li>
603 <li><code>GEDCOM_PARSE_VERSION_MINOR</code></li>
604 <li><code>GEDCOM_PARSE_VERSION</code><br>
607 The last one is equal to <code>(GEDCOM_PARSE_VERSION_MAJOR * 1000) + GEDCOM_PARSE_VERSION_MINOR.</code><br>
608 <hr width="100%" size="2">
610 <pre><font size="-1">$Id$<br>$Name$</font><br></pre>