======================================================================== 378 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 0612; Sun, 30 Sep 90 15:52:45 EDT Date: Sun, 30 Sep 90 15:47:28 -0400 Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: yeongw@psi.com (Wengyik Yeong) To: "MARK HINNEBUSCH, FCLA" Subject: Proposed Explain draft Cc: yeongw@psi.com The 'Explain' Extension to ANSI Z39.50 Version 2 1. Introduction The Information Retrieval application service and protocol specified in ANSI Z39.50 Version 2 is intended to be general in scope, providing information retrieval capabilities to a wide range of possible applications. In light of this, certain sections of the service and protocol definition have been left unspecified so as not to limit the applicability of the information retrieval facilities offered. The omission of portions of the service and protocol definition gives rise to a problem however: the infrastructure within which an information retrieval application is to operate is incomplete as a result. This proposal is an attempt at a solution to the problem outlined, by providing a discovery mechanism by which information retrieval applications may dynamically obtain the additional infrastructural information required for operation. This document first proposes an extension to the information retrieval facilities described in ANSI Z39.50 Version 2 in the form of an additional discovery facility called 'Explain'. Protocol extensions in support of the new 'Explain' facility and its associated service element are then developed. 2. The 'Explain' facility The goal of the 'Explain' facility is to provide origins with the ability to retrieve predefined infrastructural information pertaining to the environment supported on a target, such as databases that may be searched. To this end, a collection of 'explain topics' are defined as part of the 'Explain' facility, with each 'explain topic' representing an item of infrastructural information which may be required by an origin in order to conduct a meaningful Information Retrieval session with a target. The 'Explain' facility consists of a single application layer service element, the 'Explain Service Element'. 2.1 Explain Service Element The 'Explain' facility is realized by the 'Explain Service Element'. Origins utilize the 'Explain' facility by sending an 'Explain Request' to a target consisting of a list of 'explain topics' for which information is requested. The target then returns the information pertaining to the 'explain topics' requested in an 'Explain Response', if possible, after performing any necessary resource control or access control. Parameters to the 'Explain Service Element' are: Parameter Request Response Explain-topics x Explain-result - x Explain-information - x (if applicable) Reference-id x (optional) x (if applicable) 2.1.1 Explain-topics A list of 'explain topics' for which information is requested by the origin. Current valid topics are: (1) 'List-Databases': Provide a listing of the names of databases available for searching on the target. (2) 'Explain-Database': Provide a (textual) description of the content of the database named, which would be useful to an end-user. (3) 'List-Database-Attributes': Provide a listing of the attributes available in performing searches on the database named. (4) 'Explain-Database-Attribute': Provide a (textual) description of the attribute named in the context of the database named which would be useful to an end-user. (5) 'List-Database-Element-Sets': Provide a listing of the names of legal element sets that may be used in retrieving database records. (6) 'Explain-Database-Element-Set': Provide a (textual) description of the element set named, in the context of the database named, that would be useful to the end-user. (7) 'Explain-Database-Default-Element-Set': Provide the name of the default element set for the database named. (8) 'List-Database-Record-Syntaxes': Provide a list of the abstract syntaxes that may be used to define the structure of records returned from the database named. (9) 'List-Database-Default-Record-Syntax': Provide the identity of the default abstract syntax which defines the structure of records returned from the database named. 2.1.2 Explain-result An indication from the target to the origin as to the status of the request for information on the 'explain topics' listed in the request corresponding to the response of which this 'Explain-Result' is a part. Valid return indications are: Success All information requested was available. Failure-Message-Size Not all the information on the 'explain topics' requested can be returned due to the negotiated preferred message size being exceeeded. As much information as possible is being returned without exceeding the negotiated preferred message size. Failure-Access-Control Not all the information on the 'explain topics' requested can be returned due to an access control failure. As much information as possible is being returned. Failure-Resource-Control Not all the information on the 'explain topics' requested can be returned due to an resource control failure. As much information as possible is being returned. Failure-Unspecified Due to an unspecified failure at the target, not all (if any) of the information on the 'explain topics' requested can be returned. 2.1.3 Explain-information A list of the information on each 'explain topic' that was requested in the request corresponding to the response of which this 'Explain-information' is a part. Each item of information is a tuple consisting of the 'explain-topic', and the information corresponding to the 'explain topic' in the form predefined for information on the 'explain topic'. 2.1.4 Reference-id See section 3.4 in the ANSI Z39.50 Version 2 draft. 3. Protocol Modifications In support of the proposed 'Explain' facility and its service element, some modifications need to be made to the Information Retrieval protocol defined in ANSI-Z39.50-2. These modifications fall into two categories: modifications to the Information Retrieval Protocol Machines, and additions to the ASN.1 definition. 3.1 Protocol Machine modifications The following are the changes that need to be made to the State Table for Origin Systems in section 4.2.3 of the ANSI-Z39.50-2 document: (1) Two new incoming events for origins are defined, 'Explain request', and 'Explain Response PDU'. (2) Two new outgoing actions for origins are defined, 'Explain Request PDU', and 'Explain confirm'. (3) State 'Open' is modified so that if event 'Explain Request' occurs, an 'Explain Request PDU' action is performed to generate a request to the target and a transition is made to state 'Explain sent'. (4) A new state is defined 'Explain sent', in which the following event-action-new_state tuples are legal: Event Action New_State Explain Response PDU Explain confirm Open Rsc PDU Rsc ind; stkst Rsctrl recvd Acc PDU Acc ind; stkst Acctrl Recvd Aab ind Iab ind closed Apab ind Iab ind closed Iab req Aab req closed The following are the changes that need to be made to the State Table for Target Systems in section 4.2.3 of the ANSI-Z39.50-2 document: (1) Two new incoming events for targets are defined, 'Explain response', and 'Explain Request PDU'. (2) Two new outgoing actions for origins are defined, 'Explain Response PDU', and 'Explain indication'. (3) State 'Open' is modified so that if event 'Explain Request PDU' occurs, an 'Explain indication' action is performed, and a transition is made to state 'Explain received'. (4) A new state is defined 'Explain received', in which the following event-action-new_state tuples are legal: Event Action New_State Explain response Explain Response PDU Open Rsc req Rsc PDU; stkst Rsctrl sent Acc req Acc PDU; stkst Acctrl sent Aab ind Iab ind closed Apab ind Iab ind closed Iab req Aab req closed 3.2 ASN.1 Additions The following are the additions to be made to the ASN.1 definition in the ANSI-Z39.50 draft: ExplainRequest ::= SEQUENCE { referenceId ReferenceId OPTIONAL, explainTopics [113] IMPLICIT SEQUENCE OF ExplainTopic } ExplainResponse ::= SEQUENCE { referenceId ReferenceId OPTIONAL, explainResult ExplainResult, explainInformation [114] IMPLICIT SEQUENCE OF ExplainInformationElement OPTIONAL } ExplainTopic ::= CHOICE { listDatabases [1] IMPLICIT NULL, explainDatabase ExplainDatabase, listDatabaseAttributes ListDatabaseAttributes, explainDatabaseAttribute ExplainDatabaseAttribute, listDatabaseElementSets ListDatabaseElementSets, explainDatabaseElementSet ExplainDatabaseElementSet, explainDatabaseDefaultElementSet ExplainDatabaseDefaultElementSet, listDatabaseRecordSyntaxes ListDatabaseRecordSyntaxes, listDatabaseDefaultRecordSyntax ListDatabaseDefaultRecordSyntax } ExplainResult ::= [115] IMPLICIT INTEGER { success (0), failureMessageSize (1), failureAccessControl (2), failureResourceControl (3), failureUnspecified (4) } ExplainDatabase ::= [2] IMPLICIT DatabaseName ListDatabaseAttributes ::= [3] IMPLICIT DatabaseName ExplainDatabaseAttribute ::= [4] IMPLICIT SEQUENCE { databaseName DatabaseName, attribute AttributeElement } ListDatabaseElementSets ::= [5] IMPLICIT DatabaseName ExplainDatabaseElementSet ::= [6] IMPLICIT SEQUENCE { databaseName DatabaseName, elementSetName RecordCompositionId } ExplainDatabaseDefaultElementSet ::= [7] IMPLICIT DatabaseName ListDatabaseRecordSyntaxes ::= [8] IMPLICIT DatabaseName ListDatabaseDefaultRecordSyntax ::= [9] IMPLICIT DatabaseName ExplainInformationElement ::= CHOICE { listDatabasesElement ListDatabasesElement, explainDatabaseElement ExplainDatabaseElement, listDatabaseAttributesElement ListDatabaseAttributesElement, explainDatabaseAttributeElement ExplainDatabaseAttributeElement, listDatabaseElementSetsElement ListDatabaseElementSetElement, explainDatabaseElementSetElement ExplainDatabaseElementSetElement, explainDatabaseDefaultElementSetElement ExplainDatabaseDefaultElementSetElement, listDatabaseRecordSyntaxesElement ListDatabaseRecordSyntaxesElement, ListDatabaseDefaultRecordSyntaxElement ListDatabaseDefaultRecordSyntaxElement } ListDatabasesElement ::= [1] IMPLICIT SEQUENCE OF DatabaseName ExplainDatabaseElement ::= [2] IMPLICIT SEQUENCE { databaseName DatabaseName, explanation ExplainText } ListDatabaseAttributesElement ::= [3] IMPLICIT SEQUENCE { databaseName DatabaseName, attributeList SEQUENCE OF SEQUENCE { attributeName VISIBLESTRING, attribute AttributeElement } } ExplainDatabaseAttributeElement ::= [4] IMPLICIT SEQUENCE { databaseName DatabaseName, attribute AttributeElement, explanation ExplainText } ListDatabaseElementSetsElement ::= [5] IMPLICIT SEQUENCE { databaseName DatabaseName, elementSetList SEQUENCE OF RecordCompositionId } ExplainDatabaseElementSetElement ::= [6] IMPLICIT SEQUENCE { databaseName DatabaseName, elementSetName RecordCompositionId, explanation ExplainText } ExplainDatabaseDefaultElementSetElement ::= [7] IMPLICIT SEQUENCE { databaseName DatabaseName, elementSetName RecordCompositionId } ListDatabaseRecordSyntaxesElement ::= [8] IMPLICIT SEQUENCE { databaseName DatabaseName, recordSyntaxList SEQUENCE OF OBJECT IDENTIFIER } ListDatabaseDefaultRecordSyntaxElement ::= [9] IMPLICIT SEQUENCE { databaseName DatabaseName, defaultRecordSyntax OBJECT IDENTIFIER } ExplainText ::= [116] IMPLICIT OCTETSTRING -- an octet string so -- multibyte character sets -- can be supported In addition to the above additions, it is necessary to modify the definition of the 'Z39-50-APDU'. After modification, the APDU is defined as follows: Z39-50-APDU ::= CHOICE { initRequest [20] IMPLICIT InitRequest, initResponse [21] IMPLICIT InitResponse, searchRequest [22] IMPLICIT SearchRequest, searchResponse [23] IMPLICIT SearchResponse, presentRequest [24] IMPLICIT PresentRequest, presentResponse [25] IMPLICIT PresentResponse, deleteRequest [26] IMPLICIT DeleteRequest, deleteResponse [27] IMPLICIT DeleteResponse, accessControlRequest [28] IMPLICIT AccessControlRequest, accessControlResponse [29] IMPLICIT AccessControlResponse, resourceControlRequest [30] IMPLICIT ResourceControlRequest, resourceControlResponse [31] IMPLICIT ResourceControlResponse, explainRequest [32] IMPLICIT ExplainRequest, explainResponse [33] IMPLICIT ExplainResponse } ======================================================================== 121 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 6344; Sat, 29 Sep 90 15:55:36 EDT To: "MARK HINNEBUSCH, FCLA" Subject: A philosophical problem with Cliff's Explain draft Reply-To: yeongw@psi.com Cc: yeongw@psi.com Date: Sat, 29 Sep 90 15:51:20 -0400 Sender: "Z3950IW@NERVM via" From: yeongw@psi.com Mark Hinnebusch was kind enough to provide me with a copy of Cliff's Explain draft of December 1989 (which is annotated ZIG90-9, so I guess that's its ZIG number). I found it somewhat difficult to read because of the use of terminology different from that in the Z39.50 draft, which makes the Explain proposal difficult to understand in the context of Z39.50. However, that is perfectly understandable, given the current lack of maturity of Explain. I'm sure that the terminology of the Explain draft will be aligned with that of the Z39.50 document when Explain matures. Based on my understanding of the Explain draft, and my own perceptions of the functionality that needs to be provided by Explain, I have a general philosophical problem with the functionality of an Explain service being provided through the use of an additional database, as opposed to by the addition of new service elements to Z39.50. To put it simply, I don't think that the range of information that has to be provided by Explain can be meaningfully represented as records with a common structure. And that is exactly what I consider a database to be: a store of information represented as a collection of records with a common structure. Take for example two possible items of information that could reasonably be expected to be provided by Explain: a list of the databases that a Z39.50 target provides for searching, and a description of a database given its name. The former is a list, consisting of repeated instances of data elements all of which have a common, identifiable structure. The latter, on the other hand, is "free form", in that it possesses no internal structure. While a record type which is essentially a large discriminated union can certainly be (artificially) imposed, I think that imposing such structure would violate the spirit of what a database is. Similarly, consider the problem of searching a database providing the functionality of an Explain service. At one level, a search of a database can be viewed as the process of identifying records that are possessed of certain 'desirable' attributes. For example, at this level, a search of a bibliographic database for books with the word "industry" in the title could be viewed as the process of identifying all bibliographic records whose "title word" attribute had among its value "industry" (assuming that the database only contained information on books). Now examine the problem of 'searching' an Explain database. As I pointed out in the previous paragraph, records in such a database would be possessed of no common structure, save perhaps that they may all be 'branches' of a discriminated union at the highest level of the definition of their structure. As such, they will share no common attribute, making searching (at the level I defined above) difficult, to say the least :-). Now a reading of Cliff's Explain draft reveals that the draft defines both "access points" (which I take to be synonymous with "search attributes" or "search indices") and "expanation formats" (synonymous with "record structure"?). So it would seem that Cliff has essentially provided an existence proof that a database can indeed provide Explain functionality. I submit however that this is not the case, which I will attempt to demonstrate in the following. [Cliff: please don't take the next few paragraphs personally. I'm not trying to attack you, or ridicule your work. The concept of Explain in itself is a major innovation, but I think you are having a problem with not being able to see the woods for the trees, and am just trying to inject a viewpoint from someone a little bit further removed from Explain than you are] I submit however that a more careful reading will reveal that the "access points" do not actually correspond to characteristics of the data, but are in fact artificially introduced 'discriminators' that exist solely to discriminate between the various types (not values) of information in the database. For example, the 'explanation-name' access point has a possible instantiation ("explanation", or "explanatory segment", the two of which I take to be synonymous) of "DATABASES". I contend that neither is "DATABASES" an intrinsic value that may be taken on by any component of a data record, nor is 'explanation-name' intrinsically a characteristic of any record. With regard to explanation formats, Cliff appears to have arrived at a common format by taking the least common denominator and declaring everything to be "free form" (I understand that SGML and Postcript both have embedded formatting information, but the information embedded does not contribute in any way to the ability to decompose the data for the purposes of further processing). While this certainly solves the problem of providing a common format for transmitting the data, it is at the cost of severely limiting the processing options available to the recipient of the data. I contend that the more structure the data received has, the more processing options there are available, and that taking the least common denominator of "free form" limits processing options to only one: displaying it to the user. As such, I think that providing Explain functionality through an additional database is inappropriate. I will be the first to admit that the idea was attractive: I thought that the concept of Explain, and its realization as an additional database was a truly elegant solution to a pressing problem until I read the draft. Cliff's points about generality and building on existing protocol infrastructure still hold. However I regard those points as moot now: it doesn't really matter how many savings result from a solution, if the solution is inappropriate to the problem. So I suggest that the Explain functionality be provided through the addition of a new Explain facility to Z39.50. Basically, the Explain service would simply consist of a request from the origin to a target for certain categories of information, with the categories being parameters of the service call. The target would then return the information requested in a format that is predefined for each category of information. This I think, is the more appropriate realization of Explain, as compared to an additional database. I realize that I've only outlined the Explain facility in very vague terms. I will rectify that with a later posting containing my detailed proposal for an Explain facility. Then Cliff can return the favor by picking my proposal apart too :-) :-). Wengyik ======================================================================== 128 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 8074; Fri, 28 Sep 90 17:19:13 EDT To: "MARK HINNEBUSCH, FCLA" Subject: Double Wrapping: a request Reply-To: yeongw@psi.com Cc: yeongw@psi.com Date: Fri, 28 Sep 90 15:38:59 -0400 Sender: "Z3950IW@NERVM via" From: yeongw@psi.com I've been looking at the ASN.1 definitions in Z39.50 Version 2, and have seen many places where ASN.1 constructs are "double wrapped", unnecessarily, in my opinion. So, I'd like to make a pitch here for the removal of (unnecessary) double wrapping. Double wrapping adds (unnecessary) complexity to the code to convert from BER encoded constructs to the structures native to a Z39.50 implementation and vice-versa. As a result, the practice of double wrapping constructs incurs unnecessary system overhead when a Z39.50 implementation is used. In addition, unnecessary bytes are transmitted (the extra id and length of the second wrapper) across network, a waste of telecommunications resources as well. In addition, the double wrapped constructs are just added work in those implementations that hand-craft ASN.1 parsers/encoders, as opposed to using ASN.1 compilers. In definitions where not everything is uniformly double wrapped (the Z39.50 Version 2 ASN.1 definitions being such a beast), it also means unnecessary code to deal with the inconsistency. For example, at one level, both the ProtocolVersion and Options constructs are bitstrings. But whereas ProtocolVersion is defined as ProtocolVersion [3] IMPLICIT BITSTRING and has only one wrapper, Options is defined as Options [4] BITSTRING with two wrappers (the complete bitstring, with BITSTRING id and length is wrapped in a new type with Context Specific ID of 4 and another length). If not for this inconsistency, the two constructs are the same, except for the ID (which can be easily parameterized in the parsing function), and could be parsed with the same function, a significant optimization when applied to all the constructs defined in Z39.50. [The optimization is irrelevant to those who use ASN.1 compilers that automagically generate one parsing function, one encoding function, and one data structure per ASN.1 construct, but it is very significant to those whose compilers are capable of such optimizations, and to the implementors that hand-code, of course.] Perhaps most importantly from the point of view of this group, since most of us are here with the intent of producing Z39.50 implementations, double wrapping is an obstacle towards interoperability. The second wrapper is just one extra construct that has to be dealt with, one more opportunity to make a mistake in implementation. All it takes is one overlooked second wrapper to destroy any chance of interoperability. Why risk the added aggravation for an extraneous wrapper? In my opinion, the bare minimum of tagging/wrapping possible should be used in the Z39.50 definition, while still allowing unambiguous parsing/encoding in implementations. To do otherwise would a waste of resources in my opinion: system resources in dealing with the extra wrappers, telecommunications resources in transmitting the extra bytes, and programmer resources in coding/debugging the extra ASN.1 modules. [The last saving applies only to implementors that hand-craft their ASN.1 handling code, but since it makes no difference to those using ASN.1 compilers whether or not a construct is double-wrapped, could I request that the first group of implementors with intentions of hand-crafting be indulged please?] I realize that this request may fall into the "frivolous" category, since it is essentially an issue of optimization in implementations. In fairness, whether or not ASN.1 constructs are double-wrapped, and whether or not the double-wrapping is done consistently, Z39.50 will work/not work regardless. However the possible optimizations that can result from consistently using the minimum possible tagging/wrapping necessary can be very significant. Also, implementation considerations aside, it makes the ASN.1 definition more compact, 'cleaner' and more internally consistent. Since the move from DIS --> IS is supposed to incorporate changes from implementation experience, I'd like to request that this be considered as derived from "implementation experience". Although I haven't actually gone through and done a complete Z39.50 implementation, I have no doubt that if I did write code to do ASN.1 handling, consistently single wrapped ASN.1 would be a great help. That said, here are the specific changes I'm proposing: Page Construct Proposed new definition ---- --------- ----------------------- asn1-3 Options [4] IMPLICIT BITSTRING asn1-4 smallSetElementSetNames [100] IMPLICIT ElementSetNames asn1-4 mediumSetElementSetNames [101] IMPLICIT ElementSetNames asn1-5 Argument None; This can be deleted asn1-5 RPNStructure CHOICE { [0] Operand, [1] IMPLICIT SEQUENCE { RPNStructure, RPNStructure, Operator } asn1-10 DatabaseName (in [0] IMPLICIT DatabaseName NamePlusRecord construct) asn1-11 PreferredRecordSyntax [104] IMPLICIT OBJECT IDENTIFIER asn1-13 ReferenceId [2] IMPLICIT OCTETSTRING asn1-13 resourceReportID [1] IMPLICIT OBJECT IDENTIFIER asn1-13 estimates [2] IMPLICIT SEQUENCE OF Estimate asn1-13 message [3] IMPLICIT VisibleString asn1-13 EstimateType [1] IMPLICIT INTEGER asn1-13 EstimateValue [2] IMPLICIT INTEGER In the above, I've reproduced (barring typos on my part) the constructs as they appear in the ASN.1 definition, except for indentation and OPTIONAL tags being dropped. As such there are syntax errors. I would be happy to reproduce the above with the syntax errors corrected. Thanks. Wengyik P.S. If it turns out that all the constructs I listed above are defined that way because of typos, please disregard the paragraphs dealing with how double-wrapping is evil :-) :-). Having the typos fixed would be appreciated though. ======================================================================== 52 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 0422; Wed, 26 Sep 90 16:29:16 EDT To: "MARK HINNEBUSCH, FCLA" Subject: Re: z39.50 access to OCLC Cc: YEONGW@psi.com Reply-To: yeongw@psi.com Date: Wed, 26 Sep 90 16:25:08 -0400 Sender: "Z3950IW@NERVM via" From: yeongw@psi.com > I received the following questions from Wengyik Yeong and thought they were > too good to keep private. My appologies to Wengyik for making them public. Thanks for the response. No apology is necessary. > When the system goes into production, all the databases available under EPIC, > which will include full-text applications, will be available. I realize that shipping anything but bibliographic information is a long way off, but what format are you planning to ship non-bibliographic information in? I may be showing my ignorance here (again :-)), but to my knowledge there is no commonly-agreed on format for the interchange of non-bibliographic information. > > 9) Will you be using the same [addressing information] > > as in our experiment > Shouldn't the application context be "ANSIZ39-50-2"? Is ANSIZ39-50-2 a registered application context? As far as I know, it is only the name of the ASN.1 module in the V2 draft. Actually, I don't really care what it's called, as long as we understand what it means, and you don't reject whatever object identifier I send at you to represent it in the associate. > My understanding is that > the Telenet node is going away, so testing will be over the Internet. I don't > know what the address for the application will be. I think I wasn't sufficiently clear in my question, sorry. I wasn't referring to the Network Layer addressing (ie., X.25 address, TCP/IP address), but to the addressing in the layers above, which exist only in the OSI stack you're running (ISODE, I guess). As far as I know, we can make up whatever upper-layer addresses we want provided that they don't conflict with existing OSI applications, as long as we all consistently use the same addresses. I obviously didn't make this clear. Of course, my using the wrong terminology didn't help things either :-) :-). I was actually trying to obtain the various layer selectors. Wengyik ======================================================================== 99 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 2755; Wed, 26 Sep 90 10:38:59 EDT Date: Wed, 26 Sep 90 10:41:45 EDT Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: rrl@rsch.oclc.org (Ralph LeVan) To: "MARK HINNEBUSCH, FCLA" Subject: z39.50 access to OCLC I received the following questions from Wengyik Yeong and thought they were too good to keep private. My appologies to Wengyik for making them public. > From yeongw@psi.com Wed Sep 26 09:17:09 1990 > Hi, > > Your mention of OCLC's intention to provide a TCP/IP and Z39.50 based service > has piqued my interest. I'd like to ask some questions, and would appreciate > it if you could take the time to answer them as best as you can: > > 1) Will bibliographic information be the only type of information accessible > by way of this service? When the system goes into production, all the databases available under EPIC, which will include full-text applications, will be available. But, for our initial inter-operability testing, only a subset of our bibliographic database will be available. > 2) Will the service initially be based on Z39.50 V1 with an eventual move > to Z39.50 V2, or will it be based on Z39.50 V2 right from the start? We'll be supporting V2 definitely. Our z39.50-DBI translator is table driven and we may support our original version of V1 as well, but that is not certain. > 3) How will the databases that can be searched be named? In our experiment > of last year, they were named "0" and "3". Will this still be the case? Probably. > 4) What element sets will be supported? "0" (full record) and "32767" > (author/title/subject) again? Will you support "F" as well? Probably. It wouldn't be very friendly of us not to. > 5) Will Id/Authentication be an OCTETSTRING consisting of > name/password/new password again? Yes. A suggestion was made at the meeting that Anonymous be accepted with access only to the Explain database. I like that idea, and will try to allow anonymous access to the whole system for interoperability testing. > 6) Will you be supporting the Type 1 Query, the Type 0 Query in the native > OCLC format, or both? Both. And, if we can agree on the extensions necessary to Type 1 to support proximity searching, we will probably roll that into our Type 1 support as quickly as possible. > 7) Which attribute set will you be supporting for search purposes, > SR-bib1, NISO-bib1 or the native OCLC attribute set? Or some combination > of the three? Gosh. I haven't really thought about it, but I guess SR-bib1. It doesn't work with our other databases, but will get us through testing. > 8) Since my understanding is that this will be a 'real' service, and > not an experiment, I assume you will be supporting access control and > resource control of some sort. Will it be access and resource control > as specified in Z39.50, or some native (to OCLC) access and resource > control format? We can live without access and resource control until the implementors can agree on what they look like. > 9) Will you be using the same application context ("IR Z39.50", > 1.17.1.12.2), same address (TSEL=1025, SSAPADDR="", PSAPADDR="") > as in our experiment, or is OCLC planning to officially register > the service? Similarly, will you be requesting "iso-asn1-abstract-syntax" > as your abstract syntax during A-ASSOCIATE? Will your target attempt > to negotiate alternate abstract syntaxes? Shouldn't the application context be "ANSIZ39-50-2"? My understanding is that the Telenet node is going away, so testing will be over the Internet. I don't know what the address for the application will be. > 10) In what format will your Z39.50 target ship bibliographic data > back to requesting origins? OCLC's BER encoded MARC? Either our BER encoded MARC or "true" MARC. > Thanks in advance for the time taken to answer my questions, > > Wengyik > Ralph LeVan Internet: rrl@rsch.oclc.org Online Computer Library Center UUCP: ...!saqqara!sppy00!rrl 6565 Frantz Rd, Dublin OH 43017 Phone: (614) 761-6115 ======================================================================== 27 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 0929; Wed, 26 Sep 90 09:01:26 EDT Date: Wed, 26 Sep 90 06:01:36 PDT To: "MARK HINNEBUSCH, FCLA" Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: "Ray Denenberg - LC" Subject: SR IS ballot and ASN.1 changes This is in response to Wengyik Yeong's questions about changes to ASN.1 in SR IS ballot to correst syntax errors: A syntax error IS a "truly compelling reason" for a change. But that is the type of error that often does not get caught until IS ballot, for two reasons: (1) prior to IS ballot, no one wants to scrutinize the syntax carefully enough, because it is not yet stable, and (2) often, no one compiles it until they are ready to implement it, which is only after it reaches DIS. So when I referred to errors discovered as a result of the implementation process, that included minor syntax errors. I don't consider any syntax error (major or minor)to be frivolous. I'm sorry I didn't make that clear. We at LC have compiled the ASN.1 and are putting together a list of errors discovered, which we will suggest be included as U.S. ballot comments. I urge anyone else who finds syntax errors to do likewise ======================================================================== 26 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 9876; Wed, 26 Sep 90 08:37:06 EDT Date: Wed, 26 Sep 90 05:37:33 PDT To: "MARK HINNEBUSCH, FCLA" Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: "Ray Denenberg - LC" Subject: IS SR ballot This is in response to Wengyik yeongs SECOND set of questions. I suggest that people with multiple question-sets maintain sequential numbering rather than beginning each set with (1), or alternately, assign identifiers to your question-sets. Right about (1), (3), and (4). These are typo's which will be corrected. Thank you. On question (2) regarding making default 'preferredRecordSyntax' part of the application context -- I think the application context is the wrong place for this, but Wayne Davison is more articulate on matters of application context, and I will suggest to him that he address this. On questions (5) and (6) -- there are one or two additional appendices yet to be drafted, which will cover these. ======================================================================== 66 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 6136; Tue, 25 Sep 90 17:48:38 EDT To: "MARK HINNEBUSCH, FCLA" Cc: yeongw@psi.com Reply-To: yeongw@psi.com Date: Tue, 25 Sep 90 17:44:10 -0400 Sender: "Z3950IW@NERVM via" From: yeongw@psi.com Subject: double wrapping Thanks for the responses to my questions. However I have to question some of the responses themselves. > > 4) In the definition of the Type 1 Query .......... was there > > some reason for the double-wrapping? > Yes there was a reason. It was to provide explicit tagging for the > parsing of these very complex recursive structures. The explicit > tagging might help especially when you have to build your own > parser. Now, if all commercial implementations will support > recursion (and we know now that there is some question about that) > then these extra tags might be extraneous. I'm sorry I don't understand the reason. Essentially, all I'm suggesting is that we stop double-wrapping things in 'Argument'. 'RPNStructure' and 'Operand' are still explicitly tagged, so getting rid of 'Argument' doesn't make building parsers any more difficult. In fact, getting rid of 'Argument' makes building parsers that much easier (one less module to debug), and reduces the potential for errors (one less construct to screw up in encoding, one less construct to be corrupted in transmission). For those implementors who are using ASN.1 compilers that generate one structure for every ASN.1 construct, the change would also mean one less native language structure to have to deal with, which may or may not be a big deal. With regard to the process involved in getting the change made, I'm sympathetic, believe me. I have half a mind to just live with it. The change is more an efficiency/design issue than a "showstopper" in the sense that Z39.50 will not work if the change is not made. However the alternative of propagating the inefficiency into an International Standard just because it was discovered at an inconvenient point in the standards process seems wrong too. By the way, on the subject of "the process", am I to understand that no changes are to be made unless there is a truly compelling reason for the change? Here's the problem: to date I've been raising only the more important problems I see with the Z39.50 V2 document. I've been ignoring things like bad ASN.1 syntax on the assumption that somebody was eventually going to fix syntax problems. Is this not true? Because there *are* problems with the ASN.1 syntax, most notably the SEQUENCE definitions that have identifiers that begin with capitals (eg: 'IdAuthentication' in the Initialize PDU, 'Result' in the InitializeResponse etc.) Anyway, most of the syntax problems are trivial to fix, and could easily be viewed as frivolous. However I think they need to be made, and the statement that "frivolous changes are frowned on" concerns me. [I'm not volunteering to go through the document and find all the syntax problems. Somebody with an ASN.1 compiler can just run it on the definition and send the Maintenance Organization the error output :-) :-)] Wengyik ======================================================================== 62 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 4655; Tue, 25 Sep 90 16:21:25 EDT Date: Tue, 25 Sep 90 13:13:14 PDT To: "MARK HINNEBUSCH, FCLA" Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: "Ray Denenberg - LC" Subject: Z39.50 V2 questions Response to questions from Wengyik yeong. > 1) Are names (database names, result set names etc.) case >sensitive? All those defined as VisibleString (which I think all are) are case sensitive. > 2) In result sets, are records numbered beginning with 0 or 1? Result set records are referenced beginning with 1. See 3.2.2.1, last sentence. >3) In the references to bit strings in the ASN.1, does "first bit" > refer to the leftmost or rightmost bit in the bit string? Leftmost. See ISO 8825, section 9.2.1. > 4) In the definition of the Type 1 Query .......... was there > some reason for the double-wrapping? Yes there was a reason. It was to provide explicit tagging for the parsing of these very complex recursive structures. The explicit tagging might help especially when you have to build your own parser. Now, if all commercial implementations will support recursion (and we know now that there is some question about that) then these extra tags might be extraneous. Consider the PROCESS for changing this. We would first want to get it changed in SR. That means making it a U.S. comment on the current IS ballot. Frivolous comments on an IS ballot are frowned upon (I'm not saying this would be frivolous). For a draft standard which is considered "stable", comments on an IS ballot pertaining to ASN.1 should result from evaluation processes not available prior to the IS ballot. For example, someone tries to implement only after a standard reaches DIS. Once they try to implement, they discover a necessary change. Therefore, I suggest that all implementors who have tried to implement the ASN.1 description evaluate, based on implementation experience, whether the suggested change is desirable. If there is a consensus that it is, I will assume responsibility to forward this suggestion to the U.S. group responsible for the SR comments. >5) Result Set Name ... and Result Set Id ... seem to have the > same semantics. In the ASN.1 definition, they have distinct tags. > Is there any reason for this? Yes. ResultSetName is used to NAME a (new) result set, and ResultSetId is used to IDENTIFY an (existing) result set. This distinction is particularly important in the search, containing a ResultSetName in the body, and which possibly includes a query containing a resultSetId. Questions (6) and (7) I'll leave to Clifford to respond to. ======================================================================== 54 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 2211; Tue, 25 Sep 90 15:12:10 EDT Date: Tue, 25 Sep 90 14:38:01 -0400 Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: yeongw@psi.com (Wengyik Yeong) To: "MARK HINNEBUSCH, FCLA" Subject: more comments/questions on Z39.50 V2 Cc: yeongw@psi.com I did some more reading, and have some more questions/comments: 1) In the definition of the Initialize Response PDU, the field 'maximumMessageSize' is defined to be of type 'MaximumMessageSize', which is not further defined. I think this is a typo, and that the field should be of type 'MaximumRecordSize'. This change would make the Initialize Response consistent with the Initialize Request also. 2) I notice that the 'preferredRecordSyntax' field in both the Search Request and the Present Request PDUs is marked OPTIONAL. What's the default record syntax if the field is optional? I would suggest making it part of the application context definition (actually I'm suggesting that language to that effect be put in the standard). 3) The ASN.1 type 'ElementSetNames' is not defined in the ASN.1 definition. On the other hand, there is a 'RecordComposition' type defined which is not used anywhere as far as I can see, but which looks remarkably like element sets from Version 1. I think this is another typo ... 4) The 'resourceReport' field in the Resource Control Request PDU is defined to be a 'VisibleString', whereas the 'ResourceReport' type is defined but not used. I think that 'resourceReport' should be of type 'ResourceReport'. Another typo? 5) What's the application context (object identifier) to be used in setting up Z39.50 sessions (during A-ASSOCIATE)? I realize that there will undoubtedly be several, but could I request that at least one be made "first among equals" with an Appendix along the lines of Appendices C and D devoted to it? It gets real difficult to interoperate if everybody starts using different application contexts ... :-) [This is maybe a ZIG issue, not a standards issue. I'm basically just trying to get agreement on a common application context within some group here]. 6) What's the abstract syntax to be used in Z39.50 sessions? I suspect the answer is "iso-asn.1-abstract-syntax" (ie., ASN.1), but the standard doesn't say (though it certainly implies it, since all the definitions are in ASN.1 :-) :-)). Wengyik ======================================================================== 10 Date: Tue, 25 Sep 90 13:01:55 EDT From: "Mark Hinnebusch, FCLA" Subject: Re: Z39.50 V2 draft To: Distribution List In-Reply-To: Message of Tue, 25 Sep 90 09:43:11 -0400 from The draft of version 2 maintains the version 1 paging scheme for comparison's sake. There are no pages 39 thru 46 in version 2. See the tiny note at the top of page 47. --m ======================================================================== 26 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 5366; Mon, 24 Sep 90 15:26:18 EDT Date: Mon, 24 Sep 90 15:21:46 -0400 Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: yeongw@psi.com (Wengyik Yeong) To: "MARK HINNEBUSCH, FCLA" Subject: nominations for Explain service Cc: yeongw@psi.com I'd like to nominate the following for inclusion as information that can be retrieved via the Explain service: - Names of searchable databases - Names of legal element sets for a given named database (assumes (1)) - Default element set for a given named database (assumes (1)) - Brief description of type of information in a given, named, database (assumes (1)). - Whether target supports/does not support result set naming - Names of attributes that can be used for searching That's all I can think of for now. Apologies if these are already in the Explain service, since I've never seen its definition. Wengyik ======================================================================== 100 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 4828; Mon, 24 Sep 90 15:12:04 EDT To: "MARK HINNEBUSCH, FCLA" Subject: Questions/Comments on Z39.50 V2 Reply-To: yeongw@psi.com Cc: yeongw@psi.com Date: Mon, 24 Sep 90 15:07:52 -0400 Sender: "Z3950IW@NERVM via" From: yeongw@psi.com You knew this was coming, didn't you ... :-) I finally got back to the office and spent some time this morning skimming the Z39.50 V2 draft. Based on my first reading, I have the following comments/questions: 1) Are names (database names, result set names etc.) case sensitive? I would imagine not, since not all implementations may run on machines that support case sensitivity. But in any case, I think the document ought to say something, one way or the other. 2) In result sets, are records numbered beginning with 0 or 1? This is admittedly a small thing, but since the document doesn't specify, I can easily see some implementors assuming 0, while others assume 1. 3) In the references to bit strings in the ASN.1, does "first bit" refer to the leftmost or rightmost bit in the bit string? 4) In the definition of the Type 1 Query, it seems like there is some redundancy. Specifically, operands are always double-wrapped, as values of type Operand, wrapped again as Argument. Also, all RPNStructures with the exception of the topmost (the one directly subordinate to RPNQuery) seem to be double-wrapped also. Could I ask that the double-wrapping be removed? It seems wasteful (more bytes to transfer, more parsing to be done) to double wrap like that. Or was there some reason for the double-wrapping? To remove the double wrapping, the Argument construct is removed, with RPNStructure rewritten as follows: RPNStructure ::= CHOICE { [0] Operand, [1] IMPLICIT SEQUENCE { RPNStructure, RPNStructure, Operator } 5) Result Set Name (in the Search service) and Result Set Id (elsewhere, in the Present and Delete among other places) seem to have the same semantics. In the ASN.1 definition, they have distinct tags. Is there any reason for this? It seems to me that they should be the same. I suspect NISO thinks so too, since they "copped out" and introduced text (page 13, final sentence section 3.2.3.1.2) to the effect that Result-set-name and Result-set-id are the same :-) :-). 6) I am disturbed by the fact that the Id/Authentication service parameter to the Init is defined "outside the scope of the standard." While there are other things in this category (which will be the subject of a subsequent message), the others can be dealt with in the Explain service, whereas this one cannot: there is a "chicken and egg" problem here since an Explain cannot be issued before an Init (since there is no IR session) and the Init cannot be issued before an Explain is done to (hopefully) find the format of the ID/Authentication field. I think I understand why Id/Authentication is "outside the scope of the standard": because all database providers may potentially have their own authentication mechanisms. Hence a compromise was reached to just leave it out and make everybody happy. However I think leaving out Id/Authentication causes the problem I outlined above, and will ultimately make nobody happy. Couldn't another compromise be reached that *does* define the syntax of the Id/Authentication field? 7) Similarly (to (6)), I am disturbed that the format of the security challenge and security challenge response parameters of the Access Control service are left undefined. Pushing off definitions to the Explain service is one thing; pushing off definitions of definitions (the definition of the definition of the security challenge/response) is another. While I realize that having Explain provide definitions of definitions is ultimately doable, it requires the implementation to be able to compile a definition at run-time, and then dynamically load it. This is hard, and I'm not sure that such technology exists on all platforms where Z39.50 can be implemented. Again, I suspect that the decision was made to leave the definition of security challenge/response outside of the standard for much the same reasons as in (6), but again, I think that another compromise has to be reached. Finally, for the ZIG (I know this should not be part of the standard): does anyone want to propose a common format for User Information Fields (which are also left to the implementors' imagination)? I like Printable Strings personally, with the understanding that implementations "pass through" the contents of user information fields directly to the user without manipulating them. Wengyik ======================================================================== 51 Received: from NERVM by NERVM.NERDC.UFL.EDU (Mailer R2.07) with BSMTP id 4787; Tue, 04 Sep 90 13:58:43 EDT Date: Tue, 4 Sep 90 10:56:50 PDT To: "MARK HINNEBUSCH, FCLA" Sender: "Z3950IW@NERVM via" Reply-to: Distribution List From: "Lennie Stovel" Subject: ASN.1 for MARC records About the high road and the low road in ASN.1 definitions for MARC records: I'm having a struggle between them. Intellectually the concept of a complete ASN.1 definition is appealing. I think in the long run, it might be beneficial to the library community to adopt a more standard and structured view of machine-readable bibliographic and other records, and to separate the definition and the encoding. Also with a more complete encoding the recipient of the record has a better idea of the purpose of the record. But when I consider actual use, I find myself leaning towards the simpler definition and agreeing with Ralph that it's the application (not the Application Layer) that should deal with the specifics. We (RLG) already keep our internal (non-MARC) format and our conversion tables in sync with USMARC Updates; the thought of maintaining yet another complex syntax/encoding makes me shudder. When USMARC switches to ASN.1/BER... However, I wonder what level of generality is appropriate for intersystem information retrieval. Some questions about the OCLC ASN.1 definition: I'm not sure why CHOICE is used, because I don't understand how a record would actually be encoded. Got any examples, Ralph? Why are there definitions for indicator and Subfield? What was the need for breaking fields into their parts, given that the definition as a whole is so general? 007 doesn't have subfields and 010 does, so it seems like they are in the wrong places. 006, when it shows up with format integration, won't have subfields either. It does seem like this definition might have some characteristics that meet OCLC-specific needs that are not shared by the Z39.50 community. True? Lennie Stovel, Research Libraries Group To: Z3950IW@NERVM